Philippines/MAPANAN
Description
Project MAPANAN stands for Mapping and Aerial Photography for Area Navigation, Analysis and Network. This represents a strategic approach utilized by Baguio City, employing UAVs to capture high-resolution aerial images for urban planning, boundary delineation, road networks and street evaluation, and detailed mapping of alleyways and house numbers with the potential to extend the scope of work in the future.
The acronym MAPANAN underscores the city's commitment to leveraging modern technology for comprehensive mapping, navigation, and spatial analysis to facilitate informed decision-making in urban development and infrastructure management.
The key components of the MAPANAN process include:
1. Mapping: UAVs equipped with advanced cameras and sensors are deployed to capture detailed aerial imagery of Baguio City's topography, land use, and infrastructure. These high-resolution visuals serve as the foundation for accurate mapping and spatial analysis.
2. Aerial Photography: The collected imagery is processed to generate precise georeferenced orthographic images, orthomosaics, and digital elevation models (DEMs). These products provide valuable insights for navigation, planning, and analysis of the city's physical and human geography.
3. Area Navigation and Analysis: The geospatial data and imagery obtained through mapping and aerial photography are employed for area navigation, urban planning, boundary demarcation, road and street assessment, and the detailed mapping of alleyways, houses, and properties to establish a comprehensive and accurate address system.
MAPANAN embodies a synergy of mapping technology, aerial photography, and geospatial analysis in support of Baguio City's efforts to enhance urban development, infrastructure assessment, and spatial data management. By harnessing UAV technology and advanced spatial imaging techniques, the city gains a valuable resource for navigation, strategic planning, and evidence-based decision-making, ultimately contributing to the sustainable development and effective governance of the local area.
Projection
For our projection we adopt the NAMRIA recommended PRS92 Zone 3[1] aka EPSG:3123, which according to NAMRIA has an accuracy of 0.05m
Accuracy
- Main article: Accuracy
At the city we work with equipment according to the manufacturer 0.05m accuracy. After extensive experiments using total stations we found our GNSS data with the Satlab equipment to be within 2-3cm accuracy and thus within the claims of the manufacturer. Using WebODM the resulting orthographic images were compared in QGIS to the raw data of the total station and after extensive testing with the ground control points properly measured and arranged the margin of error was up to 5cm with certain settings, so while we can confirm the orthographic algorithms do create some additional inaccuracies to the orthographic image data, this can be minimized to remain within 5cm.
At certain areas with lots of electrical wires, narrow streets, antennas and other factors disturbing signals and on very steep slopes along Quirino Hill most notably, we found our worst accuracy is 12cm, compared to the data we measured with the total station.
As of February 2024 we add a special accuracy tag to our data to point this out to all mappers and they can then use existing OSM data to align their edits with our data.
For more information see special tags
Equipment
Placeholder so I know I need to make a list of all the equipment being used for the project
total stations
drones
rtk equipment
tablets
servers
Analysis and data verification
placeholder for future information how to analyze and verify surveying and mapping data
Capturing Drone Aerial Imagery
Currently we have a several groups of aspirant drone operators undergoing their training to get their CAAP certificates. The goal is once everyone got their license to systematically capture, collect and process drone imagery for the whole city. Our teams will mostly be working with DJI as we found this gives us the best results for a reasonable procurement investment.
The concept is to do everything by the book following all legal obligations and regulations as set by the national government - to ensure we adhere to the mandate "public safety comes first"
Information Technology and Software Deployment (Server)
Processing Drone Aerial Imagery
- Main article: WebODM
Generally read the WebODM article to setup and process drone images. The result is good, yet there's room for optimization. For instance some software, operating systems, or services have limited file-size they can work with, or at least you'll notice lag or other unwanted slow-downs. Anything that's limited to 32bit pointers automatically means ~4GB is the max you could work with
Geoserver
- Main article: GeoServer
This article is a tutorial page how to set it up for hosting your own GeoTIFF, Shape, Satellite or any other file. Geoserver is very versatile and you're encouraged to read up on it.
Entwine
- Main article: Entwine
Entwine in combination with Potree is powerful software to visualize captured drone imagery in a 3D environment. For mapping buildings but also other objects this can be very handy to have a 3d representation of a city so within the scope of MAPANAN we provide this to our surveyors and mappers.
Potree
- Main article: Potree
Potree is a FOSS WebGL based point cloud renderer for large point clouds, developed at the Institute of Computer Graphics and Algorithms, TU Wien
Mantis
- Main article: Mantis
Mantis is originally a bug tracker software for software development. However due to the open source nature over time plugins have been created that expand its features to be deployed also as project management tool and general issue tracking system through the utilization of these plugins.
This is especially very practical for organized surveying and mapping efforts conducted by teams of various offices especially taken into consideration possible participation and collaboration with projects such as YouthMappers or even including the broader mapping community for both contributing and using OpenStreetMap for and by LGUs.
NextCloud
For Nextcloud we use the snap installation, first we install snap itself[2]
sudo apt update && sudo apt upgrade
sudo apt install snapd
snap --version
systemctl status snapd
Now we install nextcloud[3]
sudo snap install nextcloud
Configuration
We set the ports to work without interfering with defaults[4]
- snap set nextcloud ports.http=8080 ports.https=8443
You can issue SSL by entering the command[5]
- nextcloud.enable-https self-signed
This will cause an error message:"Access through untrusted domain", you can fix this with the following steps[6][7]
vi /var/snap/nextcloud/current/nextcloud/config/config.php
- add the corresponding line(s), example
array ( 0 => '192.168.90.44:8080', 1 => '192.168.90.44:8443', ),
Trash bin[8]
Make sure you set the trash bin cleanup properly! We're dealing with huge amounts of GB with our drone imagery projects!
For a snap installation follow these steps
sudo nextcloud.occ config:app:set files_trashbin trashbin_retention_obligation --value='auto, 7'
[9]
You can verify correct settings of your config.php
by the following command
sudo nextcloud.occ config:list
now connect to the database[10]
sudo nextcloud.mysql-client
now you're connected to the MySQL prompt and proceed as follow
USE nextcloud;
You should receive a message saying Database changed
if successful.
UPDATE oc_jobs SET last_run = 0;
Verify this by entering
SELECT id, class, last_run FROM oc_jobs;
It should say 0
for all values
EXIT;
as mentioned on the nextcloud issue tracker reference, after doing this it can take some time for the next 'clean-up' to be initiated. Be patient...
Update files on Nextcloud automatically
If we wish to have WebODM or Geoserver acces the nextcloud from within the host OS, WebODM writing files to the Nextcloud folder, then we need a for Nextcloud to scan the files[11]sudo nextcloud.occ files:scan --all
Run a cron job which runs automatically as rootsudo crontab -e
In the crontab file enter the following for every 5 minutes */5 * * * * nextcloud.occ files:scan --all
Troubleshooting
For changing the username on Nextcloud there's no official information, though the following works help
Nextcloud snap config.ini location
/var/snap/nextcloud/current/nextcloud/config/config.php
Upon speed issues, copying goes very slow? Disable richdocuments![12]
sudo snap run nextcloud.occ app:disable richdocuments
sudo snap run nextcloud.occ app:disable richdocumentscore
Linux
General setup and commands
Here we're going to add general commands and information which we use to setup/maintain our servers. This is helpful both for ourselves to have our own feedback and record of what we have been doing but also to cover what other people/LGU/companies might experience so they know what to do specifically when it comes to projects like these.
Swap memory[13]
swapon --show |
Shows the current swap configuration of your system |
swapoff /swap.img |
Turns off the corresponding swap file |
fallocate -l 64G /swap.img |
Increasing the swap file to 64G |
mkswap /swap.img |
Applies the 64G to the file |
swapon /swap.img |
Activating the swap, you now have 64G more memory for your processes in swap space. Keep in mind this is notably slower than real RAM! |
File system commands
du -hs /path/to/directory
|
Show the total size a given folder is using[14]-h means humanly readable (Mb,Gb, etc)
-s means summary Append "| |
df -h |
Shows all disk usage for whole system |
System information
free -hm
|
Shows RAM and Swap information in humanly readable format |
top -i
|
Shows cpu load and memory usage on current processes
Changing memory units, press following:[15]
|
General commands
find / -name filename |
To find a specific file if the location is unknown |
php --ini | grep "Loaded Configuration File" |
Locate the PHP ini file |
kill -9 <process_id> |
Kill specific process |
Multiple instances for multi-tasking, SSH without killing[16]
Login to a new SSH session | |
tmux |
This command starts a new tmux session, they're numbered 0,1, 2, ... |
start the process you'd like to start, for instance an entwine process | |
ctrl+b followed by d |
Allows to detach from a tmux process or SSH without killing it |
tmux list-sessions |
Lists all active running processes within a tmux instance |
tmux attach-session -t 'session number' |
Login back to a running session, for instance tmux attach-session -t 0 |
ctrl+b and $ |
Allows to change the default names, which is 0, 1, 2, etc to whatever you wish it to be |
Extending the storage capacity from extended physical/virtual disk[17]
Firstly of course increase the actual physical size of the disk presented to the Linux OS. | |
cfdisk |
This command gives you the /dev/sda with the partitions and the "Free space." To expand the LVM partition:
|
echo 1>/sys/class/block/sda/device/rescan |
If you don't see the newly added free space, use this command and run "cfdisk" again |
pvresize /dev/sda3 |
extending the PV |
pvdisplay |
Verify the new size |
vgdisplay |
Verify Volume Group free space |
lvdisplay |
Show Logical Volume |
lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv |
Extend the Logical Volume |
resize2fs /dev/mapper/ubuntu–vg-ubuntu–lv |
Make the filesystem itself fit the new volume |
Supporting applications and tools
Why using these tools and not something more fancy like SolarWinds Solar-PuTTY?
The tools we're using are "relatively" easy to use also for novices in the field of Linux. Not everyone at every office is an IT specialist sadly.
PuTTY
PuTTY is a handy app for remote administration of Linux servers, the ease of use lies in the primitive interface. The latest version can be downloaded from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html we simply use the putty.exe binary.
KiTTY
KiTTY is pretty much like PuTTY allows additionally features like scripting and collaboration and file editing, while still fairly easy to use for novices. KiTTY can be downloaded from https://www.9bis.net/kitty/#!pages/download.md for the sake of simplicity we use the precompiled portable version of KiTTY mentioned under the "Other available downloads"
Bitvise SSH Client
Bitvise is a bit more complicated for novice users but can legally be used by LGUs for free and allows significant more features that come in handy especially the easy copy-paste of files between Windows & Linux operating systems. It can be found here https://www.bitvise.com/ssh-client
Desktop Applications
Here we describe the applications which we use and what we use them for, or we link to the corresponding OSM wiki article
MOBAC
JOSM
Paint.NET Optimization
This piece of software can significantly reduce the file size without sacrificing detail of your images.
Evidently the amount of optimization depends on what's visible on the image, some images can be optimized more than others and what compression was used for the original GeoTIFF.
Another nice feature of Paint.NET, some software when you export or change GeoTIFF it removes the transparency and you get white/black background instead. GeoTIFF turns this back into transparency, very handy when working with multiple layers, like Mosaic deployment of drone imagery.
Note, as of Jan 5 2024, Paint.NET does not support BigTIFF!
Comparison Table | |||
---|---|---|---|
QGIS
Splitting large files using QGIS[18]
- Open your file in QGIS (drag & drop)
- Right click on the filename in the Layers panel
- Go to 'Export' and select 'Save As'
- At top right of 'Format' check the box for 'Create VRT'
- Under 'File name' select the folder where you wish your split files to be copied to
- Under VRT Tiles set the 'Max xolumns' and 'Max rows' to split , examples:
- If you wish to split in two files just half the value of either columns or rows
- If you wish to split in four files, half the value of both columns and rows
- If you wish to retain quality, leave all other settings as they are
GDAL
We use this library of tools for a variety of functions but below we'll list the most common tasks one would have to do with orthographic imagery and hosting it. The main thing is of course to get our data as accurate as possible and sometimes imagery has an offset.
In our city we work with PRS92 Zone 3 as mandated by NAMRIA, which equals the EPSG:3123
Our recommendation is to use the OSGeo4W installer/repository since it is versatile and allows not only GDAL but also QGIS, Python, GRASS, OGR, SpatiaLite and many more and does all the configuration for you.Just download the OSGeo4W network installer and execute.
The OSGeo4W Shell which you'll find in your start menu after installation allows you to run all GDAL all python commands in a shell environment.
Practical commands for working with BigTIFF files, especially offset corrections:
Command | Comment | Example | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
gdalinfo filename.tiff
|
This command reveals the metadata and statistical data of the file, important to notice is the upper left and the lower right corners, as this is the data you need for offset corrections. | Driver: GTiff/GeoTIFF
Files: C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif
Size is 14194, 13041
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 51N",
BASEGEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]],
CONVERSION["UTM zone 51N",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",123,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",0.9996,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["(E)",east,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["(N)",north,
ORDER[2],
LENGTHUNIT["metre",1]],
USAGE[
SCOPE["Navigation and medium accuracy spatial referencing."],
AREA["Between 120┬░E and 126┬░E, northern hemisphere between equator and 84┬░N, onshore and offshore. China. Indonesia. Japan. North Korea. Philippines. Russian Federation. South Korea. Taiwan."],
BBOX[0,120,84,126]],
ID["EPSG",32651]]
Data axis to CRS axis mapping: 1,2
Origin = (243886.811842020950280,1814572.054954239632934)
Pixel Size = (0.015863345742173,-0.015862821487728)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=DEFLATE
INTERLEAVE=PIXEL
LAYOUT=COG
PREDICTOR=2
Corner Coordinates:
Upper Left ( 243886.812, 1814572.055) (120d36' 7.22"E, 16d23'55.98"N)
Lower Left ( 243886.812, 1814365.188) (120d36' 7.31"E, 16d23'49.25"N)
Upper Right ( 244111.976, 1814572.055) (120d36'14.81"E, 16d23'56.06"N)
Lower Right ( 244111.976, 1814365.188) (120d36'14.89"E, 16d23'49.34"N)
Center ( 243999.394, 1814468.621) (120d36'11.06"E, 16d23'52.66"N)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
Description = red
Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Mask Flags: PER_DATASET ALPHA
Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
Description = green
Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Mask Flags: PER_DATASET ALPHA
Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
Description = blue
Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Mask Flags: PER_DATASET ALPHA
Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha
Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
In the above example we see for the upper-left easting: 243886.812 and the upper-left northing:1814572.055
Similar for the lower-right easting: 244111.976 and the lower-right northing:1814365.188 So if we want to fix the offset by 1 meter to the east, we add +1 to both easting values (analog for north, west or south) and the result becomes upper-left easting: 243887.812 and the upper-left northing:1814572.055 lower-right easting: 244112.976 and the lower-right northing:1814365.188 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
gdal_edit -a_ullr <ul_easting> <ul_northing> <lr_easting> <lr_northing> filename_fullpath
|
Example on the right, will shift the whole file 1 meter to the east. | gdal_edit -a_ullr 243887.812 1814572.055 244112.976 1814365.188 C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
-oo IGNORE_COG_LAYOUT_BREAK=YES
|
if you get a COG warning - this is just an optimization in normal situations you can proceed by simple overriding the optimization | gdal_edit -a_ullr 243887.812 1814572.055 244112.976 1814365.188 C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif -oo IGNORE_COG_LAYOUT_BREAK=YES
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
gdal_translate -scrwin <xoff> <yoff> <xsize> <ysize> <source_file> <destination_file> |
The values to enter here are the pixels. You enter first the offset to the upper left corner and then the width and height you wish to split from the main image | gdal_translate -srcwin 50275 128955 25137 21492 E:\NextCloud-GeoServer\survey\satellite\2019\2019-epsg3123-ort-baguio.tif E:\NextCloud-GeoServer\survey\satellite\2019\2019-epsg3123-ort-baguio_30.tif So the image will be split into, the above formula assumes the top left starts at 1 in x and 3 in y directions and then add in x-direction 25137 and in y-direction 21492 every step. The reason is that we get everything nicely split in 8 by 8 of equal sizes if we cut a bit at the boundaries.
so in x-direction we get the following incremental numbers
And in the y-direction we get the numbers
|
Mobile devices Applications
For our mobile devices like tablets and phones we use the following applications
Other installations
In this section we cover any other software that's relevant to mapping, GIS, setting up servers, etc.
Oracle VM aka VirtualBox
This is critical to split the various tasks/software/functions we're setting up also for you to have a testing environment.
Since this is very out of the scope of mapping, I merely wish to list links to up-to-date tutorials how to get Oracle VM properly setup for your needs within a mapping environment as you will be using Python a lot in the field of GIS.
Make sure before you run the installer of VirtualBox to get your python environment up and running:
- https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 Download the latest version, which is the 2015-2022 as of Apr 2024
- https://www.sysnettechsolutions.com/en/fix-python-win32api-virtualbox/ Critical to get Python working with Oracle VM
References
- ↑ https://epsg.io/3123
- ↑ https://www.cyberithub.com/how-to-install-snap-on-ubuntu-22-04/
- ↑ https://github.com/nextcloud-snap/nextcloud-snap/wiki/Install-Nextcloud-snap
- ↑ https://github.com/nextcloud-snap/nextcloud-snap/wiki/Port-configuration
- ↑ https://github.com/nextcloud-snap/nextcloud-snap/wiki/Managing-HTTP-encryption-(HTTPS)
- ↑ https://docs.nextcloud.com/server/27/admin_manual/installation/installation_wizard.html#trusted-domains
- ↑ https://github.com/nextcloud-snap/nextcloud-snap/wiki/Configure-config.php
- ↑ https://help.nextcloud.com/t/empty-trashbin-automatically/75933/14
- ↑ https://help.nextcloud.com/t/issue-with-automatic-deletion-of-files-from-trashbin-in-nextcloud-snap/180068
- ↑ https://help.nextcloud.com/t/ran-update-oc-jobs-set-last-run-0-to-database/186452/3
- ↑ https://help.nextcloud.com/t/using-occ-on-snap-nextcloud-installation/107129/2
- ↑ https://www.reddit.com/r/NextCloud/comments/k7b1of/extremely_slow_interface_nextcloud_snap/
- ↑ https://linuxhandbook.com/increase-swap-ubuntu/
- ↑ https://askubuntu.com/questions/1224/how-do-i-determine-the-total-size-of-a-directory-folder-from-the-command-line#1226
- ↑ https://unix.stackexchange.com/questions/122475/human-readable-memory-sizes-in-top
- ↑ https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
- ↑ https://packetpushers.net/blog/ubuntu-extend-your-default-lvm-space/
- ↑ https://gis.stackexchange.com/questions/221671/splitting-tif-image-into-several-tiles
|