Philippines/MAPANAN

 

 

The MAPANAN project of the City of Baguio is organized and initiated as part of Executive Order No. 104, S. 2023 with the purpose to support mapping in Baguio for OpenStreetMap and in return OpenStreetMap serving all offices, Baguio Smart City and all Barangays as well as tourists, constituents and companies like Grab, Facebook with the most accurate data for all.

Description

Project MAPANAN stands for Mapping and Aerial Photography for Area Navigation, Analysis and Network. This represents a strategic approach utilized by Baguio City, employing UAVs to capture high-resolution aerial images for urban planning, boundary delineation, road networks and street evaluation, and detailed mapping of alleyways and house numbers with the potential to extend the scope of work in the future.

The acronym MAPANAN underscores the city's commitment to leveraging modern technology for comprehensive mapping, navigation, and spatial analysis to facilitate informed decision-making in urban development and infrastructure management.

The key components of the MAPANAN process include:

1. Mapping: UAVs equipped with advanced cameras and sensors are deployed to capture detailed aerial imagery of Baguio City's topography, land use, and infrastructure. These high-resolution visuals serve as the foundation for accurate mapping and spatial analysis.

2. Aerial Photography: The collected imagery is processed to generate precise georeferenced orthographic images, orthomosaics, and digital elevation models (DEMs). These products provide valuable insights for navigation, planning, and analysis of the city's physical and human geography.

3. Area Navigation and Analysis: The geospatial data and imagery obtained through mapping and aerial photography are employed for area navigation, urban planning, boundary demarcation, road and street assessment, and the detailed mapping of alleyways, houses, and properties to establish a comprehensive and accurate address system.

MAPANAN embodies a synergy of mapping technology, aerial photography, and geospatial analysis in support of Baguio City's efforts to enhance urban development, infrastructure assessment, and spatial data management. By harnessing UAV technology and advanced spatial imaging techniques, the city gains a valuable resource for navigation, strategic planning, and evidence-based decision-making, ultimately contributing to the sustainable development and effective governance of the local area.

Projection

For our projection we adopt the NAMRIA recommended PRS92 Zone 3 aka EPSG:3123, which according to NAMRIA has an accuracy of 0.05m

Accuracy

Main article: Accuracy

At the city we work with equipment according to the manufacturer 0.05m accuracy. After extensive experiments using total stations we found our GNSS data with the Satlab equipment to be within 2-3cm accuracy and thus within the claims of the manufacturer. Using WebODM the resulting orthographic images were compared in QGIS to the raw data of the total station and after extensive testing with the ground control points properly measured and arranged the margin of error was up to 5cm with certain settings, so while we can confirm the orthographic algorithms do create some additional inaccuracies to the orthographic image data, this can be minimized to remain within 5cm.

At certain areas with lots of electrical wires, narrow streets, antennas and other factors disturbing signals and on very steep slopes along Quirino Hill most notably, we found our worst accuracy is 12cm, compared to the data we measured with the total station.

As of February 2024 we add a special accuracy tag to our data to point this out to all mappers and they can then use existing OSM data to align their edits with our data.

For more information see special tags

Equipment

Placeholder so I know I need to make a list of all the equipment being used for the project

total stations

drones

rtk equipment

tablets

servers

Analysis and data verification

placeholder for future information how to analyze and verify surveying and mapping data

Capturing Drone Aerial Imagery

Currently we have a several groups of aspirant drone operators undergoing their training to get their CAAP certificates. The goal is once everyone got their license to systematically capture, collect and process drone imagery for the whole city. Our teams will mostly be working with DJI as we found this gives us the best results for a reasonable procurement investment.

The concept is to do everything by the book following all legal obligations and regulations as set by the national government - to ensure we adhere to the mandate "public safety comes first"

Information Technology and Software Deployment (Server)

Processing Drone Aerial Imagery

Main article: WebODM

Generally read the WebODM article to setup and process drone images. The result is good, yet there's room for optimization. For instance some software, operating systems, or services have limited file-size they can work with, or at least you'll notice lag or other unwanted slow-downs. Anything that's limited to 32bit pointers automatically means ~4GB is the max you could work with

Geoserver

Main article: GeoServer

This article is a tutorial page how to set it up for hosting your own GeoTIFF, Shape, Satellite or any other file. Geoserver is very versatile and you're encouraged to read up on it.

Entwine

Main article: Entwine

Entwine in combination with Potree is powerful software to visualize captured drone imagery in a 3D environment. For mapping buildings but also other objects this can be very handy to have a 3d representation of a city so within the scope of MAPANAN we provide this to our surveyors and mappers.

Potree

Main article: Potree

Potree is a FOSS WebGL based point cloud renderer for large point clouds, developed at the Institute of Computer Graphics and Algorithms, TU Wien

Mantis

Main article: Mantis

Mantis is originally a bug tracker software for software development. However due to the open source nature over time plugins have been created that expand its features to be deployed also as project management tool and general issue tracking system through the utilization of these plugins.

This is especially very practical for organized surveying and mapping efforts conducted by teams of various offices especially taken into consideration possible participation and collaboration with projects such as YouthMappers or even including the broader mapping community for both contributing and using OpenStreetMap for and by LGUs.

NextCloud

For Nextcloud we use the snap installation, first we install snap itself

1. sudo apt update && sudo apt upgrade
2. sudo apt install snapd
3. snap --version
4. systemctl status snapd

Now we install nextcloud

1. sudo snap install nextcloud

Configuration

We set the ports to work without interfering with defaults

1. snap set nextcloud ports.http=8080 ports.https=8443

You can issue SSL by entering the command

1. nextcloud.enable-https self-signed

This will cause an error message:"Access through untrusted domain", you can fix this with the following steps

1. vi /var/snap/nextcloud/current/nextcloud/config/config.php
2. add the corresponding line(s), example

    array (
             0 => '192.168.90.44:8080',
             1 => '192.168.90.44:8443',
     ),

Trash bin

Make sure you set the trash bin cleanup properly! We're dealing with huge amounts of GB with our drone imagery projects!

For a snap installation follow these steps

sudo nextcloud.occ config:app:set files_trashbin trashbin_retention_obligation --value='auto, 7'

You can verify correct settings of your config.php by the following command

sudo nextcloud.occ config:list

now connect to the database

sudo nextcloud.mysql-client

now you're connected to the MySQL prompt and proceed as follow

USE nextcloud;

You should receive a message saying Database changed if successful.

UPDATE oc_jobs SET last_run = 0;

Verify this by entering

SELECT id, class, last_run FROM oc_jobs;

It should say 0 for all values

EXIT;

as mentioned on the nextcloud issue tracker reference, after doing this it can take some time for the next 'clean-up' to be initiated. Be patient...

Update files on Nextcloud automatically

If we wish to have WebODM or Geoserver acces the nextcloud from within the host OS, WebODM writing files to the Nextcloud folder, then we need a for Nextcloud to scan the files
sudo nextcloud.occ files:scan --all

Run a cron job which runs automatically as root
sudo crontab -e

In the crontab file enter the following for every 5 minutes
*/5 * * * * nextcloud.occ files:scan --all

Troubleshooting

For changing the username on Nextcloud there's no official information, though the following works help

Nextcloud snap config.ini location

/var/snap/nextcloud/current/nextcloud/config/config.php

Upon speed issues, copying goes very slow? Disable richdocuments!

sudo snap run nextcloud.occ app:disable richdocuments 
sudo snap run nextcloud.occ app:disable richdocumentscore

Linux

General setup and commands

Here we're going to add general commands and information which we use to setup/maintain our servers. This is helpful both for ourselves to have our own feedback and record of what we have been doing but also to cover what other people/LGU/companies might experience so they know what to do specifically when it comes to projects like these.

Swap memory

swapon --show	Shows the current swap configuration of your system
swapoff /swap.img	Turns off the corresponding swap file
fallocate -l 64G /swap.img	Increasing the swap file to 64G
mkswap /swap.img	Applies the 64G to the file
swapon /swap.img	Activating the swap, you now have 64G more memory for your processes in swap space. Keep in mind this is notably slower than real RAM!

File system commands

du -hs /path/to/directory	Show the total size a given folder is using-h means humanly readable (Mb,Gb, etc) -s means summary Append "|sort -h" to sort the output
df -h	Shows all disk usage for whole system

System information

free -hm	Shows RAM and Swap information in humanly readable format
top -i	Shows cpu load and memory usage on current processes Changing memory units, press following: e - running processes E - summary units at the top W - write permanently to /home/user/.toprc

General commands

find / -name filename	To find a specific file if the location is unknown
php --ini | grep "Loaded Configuration File"	Locate the PHP ini file
kill -9 <process_id>	Kill specific process

Multiple instances for multi-tasking, SSH without killing

Login to a new SSH session
tmux	This command starts a new tmux session, they're numbered 0,1, 2, ...
start the process you'd like to start, for instance an entwine process
ctrl+b followed by d	Allows to detach from a tmux process or SSH without killing it
tmux list-sessions	Lists all active running processes within a tmux instance
tmux attach-session -t 'session number'	Login back to a running session, for instance tmux attach-session -t 0
ctrl+b and $	Allows to change the default names, which is 0, 1, 2, etc to whatever you wish it to be

Extending the storage capacity from extended physical/virtual disk

Firstly of course increase the actual physical size of the disk presented to the Linux OS.
cfdisk	This command gives you the /dev/sda with the partitions and the "Free space." To expand the LVM partition: select the partition you wish to increase select resize select write and quit
echo 1>/sys/class/block/sda/device/rescan	If you don't see the newly added free space, use this command and run "cfdisk" again
pvresize /dev/sda3	extending the PV
pvdisplay	Verify the new size
vgdisplay	Verify Volume Group free space
lvdisplay	Show Logical Volume
lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv	Extend the Logical Volume
resize2fs /dev/mapper/ubuntu–vg-ubuntu–lv	Make the filesystem itself fit the new volume

Supporting applications and tools

Why using these tools and not something more fancy like SolarWinds Solar-PuTTY?

The tools we're using are "relatively" easy to use also for novices in the field of Linux. Not everyone at every office is an IT specialist sadly.

PuTTY

PuTTY is a handy app for remote administration of Linux servers, the ease of use lies in the primitive interface. The latest version can be downloaded from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html we simply use the putty.exe binary.

KiTTY

KiTTY is pretty much like PuTTY allows additionally features like scripting and collaboration and file editing, while still fairly easy to use for novices. KiTTY can be downloaded from https://www.9bis.net/kitty/#!pages/download.md for the sake of simplicity we use the precompiled portable version of KiTTY mentioned under the "Other available downloads"

Bitvise SSH Client

Bitvise is a bit more complicated for novice users but can legally be used by LGUs for free and allows significant more features that come in handy especially the easy copy-paste of files between Windows & Linux operating systems. It can be found here https://www.bitvise.com/ssh-client

Desktop Applications

Here we describe the applications which we use and what we use them for, or we link to the corresponding OSM wiki article

Paint.NET Optimization

This piece of software can significantly reduce the file size without sacrificing detail of your images.

Evidently the amount of optimization depends on what's visible on the image, some images can be optimized more than others and what compression was used for the original GeoTIFF.

Another nice feature of Paint.NET, some software when you export or change GeoTIFF it removes the transparency and you get white/black background instead. GeoTIFF turns this back into transparency, very handy when working with multiple layers, like Mosaic deployment of drone imagery.

Note, as of Jan 5 2024, Paint.NET does not support BigTIFF!

Comparison Table
QGIS 1.2 GB vs Paint.NET 0.75 GB ~ 38% file size reduction	QGIS saved file creek water around a rock	Paint.NET saved file creek water around a rock	QGIS Paint.NET comparison overview creek water around a rock
QGIS 3.3 GB vs Paint.NET 0.71 GB ~ 78% file size reduction	QGIS saved file waste bins	PAINTNET saved file waste bins	QGIS PAINTNET comparison overview waste bins
QGIS 2.2 GB vs Paint.NET 1.1 GB ~ 50% file size reduction	QGIS saved file watershed trees	PAINTNET saved file watershed trees	QGIS PAINTNET comparison overview watershed trees

QGIS

Splitting large files using QGIS

• Open your file in QGIS (drag & drop)
• Right click on the filename in the Layers panel
• Go to 'Export' and select 'Save As'
• At top right of 'Format' check the box for 'Create VRT'
• Under 'File name' select the folder where you wish your split files to be copied to
• Under VRT Tiles set the 'Max xolumns' and 'Max rows' to split , examples:
  • If you wish to split in two files just half the value of either columns or rows
  • If you wish to split in four files, half the value of both columns and rows
• If you wish to retain quality, leave all other settings as they are

GDAL

We use this library of tools for a variety of functions but below we'll list the most common tasks one would have to do with orthographic imagery and hosting it. The main thing is of course to get our data as accurate as possible and sometimes imagery has an offset.

In our city we work with PRS92 Zone 3 as mandated by NAMRIA, which equals the EPSG:3123

Our recommendation is to use the OSGeo4W installer/repository since it is versatile and allows not only GDAL but also QGIS, Python, GRASS, OGR, SpatiaLite and many more and does all the configuration for you.Just download the OSGeo4W network installer and execute.

The OSGeo4W Shell which you'll find in your start menu after installation allows you to run all GDAL all python commands in a shell environment.

Practical commands for working with BigTIFF files, especially offset corrections:

Command	Comment	Example
gdalinfo filename.tiff	This command reveals the metadata and statistical data of the file, important to notice is the upper left and the lower right corners, as this is the data you need for offset corrections.	Driver: GTiff/GeoTIFF Files: C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif Size is 14194, 13041 Coordinate System is: PROJCRS["WGS 84 / UTM zone 51N", BASEGEOGCRS["WGS 84", DATUM["World Geodetic System 1984", ELLIPSOID["WGS 84",6378137,298.257223563, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4326]], CONVERSION["UTM zone 51N", METHOD["Transverse Mercator", ID["EPSG",9807]], PARAMETER["Latitude of natural origin",0, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8801]], PARAMETER["Longitude of natural origin",123, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8802]], PARAMETER["Scale factor at natural origin",0.9996, SCALEUNIT["unity",1], ID["EPSG",8805]], PARAMETER["False easting",500000, LENGTHUNIT["metre",1], ID["EPSG",8806]], PARAMETER["False northing",0, LENGTHUNIT["metre",1], ID["EPSG",8807]]], CS[Cartesian,2], AXIS["(E)",east, ORDER[1], LENGTHUNIT["metre",1]], AXIS["(N)",north, ORDER[2], LENGTHUNIT["metre",1]], USAGE[ SCOPE["Navigation and medium accuracy spatial referencing."], AREA["Between 120┬░E and 126┬░E, northern hemisphere between equator and 84┬░N, onshore and offshore. China. Indonesia. Japan. North Korea. Philippines. Russian Federation. South Korea. Taiwan."], BBOX[0,120,84,126]], ID["EPSG",32651]] Data axis to CRS axis mapping: 1,2 Origin = (243886.811842020950280,1814572.054954239632934) Pixel Size = (0.015863345742173,-0.015862821487728) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=DEFLATE INTERLEAVE=PIXEL LAYOUT=COG PREDICTOR=2 Corner Coordinates: Upper Left ( 243886.812, 1814572.055) (120d36' 7.22"E, 16d23'55.98"N) Lower Left ( 243886.812, 1814365.188) (120d36' 7.31"E, 16d23'49.25"N) Upper Right ( 244111.976, 1814572.055) (120d36'14.81"E, 16d23'56.06"N) Lower Right ( 244111.976, 1814365.188) (120d36'14.89"E, 16d23'49.34"N) Center ( 243999.394, 1814468.621) (120d36'11.06"E, 16d23'52.66"N) Band 1 Block=256x256 Type=Byte, ColorInterp=Red Description = red Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Band 2 Block=256x256 Type=Byte, ColorInterp=Green Description = green Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Description = blue Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203 Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha Overviews: 7097x6520, 3548x3260, 1774x1630, 887x815, 443x407, 221x203
In the above example we see for the upper-left easting: 243886.812 and the upper-left northing:1814572.055 Similar for the lower-right easting: 244111.976 and the lower-right northing:1814365.188 So if we want to fix the offset by 1 meter to the east, we add +1 to both easting values (analog for north, west or south) and the result becomes upper-left easting: 243887.812 and the upper-left northing:1814572.055 lower-right easting: 244112.976 and the lower-right northing:1814365.188
gdal_edit -a_ullr <ul_easting> <ul_northing> <lr_easting> <lr_northing> filename_fullpath	Example on the right, will shift the whole file 1 meter to the east.	gdal_edit -a_ullr 243887.812 1814572.055 244112.976 1814365.188 C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif
-oo IGNORE_COG_LAYOUT_BREAK=YES	if you get a COG warning - this is just an optimization in normal situations you can proceed by simple overriding the optimization	gdal_edit -a_ullr 243887.812 1814572.055 244112.976 1814365.188 C:\Users\vince\Downloads\TEST\2021-06-01_epsg32651_ort_uc_santa-escolastica-village.tif -oo IGNORE_COG_LAYOUT_BREAK=YES
gdal_translate -scrwin <xoff> <yoff> <xsize> <ysize> <source_file> <destination_file>	The values to enter here are the pixels. You enter first the offset to the upper left corner and then the width and height you wish to split from the main image	gdal_translate -srcwin 50275 128955 25137 21492 E:\NextCloud-GeoServer\survey\satellite\2019\2019-epsg3123-ort-baguio.tif E:\NextCloud-GeoServer\survey\satellite\2019\2019-epsg3123-ort-baguio_30.tif This is a sample command which we'd use to cut the main satellite image, which is 201098 by 171943 pixels and split it into 8 by 8 ~ 64 smaller images. At the center we have 4 and we start counting in the center top left 0 So the image will be split into, the above formula assumes the top left starts at 1 in x and 3 in y directions and then add in x-direction 25137 and in y-direction 21492 every step. The reason is that we get everything nicely split in 8 by 8 of equal sizes if we cut a bit at the boundaries. 36 37 38 39 40 41 42 43 63 16 17 18 19 20 21 44 62 35 4 5 6 7 22 45 61 34 15 0 1 8 23 46 30 33 14 3 2 9 24 47 59 32 13 12 11 10 25 48 58 31 30 29 28 27 26 49 57 56 55 54 53 52 51 50 so in x-direction we get the following incremental numbers 1 25138 50275 75412 100549 125686 150823 175960 201097 And in the y-direction we get the numbers 3 21495 42987 64479 85971 107463 128955 150447 171939

Mobile devices Applications

For our mobile devices like tablets and phones we use the following applications

• Every Door
• StreetComplete
• SCEE
• OsmAnd
• Vespucci

Other installations

In this section we cover any other software that's relevant to mapping, GIS, setting up servers, etc.

Oracle VM aka VirtualBox

This is critical to split the various tasks/software/functions we're setting up also for you to have a testing environment.

Since this is very out of the scope of mapping, I merely wish to list links to up-to-date tutorials how to get Oracle VM properly setup for your needs within a mapping environment as you will be using Python a lot in the field of GIS.

Make sure before you run the installer of VirtualBox to get your python environment up and running:

• https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 Download the latest version, which is the 2015-2022 as of Apr 2024
• https://www.sysnettechsolutions.com/en/fix-python-win32api-virtualbox/ Critical to get Python working with Oracle VM