User:SomeoneElse/map.atownsend.org.uk/Setup

From OpenStreetMap Wiki
Jump to navigation Jump to search

Server Setup

Getting the server and setting up the OS

The current server is the third in that I've used for this purpose - when I need to upgrade I setup a replacement server somewhere and update the DNS (at Gandi) to point at the new one. The current server is at Hetzner (mostly for reasons of "what you get for what you pay for"). It's a CAX41 and can comfortably deal with hosting a "Britain and Ireland" map and the other things that it gets used for. It has a fairly minimal installation of Debian 12 on it - I'll add required software later.

The reason for the choice of this OS over others is:

  • It's an LTS release, and so is supported for a long time
  • The latest mod_tile and renderd are available via "apt install" - setup is much easier (also true for Ubuntu 22.04)
  • More recent versions of other software used within OSM such as osm2pgsql and osmium-tool are also available directly via "apt install" (newer than in Debian 11)

The time zone default of Debian 12 is UTC - I have not changed that. As an aside, the time zone default of "Ubuntu 22.04.1 LTS Desktop" is local.

I initially added one ssh key (for use with the initial root account) and then later added other ssh keys to the non-root account I will use for future access.

It's got an IPV4 address and an IPV6 range. I set up DNS entries for both.

I made sure that all software was up to date with "sudo apt update; sudo apt upgrade".

I created a non-root account for regular use and added it to the "sudo" group.

Once that was done I removed the ability to ssh in as root, or to ssh in with a password rather than a key.

non-map software

I installed a mail server and command-line tools for mail. Using "/etc/aliases" root's mail is forwarded to the main non-root account on the server, and mail from there is forwared to an external mailbox.

The various QA scripts are set up so an output (and hence an email) is only produced if something has gone wrong.

I installed Apache as a web server.

I set up an ssl certificate for the web server via acme.sh. I tend to do certificate renewals in standalone mode, so create a small script to stop, test for renewal and restart apache, and add that to root's crontab in place of the default .acme.sh one.

To enable ssl in apache, edit "/etc/apache2/sites-enabled/default-ssl.conf" to contain the fullchain and key of the certificate, replacing the "snakeoil" values, so that it looks something like this: 

SSLCertificateFile      /root/.acme.sh/yourserveraddress_ecc/fullchain.cer
SSLCertificateKeyFile /root/.acme.sh/yourserveraddress_ecc/yourserveraddress.key

and then run: 

sudo a2enmod ssl
sudo a2ensite default-ssl
sudo systemctl reload apache2

I set up iptables for IPV4 and IPV6 and made the settings permanent. No "odd ports" will be in use; inbound will just be ssh, http and https.

map software

This essentially involves following the Ubuntu 22.04 or Debian 12 switch2osm guide down to but not including "Stylesheet configuration".

As well as creating a database "gis" I also created "gis3" (which is setup identically). The reason for having 2 databases instead of 1 is to eliminate downtime - a new version of the map style can be loaded into "gis3" while "gis" is still serving tiles to users, tested, and then switched over if the load was successful.

stylesheet configuration, initially with "mod_tile" from Ubuntu

Most of the software I've created for this map style can be configured to work under "~/src" under any desired non-root account. See an example script here.

The initial projects to clone to get us started are

(there are others for other data layers that we'll look at later) 

cd ~/src/openstreetmap-carto-AJT
time sh -x get-shapefiles.sh

Note that "get-shapefiles.sh" may fail to download some data from Natural Earth due to the bizarre way that their CDN is configured (redirects etc.). If this happens, download files referred to in the script manually. Also check issues at https://github.com/nvkelso/natural-earth-vector/issues to see if it is currently broken. 

sudo emacs /etc/renderd.conf

Add: 

[ajt]
URI=/hot/
XML=/home/ajtown/src/openstreetmap-carto-AJT/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20

and 

[ajt3]
URI=/hot3/
XML=/home/ajtown/src/openstreetmap-carto-AJT/mapnik3.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20

We will want to support zooms > 20 later, but lets check that the basics work first. 20 is the highest value supported by the standard mod_tile.

In the "openstreetmap-carto-AJT" area, 

carto project.mml > mapnik.xml

At the top of [service] in /usr/lib/systemd/system/renderd.service Add 

Environment=G_MESSAGES_DEBUG=all

Software used by the update script

The update_render.sh script is what I use for data loads. I copy it to /usr/local/sbin . I also copy make_gis3_live.sh there too (that swaps the "gis" and "gis3" databases around).

That needs osmium, so 

sudo apt install osmium-tool

It needs osm-tags-transform, which needs to be built from scratch.

Also pandoc and pyosmium: 

sudo apt install pandoc pyosmium

The pyosmium setup is very similar to what is described at switch2osm, except that two databases are used here "gis" and "gis3" (one can be loaded while the other is live). The directories used for these are "/var/cache/renderd/pyosmium.gis" and "/var/cache/renderd/pyosmium.gis3". Amend the pyosmium scripts such as "pyismium_replag.sh" accordingly.

Wherever the stylesheet is needs to be readable by the "_renderd" account that will load the database. IF it won't cause a security issue do "chmod o+rx ~" against the non-root account in use; if it will grant access via groups or some other mechanism.

Edit "update_render.sh" so that it's trying to load a couple of small areas initially, and then run as root via "sudo".

Ensure that a map tile displays OK - in my case (because by default "update_render.sh" loads the gis3 database) http://map.atownsend.org.uk/hot3/0/0/0.png .

"update_render.sh" also initialises database updates from https://planet.osm.org ; we'll look at that later.

Zoom levels above 20

The version of mod_tile distributed with Ubuntu only supports zoom levels up to 20. I've created a fork that supports up to 24, which we'll use on this server. See also this issue.

Install some prerequisites first, as documented here: 

sudo apt install build-essential \
  autoconf \
  apache2-dev \
  libcairo2-dev \
  libcurl4-gnutls-dev \
  libglib2.0-dev \
  libiniparser-dev \
  libmapnik-dev \
  libmemcached-dev \
  librados-dev

Then build mod_tile: 

cd ~/src
git clone https://github.com/SomeoneElseOSM/mod_tile
cd mod_tile
git checkout zoom2022
time ./autogen.sh
time ./configure
time make

If that all works: 

sudo make install
sudo make install-mod_tile

You should now have things such as "renderd" in "/usr/local/bin". Edit the service file /usr/lib/systemd/system/renderd.service" to use that instead of the original. Change 20 to 24 in /etc/renderd.conf . It's recommended to rename the "/usr/bin" ones out of the way to avoid calling them by accident. As an example, the old "render_expired" will run, but won't expire the right tiles, so you don't want to run it by mistake.

sudo systemctl daemon-reload sudo systemctl restart renderd sudo systemctl restart apache2

Remove all cached tiles (the directory format of the newly built version is different).

Test that everything still works, and that you can now zoom in to zoom 24. You can use test scripts "gis_test.html" and "gis3_test.html", which are in the "SomeoneElse-map" area for this. If not testing locally, you'll need to edit the IP address from "127.0.0.1" in those files.

SomeoneElse-map

This view shows the software in here. That is all set up below "/var/www/html/".

"leaflet_dist" and "control" are in ldc.tar. The leaflet version is quite old (0.7.7), but I much prefer the way that it works, especially on mobile devices.

The "gis_test.html" and "gis3_test.html" pages can be edited to look at the local "gis" and "gis3" databases (provided that /etc/renderd.conf is set up appropriately) by editing "127.0.0.1" to the desired name or IP address.

You'll want to increase "ModTileMissingRequestTimeout" to 30 or 60 (see the end of the relevant switch2osm page) to avoid displaying grey tiles to users.

"map" is the main site. "leaflet_embed_small.js" can be edited so that the source of all the layers is correct. Most are local, but some other useful layers (e.g. OSM DE) are also available. Setting up of the other local layers such as "floodedUrl" and "novisUrl" will be discussed later.

The HTML files here are created using "pandoc" by "update_render.sh" and are available from buttons at the top here. The sources are markdown files in various repositories.

Uncomment "AddDefaultCharset UTF-8" in "/etc/apache2/conf-available/charset.conf" so that the HTML files display correctly.

The "legend" button jumps to a location in the middle of Australia where the legend has been appended to the database. Zoom in and out and move around to see everything

Daily housekeeping scripts

Clone this directory

Set these up as required from cron as described here, and note other dependencies. Example cron files for each user are in this repository as e.g. "~/src/database_qa_scripts/report_database_size.sh".

In each case I've copied the script to /usr/local/sbin and edited (if necessary) to match local account names.

Note that "rerender_low.sh" depends on https://github.com/alx77/render_list_geo.pl , which also needs to be "git cloned" locally.

In addition, there are 3 files ("osm_ldp1", "osm_ldp2", "osm_ldp3") to be copied to "/etc/cron.d/" that depend on "/usr/local/sbin/count_generic.sh".

maps showing current flooding

Clone this directory

As described here, this uses an existing field in the database ("wetland") to store whether a particular linear or area feature is flooded or not. This script contains a list of OSM ways, Environment Agency station numbers and values. In each case the "is it flooded at that level" is as a result of survey. There's a download for each EA station (see e.g. on this page). The EA information is cached locally.

Install prerequisite software: 

sudo apt install bc

The stylesheet relies on this entry in /etc/renderd.conf: 

[ajt4]
URI=/hot4/
XML=/home/youruseraccount/src/floodedmap/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=28

The update scripts are set up in cron to run a few times per day, including "just before lunch". If the server time zone is UTC and the local time zone supports daylight saving, then the time when these scripts are run may need adjusting as local daylight saving comes in and out, as "just before lunch" changes by an hour at that time.

The "flood" overlay in "map.html" defaults to "map.atownsend.org.uk"; you'll want to change it to your server.

munin

This is covered by this guide.

updates via pyosmium

Most of this is covered here, but note that we have both "gis" and "gis3", so we will expect to have both "pyosmium.gis" and "pyosmium.gis3" below "/var/cache/renderd". The "make_gis3_live.sh" (see below) will swap databases around and will also swap the two pyosmium state directories around too.

Also note that the "call_pyosmium.sh" script needs to be edited so that the "osm2pgsql" parameters, such as the ".lua" and ".style" files, are correct.

"no vis paths" tiles

By default, paths that are not public footpaths and are not very visible on the ground are suppressed from the main map. The "no vis paths" tile layer optionally shows them. The Leaflet layer selector uses "hot5", and the entry in "/etc/renderd.conf" is "ajt5", pointing to a "mapnik.xml" below "novispaths" in "openstreetmap-carto-AJT" that is created by "update_renderd.sh".

boundaries

Clone this repository:

The script in there cuts down a PBF database and loads it into a "gis6" database, only containing boundaries. It then creates "hot6" overlay tiles based on a cut-down map style (project6.mml / mapnik6.xml) that only shows boundaries.

Create the "gis6" database in the same way as "gis" and "gis3", and add "hit6" to "/etc/renderd.conf" like "hot" and "hot3".

Garmin

The website also has pages for a couple of Garmin map styles, created using "mkgmap". To set that up:

Clone this repository:

Install prerequisite software: 

sudo apt install openjdk-17-jre-headless mkgmap-splitter mkgmap

Create these directories: 

/var/www/html/maps/mkgmap_maps
/var/www/html/maps/mkgmap_maps/ajt2
/var/www/html/maps/mkgmap_maps/ajt03

Make ajt2 and ajt03 writable by a regular user account.

There are two scripts, "garmin_map_etrex.sh" (which creates a basic mkgmap Garmin map) and "garmin_map_etrex_03.sh" (which creates a more detailed mkgmap Garmin map). Currently one of them is run weekly by cron and the other on an ad-hoc basis. If run from cron, it makes sense for the script in cron to copy resulting files under "/var/www/html/maps/mkgmap_maps/".

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:SomeoneElse/map.atownsend.org.uk/Setup&oldid=2714056"