User:SomeoneElse/Ubuntu 1604 tileserver load

From OpenStreetMap Wiki
Jump to: navigation, search

Introduction

This text describes what's required to take a "normal home PC" and to install a tile server on it. The "normal home PC" that I used was a Lenovo H30 (AMD A8-7410 with 1 socket, 4 cores, 8Gb memory, 1Tb of SATA-connected 7200 rpm disk). It runs Windows 10, but after a BIOS change it is possible to boot a different OS.

Note that I'm installing just a tile server - so I'm not going install a GUI or much other software.


Preparation

First, I installed Virtualbox after downloading from https://www.virtualbox.org/ .

If you need to control it remotely, you can do so by installing RemoteBox from http://remotebox.knobgoblin.org.uk/ .


In the "Oracle VM VirtualBox Manager" (or RemoteBox) change the default machine folder to "c:\Virtual_Machines".

Create a new guest.

I used a name of "UbuntuVM41"; you can use what you like.
OS Family: Linux
Version: Ubuntu (64-bit)
Memory: 3Gb

(if you've got less than 8Gb memory on the host machine you can use a lower value, such as 2Gb, here)

Create a new hard disk, 100Gb, VMDK format, dynamically allocated. Dynamically allocated means that the full 100Gb won't be used immediately. Following these notes only result in about around 6 Gb being used initially, though that will grow as the database is updated and tiles are rendered..

Ubuntu Installation

I downloaded Ubuntu 16.04.1 server from http://www.ubuntu.com/download/server . Using "Oracle VM VirtualBox Manager" go into "settings" and "storage", and below the IDE controller add the .iso file

Using "Oracle VM VirtualBox Manager" start the VM, and go through the Ubuntu Setup:

English, Install, English, United Kingdom
Detect Keyboard Layout: No, English UK, English UK
Hostname: ubuntuvm41
I set networking to bridged in "Network" to make it easier to access externally.
I chose "renderaccount" for the new user; you can use what you like.
I didn't encrypt my home directory.
Partitioning: Guided - Use entire disk
Write the changes to disk
I don't need configure a proxy for the package manager; you might.
I selected "Install Security updates automatically"
In software I left "standard system utilities" in place and added "Openssh server"
Install grub onto MBR (that's the MBR of the virtual machine, not the host PC, of course!)
Yes to reboot

I'll login via ssh remotely; you can also use the console which should just be displaying a "login:" prompt.

Firstly, install any pending ubuntu updates:

sudo -i
apt update

If this gives an error "is not in the sudoers file", then you'll need to allow that account to become root via e.g. "usermod -aG sudo renderaccount" (and replace "renderaccount" with whatever non-root username you chose above).

apt upgrade

That installs a new kernel, so:

sync;sync;shutdown -r now

Tile server pre-requisite software

After rebooting, ssh back in again and install the first set of prerequisites for a tile server:

sudo apt install libboost-all-dev git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libpng12-dev libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg node-carto

Say yes to install. This will take a while, so go and have a cup of tea. This list includes various utilities and libraries, the Apache web server, and "carto" which is used to convert Carto-CSS stylesheets into something that "mapnik" the map renderer can understand. When that is complete, install the second set of prerequisites:

sudo apt-get install postgresql postgresql-contrib postgis postgresql-9.5-postgis-2.2

Here "postgresql" is the database we're going to store map data and "postgis" adds some extra graphical support to it. Again, say yes to install.


Note - at any stage of this installation you might want to take a backup of the current situation before continuing. You can either shut the machine down and do a backup of the virtual machine's files (it'll be in an obvious directory in c:\virtual_machines), or you can take a "snapshot" as a temporary backup from either "Oracle VM VirtualBox Manager" or "RemoteBox" in case you need to restore it later.

Create and set up a database

We now want to do some configuration as the "postgres" user (which was created by the installation above). We want to set up the user that we created above as a postgres user, and create a new database with that user as the owner.

sudo -u postgres -i
createuser renderaccount
createdb -E UTF8 -O renderaccount gis

That command creates an empty database called "gis" (which is the default for many OSM tools), owned by "renderaccount". Again, replace "renderaccount" with your non-root username.

While still working as the "postgres" user:

psql

(that'll put you at a "postgres=#" prompt)

\c gis

(it'll answer 'You are now connected to database "gis" as user "postgres".')

CREATE EXTENSION postgis;

(it'll answer CREATE EXTENSION)

ALTER TABLE geometry_columns OWNER TO renderaccount;

(it'll answer ALTER TABLE)

ALTER TABLE spatial_ref_sys OWNER TO renderaccount;

(it'll answer ALTER TABLE)

\q

(it'll exit psql and go back to a normal Linux prompt)

exit

(to exit back to be the user that we were before we did "sudo -u postgres -i" above)

Again, above replace "renderaccount" with the non-root username that you chose.

Install osm2pgsql

We will need install various bits of software from source. The first of this is "osm2pgsql":

mkdir ~/src
cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql

Various tools to import and manage OpenStreetMap data into a database exist. Here we'll use "osm2pgsql", which is probably the most popular.

The build mechanism used by osm2pgsql has changed since older versions, so we'll need to install some more prerequisites for that:

sudo apt install make cmake g++ libboost-dev libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 liblua5.2-dev

Again, say yes to install.

mkdir build && cd build
cmake ..

(the output from that should end with "build files have been written to...)

make

(the output from that should finish with "[100%] Built target osm2pgsql")

sudo make install

Mapnik

Next, we'll install Mapnik. We'll use the default version in Ubuntu 16.04:

sudo apt-get install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libgdal1-dev libmapnik-dev mapnik-utils python-mapnik


We'll check that mapnik is installed OK:

python

(it'll reply with ">>>")

import mapnik

(if it replies again with just ">>>" - that's good)

Press control-d to exit the "python" prompt.

mod_tile and renderd

Next, we'll install mod_tile and renderd. "mod_tile" is an Apache module that handles requests for tiles; "renderd" is a daemon that actually renders tiles when "mod_tile" requests them. We'll use https://github.com/SomeoneElseOSM/mod_tile, which is itself forked from https://github.com/openstreetmap/mod_tile, but modified so that it supports Ubuntu 16.04, and with a couple of other changes to work on a standard Ubuntu server rather than one of OSM's rendering servers.

cd ~/src
git clone git://github.com/SomeoneElseOSM/mod_tile.git
cd mod_tile
./autogen.sh

(that should finish with "autoreconf: Leaving directory `.'")

./configure

(that should finish with "config.status: executing libtool commands")

make

Note that some "worrying" messages will scroll up the screen here. However it should finish with "make[1]: Leaving directory '/home/renderaccount/src/mod_tile'".

sudo make install

(that should finish with "make[1]: Leaving directory '/home/renderaccount/src/mod_tile'")

sudo make install-mod_tile

(that should finish with "chmod 644 /usr/lib/apache2/modules/mod_tile.so")

sudo ldconfig

(that shouldn't reply with anything)

Setting up a stylesheet

The next stage is setting up a stylesheet, so now's a good time to do a backup, using one of the methods described above.


The style we'll use here is based on and still similar to "openstreetmap-carto" (the "standard" style on the OpenStreetMap.org website), but is easier to configure and use and displays some features (such as offices, sidewalks, and English/Welsh "public footpaths") that openstreetmap-carto doesn't. There are two parts to it - a "lua" script which is called by osm2pgsql ("SomeoneElse-style") and the stylesheet itself ("openstreetmap-carto-AJT").

cd ~/src
git clone git://github.com/SomeoneElseOSM/SomeoneElse-style.git
git clone git://github.com/SomeoneElseOSM/openstreetmap-carto-AJT.git

Fonts needed by the stylesheet

The names used for places around the world aren’t always written with latin characters (the familar western alphabet a-z). The actual fonts needed will depend on the stylesheet; this list is correct for the stylesheet above and also for the "openstreetmap-carto" style on which it is based. To install the necessary fonts do the following:

sudo apt-get install fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont

OpenSteetMap Carto’s own installation instructions also suggest installing “Noto Emoji Regular” from source. That is needed for the emojis in an American shop name, apparently. All the other international fonts that are likely to be needed (including ones often not supported) are including in the list just installed.

Loading Data

Initially, we'll load only a small amount of test data. Other download locations are available, but "download.geofabrik.de" has a wide range of options. We'll download the data for Derbyshire (an English county), which is just over 20Mb.

Browse to http://download.geofabrik.de/europe/great-britain/england/derbyshire.html and note the "This file was last modified" date (e.g. "2016-08-26T19:29:02Z"). We'll need that later.

mkdir ~/data
cd ~/data
wget http://download.geofabrik.de/europe/great-britain/england/derbyshire-latest.osm.pbf

Run the following command:

osm2pgsql -d gis --create --slim -C 2500 --number-processes 1 -S ~/src/openstreetmap-carto-AJT/openstreetmap-carto.style --multi-geometry --tag-transform-script ~/src/SomeoneElse-style/style.lua ~/data/derbyshire-latest.osm.pbf

It's worth explaining a little bit about what those options mean:

-d gis The database to work with ("gis" used to be the default; now it must be specified).
--create Load data into an empty database rather than trying to append to an existing one.
--slim osm2pgsql can use different table layouts; "slim" tables works for rendering.
-C 2500 Allocate 2.5 Gb of memory to osm2pgsql to the import process. If you have less memory you could try a smaller number, and if the import process is killed because it runs out of memory you'll need to try a smaller number or a smaller OSM extract..
--number-processes 1 Use 1 CPU. If you have more cores available you can use more.
-S Create the database columns in this file (actually these are unchanged from "openstreetmap-carto")
--multi-geometry Describes how multipolygons are imported
--tag-transform-script Call this lua script to perform changes to data as it is loaded.

The final argument is the data file to load.

That command will complete with something like "Osm2pgsql took 238s overall".

Compiling the Stylesheet

cd ~/src/openstreetmap-carto-AJT
carto project.mml > mapnik.xml

Here "carto" converts the stylesheet that "project.mml" is the project file for (if you look at the file you can see that it refers to many other ".mss" files inside it) into a "mapnik.xml" file that the "mapnik" map-drawing software (but few humans) can understand.

Natural Earth Data

A number of the lower-zoom "openstreetmap-carto-AJT" layers use data from "Natural Earth"

cd ~/src/openstreetmap-carto-AJT
sh -x get-shapefiles.sh

This download will take a little while - time for another cup of tea.

The configuration file for renderd

The config file for "renderd" is "/usr/local/etc/renderd.conf". Edit that with a text editor such as nano:

sudo nano /usr/local/etc/renderd.conf

A couple of lines in here may need changing. In the "renderd" section:

num_threads=4

If you've only got 2Gb or so of memory you'll want to reduce this to 2.

The "ajt" section corresponds to a "named map style". You can have more than one of these sections if you want, provided that the URI is different for each.

XML=/home/ajtown/src/openstreetmap-carto-AJT/mapnik.xml

You'll want to change "ajtown" (the default in my version of the file) to "renderaccount" or whatever non-root username you used above.

URI=/hot/

I chose that so that the tiles generated here can more easily be used in place of the HOT tile layer that OpenStreetMap.org supports. You can use something else here, but "/hot/" is as good as anything.

Configuring Apache

Manually create the directory for generated tiles:

sudo mkdir /var/lib/mod_tile
sudo chown renderaccount /var/lib/mod_tile
sudo mkdir /var/run/renderd
sudo chown renderaccount /var/run/renderd

We now need to tell Apache about "mod_tile", so with nano (or another editor):

sudo nano /etc/apache2/conf-available/mod_tile.conf

Add the following line to that file:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

and save it, and then run:

sudo a2enconf mod_tile

That will say that you need to run "service apache2 reload" to activate the new configuration; we'll not do that just yet.

We now need to tell Apache about "renderd". With nano (or another editor):

sudo nano /etc/apache2/sites-available/000-default.conf

And add the following between the "ServerAdmin" and "DocumentRoot" lines:

       LoadTileConfigFile /usr/local/etc/renderd.conf
       ModTileRenderdSocketName /var/run/renderd/renderd.sock
       # Timeout before giving up for a tile to be rendered
       ModTileRequestTimeout 0
       # Timeout before giving up for a tile to be rendered that is otherwise missing
       ModTileMissingRequestTimeout 30

And reload apache twice:

sudo service apache2 reload
sudo service apache2 reload

(I suspect that it needs doing twice because Apache gets "confused" when reconfigured when running)

If you point a web browser at: http://yourserveripaddress/index.html you should get Ubuntu / apache's "It works!" page.

(if you don't know what IP address it will have been assigned you can likely use "ifconfig" to find out - if the network configuration is not too complicated it'll probably be the "inet addr" that is not "127.0.0.1"). If you're using a server at a hosting provider then it's likely that your server's internal address will be different to the external address that has been allocated to you, but that external IP address will have already been sent to you and it'll probably be the one that you're accessing the server on currently.

Note that this is just the "http" (port 80) site - you'll need to do a little bit more Apache configuration if you want to enable https, but that's out of the scope of these instructions. However, if you use "Let's Encrypt" to issue certificates then the process of setting that up can also configure the Apache HTTPS site as well.

Running renderd for the first time

Next, we'll run renderd to try and render some tiles. Initially we'll run it in the foreground so that we can see any errors as they occur:

renderd -f -c /usr/local/etc/renderd.conf

You should see some messages about fonts. Some of them will be "unable to find font" warnings - don't worry about those for now. You shouldn't get any errors. If you do, save the full output in a pastebin and ask a question about the problem somewhere like help.openstreetmap.org (linking to the pastebin - don't include all the text in the question).

Point a web browser at: http://yourserveripaddress/hot/0/0/0.png

You should see a map of the world in your browser and some more debug on the command line, including "DEBUG: START TILE" and "DEBUG: DONE TILE". Ignore any "DEBUG: Failed to read cmd on fd" message - it is not an error. If you don't get a tile and get other errors again save the full output in a pastebin and ask a question about the problem somewhere like help.openstreetmap.org.


If that all works, now is a good time for a backup.

Press control-c to stop the foreground rendering process.

Running renderd in the background

Next we'll set up "renderd" to run in the background. First, edit the "~/src/mod_tile/debian/renderd.init" file so that "RUNASUSER" is set to the non-root account that you have used before, such as "renderaccount".

sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd
sudo cp ~/src/mod_tile/debian/renderd.service /lib/systemd/system/

The "renderd.service" file is a "systemd" service file. The version used here just calls old-style init commands. In order to test that the start command works:

sudo /etc/init.d/renderd start

(that should reply with "[ ok ] Starting renderd (via systemctl): renderd.service.")

To make it start automatically every time:

sudo systemctl enable renderd

Viewing tiles

In order to see tiles, we'll cheat and add the "Switcheroo Redirector" extension to the Chrome (or Chromium) browser:

https://chrome.google.com/webstore/detail/switcheroo-redirector/cnmciclhnghalnpfhhleggldniplelbg?hl=en

We'll add "http://tile-a.openstreetmap.fr/hot/" as "From" and "http://yourserveripaddress/hot/" as "To", and do the same also for "tile-b" and "tile-c".

From an ssh connection do:

tail -f /var/log/syslog | grep " TILE "

That will show a line every time a tile is requested, and one every time rendering of one is completed.

In your switcheroo-configured Chrome / Chromium browser go to: http://www.openstreetmap.org/#map=19/53.19583/-1.77710

and switch to the "Humanitarian" layer in OSM (note "http" not https" - we haven't set https up yet). You should see some tile requests. Zoom out gradually. You'll see requests for new tiles show up in the ssh connection. Some low-zoom tiles may take a long time (several minutes) to render for the first time, but once done they'll be ready for the next time that they are needed.

Force Rendering Tiles

Low-zoom tiles include a larger area than higher-zoom ones and can take a long time to render. Once you're happy that rendering is working correctly and you can see the results, you will probably want to pre-render some tiles. "mod_tile" includes a program called "render_list" that can be used to request renderings of specific tiles. An example would be:

   render_list -n 1 -s /var/run/renderd/renderd.sock -z 0 -Z 10 -m ajt  -a

This says "Using 1 thread, please render all tiles from zoom level 0 to zoom level 10 in the 'ajt' named map style in renderd.conf". You can use the "-f" parameter to force the re-rendering of tiles that already exist. It will take a while, so it's suggested to use something like "screen" and disconnect from the session while it runs.

If you want to pre-render tiles in a particular area, do the following:

   cd ~/src
   git clone https://github.com/alx77/render_list_geo.pl
   cd render_list_geo.pl

You can call that with latitude and longitude parameters. As an example, here's a request to generate the z11 tiles for the UK:

   ./render_list_geo.pl -n 1 -z 11 -Z 11 -x -9.5 -X 2.72 -y 49.39 -Y 61.26 -m ajt

Updating your database as people edit OpenStreetMap

The next thing we want to do is to set up database updates based on people's updates to OpenStreetMap.

If that's all working OK, now's a good time for another backup, because the next section uses "osmosis", which uses Java. "osmosis" is a general-purpose OSM data utility, and one of the things that it can do is to update a database with recent changes from OSM.

sudo apt install osmosis

(that'll eventually say "done.")

We'll use "trim_osc.py" from Zverik's "regional" scripts to trim down the updates from OpenStreetMap.org down to just the area that we're interested in. We do this so that the postgres database doesn't grow significantly as updates are applied to it.

cd ~/src
git clone https://github.com/zverik/regional
chmod u+x ~/src/regional/trim_osc.py
sudo apt-get install python-psycopg2 python-shapely python-lxml

(answer yes when prompted)


Edit the tile expiry script with nano (or another editor):

sudo nano ~/src/mod_tile/openstreetmap-tiles-update-expire

This script has been modified from the parent repository's "expire" script to run from crontab You'll want to change "renderaccount" in that script to whatever non-root username you used above.

Look further down the file to where it says 'm_ok "filtering diff"'. The bounding box on the next line is " -b -14.17 48.85 2.12 61.27". That corresponds roughly to the UK; you may want to edit it to contain different values.


sudo mkdir /var/log/tiles
sudo chown renderaccount /var/log/tiles
~/src/mod_tile/openstreetmap-tiles-update-expire 2016-08-26T19:29:02Z

Note that that date corresponds to the age of the data that we loaded. That creates a directory ".osmosis" in "/var/lib/mod_tile" with details of the last imported data. The data passed here corresponds to the data of the file used to initialise the database, which we noted above. Again, you'll need to replace renderaccount with whatever non-root username you chose.


One one ssh session run "tail -f /var/log/tiles/run.log" One another run "~/src/mod_tile/openstreetmap-tiles-update-expire"

As that runs, you should see lines being written to "run.log" ending with "Done with import". The "replag" value shown is the replication lag; by default an hours-worth of data is taken in at a time. If this doesn't work properly check that "osmosis" is correctly installed and you can run it from the command line.

Run "~/src/mod_tile/openstreetmap-tiles-update-expire" again; now you'll see that "replag" is an hour less than it was.

Next, set up "openstreetmap-tiles-update-expire" in crontab to run every few minutes. Run "crontab -e". The first time you do you will be asked to choose an editor. Go down to the end of the file. Lines starting with "#" are comments. The last comment line is:

   # m h  dom mon dow   command

These fields are "minutes", "hours", "day of month", "month", "day of week" and "comand". We'll add the following line:

   */5  * *   *   *     /home/renderaccount/src/mod_tile/openstreetmap-tiles-update-expire

so that the script runs every 5 minutes. Replace "renderaccount" with your non-root username. Monitor the output with "tail -f /var/log/tiles/run.log" and wait 5 minutes.

Eventually it'll catch up, and new edits to OpenStreetMap will appear in your tiles too.

Potential Problems with updates

If an error such as "Expiry failed" appears in "/var/log/tiles/run.log" then the likely problem is that "render_expired", one of the programs called from within there, has run out of memory. If that is the case, reduce EXPIRY_MAXZOOM in the script until it works. You can get more details about what has gone wrong from the other files in "/var/log/tiles/"

Munin

Munin can be used for server monitoring, and was installed at the big "apt install" at the top of these instructions. "mod_tile" contains some munin plugins that can be used to monitor what the tile server is doing.

sudo apt install libcgi-fast-perl libapache2-mod-fcgid

If you look at /etc/apache2/conf-available you should see that munin.conf is a symbolic link to ../../munin/apache24.conf, which is /etc/munin/apache24.conf.

The file /etc/munin/apache24.conf is Apache's munin configuration file. In that file, if you want munin to be accessed globally rather than just locally change both instances of Require local to Require all granted.

Next edit /etc/munin/munin.conf. Uncomment these lines:

dbdir /var/lib/munin
htmldir /var/cache/munin/www
logdir /var/log/munin
rundir /var/run/munin

Note that the htmldir directory /var/cache/munin/www in there matches the entry in /etc/munin/apache24.conf, and /etc/munin/apache24.conf says it is available on that server as /munin.

Restart munin and apache:

sudo /etc/init.d/munin-node restart
sudo /etc/init.d/apache2 restart

Now http://yourserveripaddress/munin/ and e.g. http://yourserveripaddress/munin/disk-day.html should show something.

To add mod_tile's plugins to munin:

sudo ln -s /home/renderaccount/src/mod_tile/munin/* /etc/munin/plugins/

(replacing renderaccount with your chosen non-root account of course). Then run munin's cron job manually once:

sudo -u munin munin-cron

Restart munin and apache again:

sudo /etc/init.d/munin-node restart
sudo /etc/init.d/apache2 restart

After a short delay, refreshing http://yourserveripaddress/munin/ should show entries for mod_tile and renderd

You'll want "munin-cron" to run every 5 minutes. This will probably already have been set up in "/etc/cron.d/munin". If that isn't run from anywhere, then to add something to munin's crontab

sudo -i
sudo -u munin crontab -e

Add the line:

*/5 * * * * /usr/bin/munin-cron

to the end of the crontab. It will run that every 5 minutes.