User:SomeoneElse/Ubuntu 1604 tileserver load

From OpenStreetMap Wiki
Jump to: navigation, search

Introduction

This text describes what's required to take a "normal home PC" and to install a tile server on it. The "normal home PC" that I used was a Lenovo H30 (AMD A8-7410 with 1 socket, 4 cores, 8Gb memory, 1Tb of SATA-connected 7200 rpm disk). It runs Windows 10, but after a BIOS change it is possible to boot a different OS.

Note that I'm installing just a tile server - so I'm not going install a GUI or much other software.


Preparation

First, I installed Virtualbox after downloading from https://www.virtualbox.org/ .

If you need to control it remotely, you can do so by installing RemoteBox from http://remotebox.knobgoblin.org.uk/ .


In the "Oracle VM VirtualBox Manager" (or RemoteBox) change the default machine folder to "c:\Virtual_Machines".

Create a new guest.

I used a name of "UbuntuVM41"; you can use what you like.
OS Family: Linux
Version: Ubuntu (64-bit)
Memory: 3Gb

(if you've got less than 8Gb memory on the host machine you can use a lower value, such as 2Gb, here)

Create a new hard disk, 100Gb, VMDK format, dynamically allocated. Dynamically allocated means that the full 100Gb won't be used immediately. Following these notes only result in about around 6 Gb being used initially, though that will grow as the database is updated and tiles are rendered..

Ubuntu Installation

I downloaded Ubuntu 16.04.1 server from http://www.ubuntu.com/download/server . Using "Oracle VM VirtualBox Manager" go into "settings" and "storage", and below the IDE controller add the .iso file

Using "Oracle VM VirtualBox Manager" start the VM, and go through the Ubuntu Setup:

English, Install, English, United Kingdom
Detect Keyboard Layout: No, English UK, English UK
Hostname: ubuntuvm41
I set networking to bridged in "Network" to make it easier to access externally.
I chose "username" for the new user; you can use what you like.
I didn't encrypt my home directory.
Partitioning: Guided - Use entire disk
Write the changes to disk
I don't need configure a proxy for the package manager; you might.
I selected "Install Security updates automatically"
In software I left "standard system utilities" in place and added "Openssh server"
Install grub onto MBR (that's the MBR of the virtual machine, not the host PC, of course!)
Yes to reboot

I'll login via ssh remotely; you can also use the console which should just be displaying a "login:" prompt.

Firstly, install any pending ubuntu updates:

sudo -i
apt update
apt upgrade

That installs a new kernel, so:

sync;sync;shutdown -r now

Tile server pre-requisite software

After rebooting, ssh back in again and install the first set of prerequisites for a tile server:

sudo apt install libboost-all-dev git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libpng12-dev libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg node-carto

Say yes to install. This will take a while, so go and have a cup of tea. Then, install the second set of prerequisites:

sudo apt-get install postgresql postgresql-contrib postgis postgresql-9.5-postgis-2.2

Again, say yes to install.


Note - at any stage of this installation you might want to take a backup of the current situation before continuing. You can either shut the machine down and do a backup of the virtual machine's files (it'll be in an obvious directory in c:\virtual_machines), or you can take a "snapshot" as a temporary backup from either "Oracle VM VirtualBox Manager" or "RemoteBox" in case you need to restore it later.

Create and set up a database

We now want to do some configuration as the "postgres" user (which was created by the installation above). We want to set up the user that we created above as a postgres user, and create a new database with that user as the owner.

sudo -u postgres -i
createuser username
createdb -E UTF8 -O username gis

That command creates an empty database called "gis" (which is the default for many OSM tools), owned by "username".

While still working as the "postgres" user:

psql:

(that'll put you at a "postgres=#" prompt)

\c gis

(it'll answer 'You are now connected to database "gis" as user "postgres".')

CREATE EXTENSION postgis;

(it'll answer CREATE EXTENSION)

ALTER TABLE geometry_columns OWNER TO username;

(it'll answer ALTER TABLE)

ALTER TABLE spatial_ref_sys OWNER TO username;

(it'll answer ALTER TABLE)

\q

(it'll exit psql and go back to a normal Linux prompt)

exit

(to exit back to be the user that we were before we did "sudo -u postgres -i" above)

Install osm2pgsql

We will need install various bits of software from source. The first of this is "osm2pgsql":

mkdir ~/src
cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql

Various tools to import and manage OpenStreetMap data into a database exist. Here we'll use "osm2pgsql", which is probably the most popular.

The build mechanism used by osm2pgsql has changed since older versions, so we'll need to install some more prerequisites for that:

sudo apt install make cmake g++ libboost-dev libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 liblua5.2-dev

Again, say yes to install.

mkdir build && cd build
cmake ..

(the output from that should end with "build files have been written to...)

make

(the output from tha should finish with "[100%] Built target osm2pgsql")

sudo make install

Mapnik

Next, we'll install Mapnik. We'll use the default version in Ubuntu 16.04:

sudo apt-get install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libgdal1-dev libmapnik-dev mapnik-utils python-mapnik


We'll check that mapnik is installed OK:

python

(it'll reply with ">>>")

import mapnik

(if it replies again with just ">>>" - that's good)

Press control-d to exit the "python" prompt.

mod_tile and renderd

Next, we'll install mod_tile and renderd. "mod_tile" is an Apache module that handles requests for tiles; "renderd" is a daemon that actually renders tiles when "mod_tile" requests. We'll use https://github.com/SomeoneElseOSM/mod_tile, which is itself forked from https://github.com/openstreetmap/mod_tile, but modified so that it supports Ubuntu 16.04, and with a couple of other changes to work on a standard Ubuntu server rather than one of OSM's rendering servers.

cd ~/src
git clone git://github.com/SomeoneElseOSM/mod_tile.git
cd mod_tile
./autogen.sh

(that should finish with "autoreconf: Leaving directory `.'")

./configure

(that should finish with "config.status: executing libtool commands")

make

(that should finish with "make[1]: Leaving directory '/home/username/src/mod_tile'")

sudo make install

(that should finish with "make[1]: Leaving directory '/home/username/src/mod_tile'")

sudo make install-mod_tile

(that should finish with "chmod 644 /usr/lib/apache2/modules/mod_tile.so")

sudo ldconfig

(that shouldn't reply with anything)

Setting up a stylesheet

The next stage is setting up a stylesheet, so now's a good time to do a backup, using one of the methods described above.


The style we'll use here is based on and still similar to "openstreetmap-carto" (the "standard" style on the OpenStreetMap.org website), but is easier to configure and use and displays some features (such as offices, sidewalks, and English/Welsh "public footpaths") that openstreetmap-carto doesn't. There are two parts to it - a "lua" script which is called by osm2pgsql ("SomeoneElse-style") and the stylesheet itself ("openstreetmap-carto-AJT").

cd ~/src
git clone git://github.com/SomeoneElseOSM/SomeoneElse-style.git
git clone git://github.com/SomeoneElseOSM/openstreetmap-carto-AJT.git

Loading Data

Initially, we'll load only a small amount of test data. Other download locations are available, but "download.geofabrik.de" has a wide range of options. We'll download the data for Derbyshire (an English county), which is just over 20Mb.

Browse to http://download.geofabrik.de/europe/great-britain/england/derbyshire.html and note the "This file was last modified" date (e.g. "2016-08-26T19:29:02Z"). We'll need that later.

mkdir ~/data
cd ~/data
wget http://download.geofabrik.de/europe/great-britain/england/derbyshire-latest.osm.pbf

Run the following command:

osm2pgsql --create --slim -C 2500 --number-processes 1 -S ~/src/openstreetmap-carto-AJT/openstreetmap-carto.style --multi-geometry --tag-transform-script ~/src/SomeoneElse-style/style.lua ~/data/derbyshire-latest.osm.pbf

It's worth explaining a little bit about what those options mean:

--create Load data into an empty database rather than trying to append to an existing one.
--slim osm2pgsql can use different table layouts; "slim" tables works for rendering.
-C 2500 Allocate 2.5 Gb of memory to osm2pgsql to the import process. If you have less memory you could try "-C 1500".
--number-processes 1 Use 1 CPU
-S Create the database columns in this file (actually these are unchanged from "openstreetmap-carto")
--multi-geometry Describes how multipolygons are imported
--tag-transform-script Call this lua script to perform changes to data as it is loaded.

The final argument is the data file to load.

That command will complete with something like "Osm2pgsql took 238s overall".

Compiling the Stylesheet

cd ~/src/openstreetmap-carto-AJT
carto project.mml > mapnik.xml

Natural Earth Data

A number of the lower-zoom "openstreetmap-carto-AJT" layers use data from "Natural Earth"

cd ~/src/openstreetmap-carto-AJT
sh -x get-shapefiles.sh

This download will take a little while - time for another cup of tea.

The configuration file for renderd

The config file for "renderd" is "/usr/local/etc/renderd.conf". Edit that with a text editor such as nano:

sudo nano /usr/local/etc/renderd.conf

A couple of lines in here may need changing:

num_threads=4

If you've only got 2Gb or so of memory you'll want to reduce this to 2.

XML=/home/ajtown/src/openstreetmap-carto-AJT/mapnik.xml

You'll want to change "ajtown" to whatever non-root username you used above.

URI=/hot/

I chose that so that the tiles generated here can more easily be used in place of the HOT tile layer that OpenStreetMap.org supports. You can use something else here, but "/hot/" is as good as anything.

Configuring Apache

Manually create the directory for generated tiles:

sudo mkdir /var/lib/mod_tile
sudo chown username /var/lib/mod_tile"
sudo mkdir /var/run/renderd
sudo chown username /var/run/renderd

We now need to tell Apache about "mod_tile", so with nano (or another editor):

sudo nano /etc/apache2/conf-available/mod_tile.conf

Add the following line to that file:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

and save it.

sudo a2enconf mod_tile

That will say that you need to run "service apache2 reload" to activate the new configuration; we'll not do that just yet.

We now need to tell Apache about renderd. With nano (or another editor):

sudo nano /etc/apache2/sites-available/000-default.conf

And add the following between the "ServerAdmin" and "DocumentRoot" lines:

       LoadTileConfigFile /usr/local/etc/renderd.conf
       ModTileRenderdSocketName /var/run/renderd/renderd.sock
       # Timeout before giving up for a tile to be rendered
       ModTileRequestTimeout 0
       # Timeout before giving up for a tile to be rendered that is otherwise missing
       ModTileMissingRequestTimeout 30

And reload apache twice:

sudo service apache2 reload
sudo service apache2 reload

(I suspect that it needs doing twice because Apache gets "confused" when reconfigured when running)

If you point a web browser at: http://yourserveripaddress/index.html you should get Ubuntu / apache's "It works!" page.

(if you don't know what IP address it will have been assigned you can use "ifconfig" to find out - it'll be the "inet addr" that is not "127.0.0.1")

Note that this is just the "http" (port 80) site - you'll need to do a little bit more Apache configuration if you want to enable https, but that's out of the scope of these instructions.

Running renderd for the first time

Next, we'll run renderd to try and render some tiles. Initially we'll run it in the foreground so that we can see any errors as they occur:

renderd -f -c /usr/local/etc/renderd.conf

You should see some messages about fonts. Some of them will be "unable to find font" warnings - don't worry about those for now. You shouldn't get any errors. If you do, save the full output in a pastebin and ask a question about the problem somewhere like help.openstreetmap.org (linking to the pastebin - don't include all the text in the question).

Point a web browser at: http://yourserveripaddress/hot/0/0/0.png

You should see a map of the world and some more debug including "DEBUG: START TILE" and "DEBUG: DONE TILE". Ignore the "DEBUG: Failed to read cmd on fd" message - it is not an error. If you don't get a tile and get other errors again save the full output in a pastebin and ask a question about the problem somewhere like help.openstreetmap.org.


If that all works, now is a good time for a backup.

Press control-c to stop the foreground rendering process.

Running renderd in the background

Next we'll set up "renderd" to run in the background.

sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd
sudo cp ~/src/mod_tile/debian/renderd.service /lib/systemd/system/

The "renderd.service" file is a "systemd" service file. The version used here just calls old-style init commands. In order to test that the start command works:

sudo /etc/init.d/renderd start

(that should reply with "[ ok ] Starting renderd (via systemctl): renderd.service.")

To make it start automatically every time:

sudo systemctl enable renderd

Viewing tiles

In order to see tiles, we'll cheat and add the "Switcheroo Redirector" extension to the Chrome (or Chromium) browser:

https://chrome.google.com/webstore/detail/switcheroo-redirector/cnmciclhnghalnpfhhleggldniplelbg?hl=en

We'll add "http://tile-a.openstreetmap.fr/hot/" as "From" and "http://yourserveripaddress/hot/" as "To", and do the same also for "tile-b" and "tile-c".

From an ssh connection do:

tail -f /var/log/syslog | grep " TILE "

That will show a line every time a tile is requested, and one every time rendering of one is completed.

In your switcheroo-configured Chrome / Chromium browser go to: http://www.openstreetmap.org/#map=19/53.19583/-1.77710

and switch to the "Humanitarian" layer in OSM. You should see some tile requests. Zoom out gradually. You'll see requests for new tiles show in the ssh connection. Some low-zoom tiles may take a long time (several minutes) to render for the first time, but once done they'll be ready for the next time that they are needed.

Updating your database as people edit OpenStreetMap

The next thing we want to do is to set up database updates based on people's updates to OpenStreetMap.

If that's all working OK, now's a good time for another backup, because the next section uses "osmosis", which uses Java.

sudo apt install osmosis

(that'll eventually say "done.")

We'll use "trim_osc.py" from Zverik's "regional" scripts to trim down the updates from OpenStreetMap.org down to just the area that we're interested in. We do this so that the postgres database doesn't grow significantly as updates are applied to it.

cd ~/src
git clone https://github.com/zverik/regional
chmod u+x ~/src/regional/trim_osc.py
sudo apt-get install python-psycopg2 python-shapely python-lxml

(answer yes when prompted)


Edit the tile expiry script with nano (or another editor):

sudo nano ~/src/mod_tile/openstreetmap-tiles-update-expire

This script has been modified from the parent repository's "expire" script to run from crontab You'll want to change "ajtown" in that script to whatever non-root username you used above.

Look further down the file to where it says 'm_ok "filtering diff"'. The bounding box on the next line is " -b -14.17 48.85 2.12 61.27". That corresponds roughly to the UK; you may want to edit it to contain different values.


sudo mkdir /var/log/tiles
sudo chown username /var/log/tiles
~/src/mod_tile/openstreetmap-tiles-update-expire 2016-08-26T19:29:02Z

That creates a directory ".osmosis" in "/var/lib/mod_tile" with details of the last imported data. The data passed here corresponds to the data of the file used to initialise the database, which we noted above.


One one ssh session run "tail -f /var/log/tiles/run.log" One another run "~/src/mod_tile/openstreetmap-tiles-update-expire"

As that runs, you should see lines being written to "run.log" ending with "Done with import". The "replag" value shown is the replication lag; by default an hours-worth of data is taken in at a time.

Run "~/src/mod_tile/openstreetmap-tiles-update-expire" again; now you'll see that "replag" is an hour less than it was.

Next, set up "openstreetmap-tiles-update-expire" in crontab to run every few minutes. Run "crontab -e". The first time you do you will be asked to choose an editor. Go down to the end of the file. Lines starting with "#" are comments. The last comment line is:

   # m h  dom mon dow   command

These fields are "minutes", "hours", "day of month", "month", "day of week" and "comand". We'll add the following line:

   */5  * *   *   *     /home/username/src/mod_tile/openstreetmap-tiles-update-expire

so that the script runs every 5 minutes Monitor the output with "tail -f /var/log/tiles/run.log" and wait 5 minutes.

Eventually it'll catch up, and new edits to OpenStreetMap will appear in your tiles too.