User:SomeoneElse/Ubuntu 1404 tileserver

From OpenStreetMap Wiki
Jump to: navigation, search

I recently installed a small (2Gb memory in a virtual machine) tile server on Ubuntu 14.04. I intend to use it for serving maps of a slightly customised "standard" style covering the middle of England (Oxford to Cleveland), but initially I just installed the Geofabrik extract for "Derbyshire". This essentially is what I did.

I've changed the actual IP address that I used to "myserveraddress" in what follows, my user name to "myusername", and my server name to "myservername".

It's based on the previous "switch2osm" instructions:

http://switch2osm.org/serving-tiles/manually-building-a-tile-server-12-04/

and incorporates a number of updates to that (mostly from pnorman). Thanks to everyone who's contributed directly or indirectly - almost none of what follows is my own work.

The idea was to rely as much as possible on Ubuntu repositories rather than PPAs, and to build the OSM-specific stuff from source (just to reduce external dependancies). The one PPA that was needed was for "nodejs" (see below). If you try and follow these notes on a server with other PPAs in place (such as might happen if you've installed TileMill) things probably won't work exactly as described below, and you might end up with things in different places.


Server installation

I downloaded and installed Ubuntu 14.04 LTS from here: http://www.ubuntu.com/download/server

Of the options available at installation, I selected only "OpenSSH server". I also installed "emacs23-nox" to use as an editor. That brings down quite a lot - if you install a different main editor you may see other dependancies lower down.

After installation, just to make sure that it was up to date I then did:

  • sudo -i

(all the apt-get commands need to be run as root)

  • apt-get update
  • apt-get upgrade


Packaged Software

Next, I installed the minimal set of software required by a "switch2osm-style" server:

  • apt-get --no-install-recommends install -y python-software-properties git unzip curl build-essential


I installed Postgres from Ubuntu's repositories:

  • apt-get install postgresql-9.3-postgis-2.1 postgresql-contrib-9.3 proj-bin


Ubuntu 14.04 contains Apache 2.4, to install that I did:

  • apt-get install apache2 apache2-mpm-worker
  • apt-get install munin-node munin munin-plugins-extra libdbd-pg-perl sysstat iotop ptop
  • sed -i "s|Allow from.*|Allow from all|" /etc/munin/apache.conf
  • service apache2 reload


Next, because I was installing some things from source I needed various development packages. This list is consolidated from what I found that I needed as I went along:

  • sudo apt-get install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libgdal1-dev mapnik-utils python-mapnik libmapnik-dev


And some things that osm2pgsql will need now (protobuf, to handle .pbf files) or that I'll need later (lua, because at some stage in the future I'll reload data with https://github.com/SomeoneElseOSM/designation-style).

  • sudo apt-get install libprotobuf-c0-dev protobuf-c-compiler lua5.2 liblua5.2-dev

Software built from source

The next thing that I did was to get osm2pgsql from source.


Checking versions

myusername@myservername:~/src/osm2pgsql$

  • proj
Rel. 4.8.0, 6 March 2012
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
  • psql --version
psql (PostgreSQL) 9.3.4
  • grep 'default_version' '/usr/share/postgresql/9.3/extension/postgis.control'
default_version = '2.1.2'
  • osm2pgsql --version
osm2pgsql SVN version 0.85.0 (64bit id space)


Although probably not needed for such as small OSM data extract, I then set up memory overcommit as per previous instructions: Edit:

  • /etc/sysctl.d/60-overcommit.conf

Adding:

# Overcommit settings to allow faster osm2pgsql imports 
vm.overcommit_memory=1

Then:

  • sudo sysctl -p /etc/sysctl.d/60-overcommit.conf

Create a database:

From

myusername@myservername:~/src/osm2pgsql$

I did:

  • sudo -u postgres createuser -s $USER
  • createdb gis
  • psql -d gis -c 'CREATE EXTENSION hstore; CREATE EXTENSION postgis;'

which just outputs:

CREATE EXTENSION

then:

  • sudo munin-node-configure --sh | sudo sh
  • sudo service munin-node restart

Get the OSM stylesheet:

(I did this from "~", but it might be better done from somewhere like "~/data", where I put the OSM data extract below)

Which returns:

Cloning into 'openstreetmap-carto'...
remote: Reusing existing pack: 1735, done.
remote: Total 1735 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1735/1735), 712.09 KiB | 411.00 KiB/s, done.
Resolving deltas: 100% (860/860), done.
Checking connectivity... done.

Fonts needed by the OSM stylesheet:

There's a section of the OSM stylesheet installation guide that talks about the fonts that are needed:

https://github.com/gravitystorm/openstreetmap-carto/blob/d778979732b8d775de77ff4c3b663ed25dc0b981/INSTALL.md

DejaVu Sans and Unifont are the two required fonts. The others are only needed if you actually expect to need to display those fonts - if you don't, you'll just see something like "square characters" in names that need those fonts to display correctly.

Loading an OSM data extract.

Initially I just used a smaller county extract from Geofabrik:


I loaded the data (the path reference to openstreetmap-carto.style may need to be corrected to the actual location of the "openstreetmap-carto.style" file):

  • osm2pgsql --create --slim -C 1500 --number-processes 1 -S ../openstreetmap-carto/openstreetmap-carto.style --hstore --multi-geometry ./derbyshire-latest.osm.pbf

Things that need nodejs

(I did this section from the "myusername@myservername:~/data$ " directory)

I initially tried to use the Ubuntu-packaged nodejs, but that didn't work, so I then did:

  • sudo add-apt-repository -y ppa:chris-lea/node.js
  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt-get install nodejs
  • npm install npm
  • npm install millstone carto


That got me:

  • ./node_modules/carto/bin/carto --version
carto 0.9.6 (Carto map stylesheet compiler)
  • cd ~/openstreetmap-carto

(which would have been ~/data/openstreetmap-carto if I'd put it where I intended to)

  • ./get-shapefiles.sh

'"renderd" and "mod_tile"'

The "autogen" below may have problems if it's run on a slightly different system, or if the packages that make up Ubuntu have changed between me writing and you reading this, or if mod_tile has changed (all of which are possible). If something goes wrong, drop me a message, or try the usual help channels (#osm and #osm-dev on IRC, the OSM help site, etc.).


Next, I edited

/usr/local/etc/renderd.conf

and changed

plugins_dir=/usr/lib/mapnik/input

to

plugins_dir=/usr/lib/mapnik/2.2/input


I installed the command-line "TileMill to Mapnik stylesheet converter" carto:

  • sudo npm install -g carto


and checked that it ran OK:

  • type carto
carto is /usr/bin/carto


I changed directory to the location of project.mml (which in my case was):

  • cd ~/openstreetmap-carto/

and ran

  • carto project.mml > mapnik.xml


Next I edited renderd.conf so that it contains the path to this:

  • XML=/home/myusername/openstreetmap-carto/mapnik.xml

(and also I set HOST=localhost in there)


I manually created the directory for generated tiles:

  • sudo mkdir /var/lib/mod_tile
  • sudo chown myusername /var/lib/mod_tile


and also the directory for the socket used to communicate from the apache module (that realises that it needs a tile) to renderd (that actually renders it):

  • sudo mkdir /var/run/renderd
  • sudo chown myusername /var/run/renderd

This needs creating on every reboot, but is looked after by the "init" script once that is set up. See below for that. It was done manually here because I initially ran renderd in debug mode from the shell.

Apache setup

In Apache 2.4, "conf-available" and "conf-enabled" have replaced "conf.d".

I created

  • /etc/apache2/conf-available/mod_tile.conf

containing

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

and then did:

  • sudo a2enconf mod_tile
Enabling conf mod_tile.
To activate the new configuration, you need to run:
 service apache2 reload

which created a symlink from "conf-enabled". The "disable" script to match "a2enconf" is "a2disconf".

Note that renderd won't actually run yet, because the config info it needs hasn't been set up yet.


In "/etc/apache2/sites-available/000-default.conf" after the server email address line I added:

  • LoadTileConfigFile /usr/local/etc/renderd.conf
  • ModTileRenderdSocketName /var/run/renderd/renderd.sock
  • # Timeout before giving up for a tile to be rendered
  • ModTileRequestTimeout 0
  • # Timeout before giving up for a tile to be rendered that is otherwise missing
  • ModTileMissingRequestTimeout 30


and ran:

  • sudo service apache2 reload

This isn't ideal, because if you run "a2disconf mod_tile" a "service apache2 reload" will have problems because the "000-default.conf" has errors in it, because "LoadTileConfigFile" is defined by the disabled "mod_tile" config. The error is something like this:

  • sudo a2disconf mod_tile

Conf mod_tile disabled. To activate the new configuration, you need to run:

 service apache2 reload
  • sudo service apache2 reload
Reloading web server apache2                                                  *
 The apache2 configtest failed. Not doing anything.
Output of config test was:
 AH00526: Syntax error on line 13 of /etc/apache2/sites-enabled/000-default.conf:
 Invalid command 'LoadTileConfigFile', perhaps misspelled or defined by a module not included in the server configuration
 Action 'configtest' failed.
 The Apache error log may have more information.


I'm sure that there's a proper Apache 2.4 way of fixing this; I haven't found it yet, and as I have no intention of running the web server without mod_tile, it's not a requirement for me.

At this point I had a look at the end of /var/log/apache2/error.log to make sure that nothing untoward was happening.

[Sat Apr 26 12:31:37.626116 2014] [mpm_event:notice] [pid 1182:tid 140453313202048] AH00493: SIGUSR1 received.  Doing graceful restart
[Sat Apr 26 12:31:37.713271 2014] [tile:notice] [pid 1182:tid 140453313202048] Loading tile config default at /osm_tiles/ for zooms 0 - 20 from tile directory /var/lib/mod_tile with extension .png and mime type image/png
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[Sat Apr 26 12:31:37.727167 2014] [mpm_event:notice] [pid 1182:tid 140453313202048] AH00489: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Sat Apr 26 12:31:37.728041 2014] [core:notice] [pid 1182:tid 140453313202048] AH00094: Command line: '/usr/sbin/apache2'

Test renderd

I then ran renderd interactively from the command line:

  • renderd -f -c /usr/local/etc/renderd.conf

(no obvious errors occurred)


And elsewhere I browsed to

and a tile was indeed visible.

Running renderd in the background

Next I set up renderd to run in the background from init:

  • sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
  • sudo chmod u+x /etc/init.d/renderd


I edited renderd.init to contain:

  • DAEMON=/usr/local/bin/$NAME
  • DAEMON_ARGS="-c /usr/local/etc/renderd.conf"

and in that file I also changed "www-data" to "myusername" above.


When I ran:

  • sudo /etc/init.d/renderd start
* Starting Mapnik rendering daemon renderd                                     iniparser: syntax error in /usr/local/etc/renderd.conf (7):
-> ;[renderd01]
iniparser: syntax error in /usr/local/etc/renderd.conf (14):
-> ;[renderd02]
iniparser: syntax error in /usr/local/etc/renderd.conf (33):
-> ;** config options used by mod_tile, but not renderd **
iniparser: syntax error in /usr/local/etc/renderd.conf (44):
-> ;[style2]
iniparser: syntax error in /usr/local/etc/renderd.conf (51):
-> ;** config options used by mod_tile, but not renderd **
                                                                        [ OK ]

(apart from objecting to semicolons) it starts OK.


I then rebooted to check that everything restarted OK - and it did.