User:Cate/Creating your own server on Gnu/Linux

From OpenStreetMap Wiki
Jump to navigation Jump to search

Creating your own OSM server is a good way to experiment the rendering, and deploying specialised slippy maps.

In this guide we will explain how to design and deploy a own public server on a GNU/Linux system, with a lot of comments. We try to explain and list different options, so that you can assemble and customise the server(s) according your own needs.

Design and overview

Basic design ideas

Our priorities on designing our osm server are::

  • security: the server should be secure (with many security layers), and it is intended to be used on public network;
  • scalability: the system should be capable of accepting further load in a easy way, e.g. by adding new servers, split workload/tasks, etc.;
  • manageability: every single subsystem should be independent and upgradable/replaceable easily, with clear separation of the tasks of every subsystems;
  • simple: the design should be as simple as possible, but without compromising the other priorities;
  • clean: we don't mess the system (it is not our business to put stuffs in /usr): we are playing fair together the distributions.

Overview of the components

An osm server has the following components:

Additionally, we implement extra data, at level of database/rendering, and at user level (openlayers):

  • contour data (SRTM) and import (e.g. ...);
  • user data.


We use only one machine, but the workload could be split (from the beginning or maybe later) into more machines, e.g.:

  • a machine for the databases, with PostgreSQL and PostGIS:
    • with a lot of disk space, few peaks on usage (on initial import, on rendering low level tiles), most of time it handles read-only queries);
    • PostgreSQL could be eventually used in clusters;
  • a machine for the initial imports and ev. for the updates of data:
    • an high end machine, with a lot of memory and power could speed-up the initial import of data;
  • a rendering machine (with Mapnik):
    • with tile cache;
  • one or more web servers:
    • serving static images.

Users and databases

We choose to separate the privilege, for extra security, but also to have a better overview of the different tasks and files.

The files, programs (not installed on system) and configurations is owned by osmdata, and stored in the osmdata home. It should be insulated from daemon interacting with outside data (webserver and rendering), with one exception in our case: raw data updates).

The rendering is done by an additional user: osmrender. This user should only write to cache tile and communicating with web server and other osmrender daemons, it needs only read-only access to the database. This user is similar to www-data/httpd.

For the initial setup we need access (or the sysadmin should set up things for us, according local policies):

  • postgres: to create and setup databases and the two database user;
  • webserver configuration, to set up the webserver;
  • root to create the two users: osmdata, osmrender and to setup the renderd deamon and the web-server;
  • a folder (e.g. /var/cache/osmrender/) to store tile cache, possibly outside backup paths.

For the access to the database, we use the same usernames:

  • osmdata: read and write to the databases;
  • osmrender: read-only access to the databases.

And we use the following databases:

  • osm, which contains the data from openstreetmap (and eventually additional data)
  • eventually contours and other databases

We use an additional copy of the each databases: osm0 and contours0 during import. Because import is very slow, and from time to time we rebuild the database from scratch (e.g. for adding new features), we uses the two databases in parallel (import and rendering), and we switch the import database only when the database is ready, so we have always a working version of the database online.

Planing the resources


  • September 2014, entire planet: 16 GB with a Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz:
    • nearly 3 days to import data
    • more than one week to create contours
    • 300 GB disk (without tile cache)

Which part of the planet

see Planet.osm


  • a lot of RAM and CPU power (especially for the initial import and updates).
  • RAID + LVM (highly recommended). Rebuilding regions requires a lot of resources, thus RAID could avoid days-long downtime.
  • a modern GNU/Linux distribution or a *BSD system.

Initial setup

Get the needed packages

Get the following packages (either installing from the distribution, or compiling them

  • PostgreSQL, version 9.0 or later (available on most GNU/Linux distributions, or check PostgreSQL downloads)
  • PostGIS, a version compatible with your PostgreSQL (available on many GNU/Linux distributions or check PostGIS install)
  • python, a version equal or later than 2.5.x but minor (for now) to the 3.0


  • We install separately many map specific packages, without requiring superuser (but for few package dependencies).

Preparing the setup

To allow copy and paste easily from this site, we put the some configuration as environment/shell variable in a shell file and we load it at the beginning of each session. If you are copying things manually, just remove the $ signs in the commands.

# run it typing ". ./" (it starts with a dot and a space)
# The users: we don't distinguish system user and database users
# paths
# databases
# an additional working copy during new full imports
# maximal zoom level of the rendering (used also to check what tiles need to be updated)
# python specific configuration (to allow local python modules):
export PYTHONPATH="$osmroot/lib/python2.7/site-packages"
# let's be sure that osmroot/bin in the PATH 
export PATH="$osmroot/bin:$PATH"
# and also the library path for the dynamic linker
export LD_LIBRARY_PATH="$osmroot/lib:$LD_LIBRARY_PATH"


  • We will move this file to $osmroot/conf, after creating the relevant user and directory.
  • The following code could help to find the ABI version of local python: python -c 'import sys; sys.stdout.write("\n".join(sys.path)+"\n")'. Alternatively you can check where mapnik puts the python module.

Setup the users

Ask the sysadmin (if the machine has one) and follow his/her recommendations. Our suggestion on a normal setup is:

# as su root (or using sudo)
adduser "$osmdata"
adduser --disabled-password --disabled-login --home "$tilecache" "$osmrender"
addgroup "$osmdata" "$osmrender"
# set the tilecache directory with relevant permissions
# (files writable also by the group, public access to the cache)
chmod u=rwx,g=rws,o=rx "$tilecache"

Setup also:

  • a backup;
  • a convenient way for selected regular users to log into $osmdata, e.g. with "su -osmdata" or ssh; or set umask 0002, and setgid to the home, so people in group $osmdata could change files;
  • disable remote login for all non real users;
  • the entire $osmroot could be cloned in the different machines.

Setup the $osmroot and osmrender

Now we make the directories in $osmroot ($osmdata home on standard setup)

# as user "$osmdata"
# Create directory structure
cd "$osmroot"
mkdir bin             # binaries go there
mkdir conf            # configuration and styles
mkdir conf/db         # the *.style files: what to import in the databases
mkdir conf/render     # rendering style
mkdir data            # data to be imported in the database
mkdir data/planet     # planet data from osm (country or region) and diffs
mkdir src             # various sources

Now you can copy the file into $osmroot/conf.

Also osmrender home $tilecache needs some setup:

cd "$tilecache"
ln -s "$osmroot"/bin bin
ln -s "$osmroot"/lib lib 
ln -s "$osmroot"/share share 

and at the bottom of .profile there should be:

# PATH="$HOME/bin:$PATH"  # this should be already here, else add it.
export PYTHONPATH="$HOME/lib/python2.7/site-packages"

Setup the database

See also the PostgreSQL documentation, in particular the server administration (of PostgreSQL) and the PostGIS documentation
See also Mapnik/PostGIS for some tuning hints

Currently we recommend PosgreSQL 9.1 with PostGIS. The PostgreSQL could be setup on a separate system, and eventually on a cluster. The entire planet requires about 300 GB of disk space (the contours need much more space, also depending the method used to display them).

We create one database per source, for now only the gis database (for osm data), and the two database users

  • osmdata with write permissions to the database
  • osmrender with read-only permission to the above database

Step 1: create users

# as database superuser in database machine, e.g. "su postgres" (as root)
# before to give the following commands:
# Note: answer no to all the questions (more secure, anyway after the initial setup
# we don't need to create further databases and roles)
createuser "$osmdata"
createuser "$osmrender"

Step 2: create main database

Important note: you need to install osm2pgsql sources before to proceed, in order to get the schema. See next section.

Create the main database and give them the GIS and OSM infrastructure to the $osm0 database:

# as database superuser in database machine, e.g. "su postgres" (as root) 
. ./conf/ 
dropdb "$osm0"
# now we create the new database
createdb --encoding=UTF8 --owner="$osmdata" "$osm0"
psql --dbname="$osm0" --command="CREATE EXTENSION postgis;"
psql --dbname="$osm0" --command="CREATE EXTENSION postgis_topology;"
psql --dbname="$osm0" --command="ALTER TABLE geometry_columns OWNER TO $osmdata; ALTER TABLE spatial_ref_sys OWNER TO $osmdata;"
psql --dbname="$osm0" --command="GRANT CONNECT ON DATABASE $osm0 TO $osmrender;"
psql --dbname="$osm0" --command="GRANT SELECT ON ALL TABLES IN SCHEMA public TO $osmrender;"


  • Adapt the paths if you have manually installed postgis. You may use "find /usr /opt/ ~ -name postgis.sql" to locate the relevant file. A common location is also /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql
  • For import we name the database $osm0, and when import is done successfully, we rename it as $osm, to be able to use database during import (e.g. when changing region size), executing (in psql) ALTER DATABASE $osm0 RENAME TO $osm
  • After executing su postgres you should source again the . ./conf/ in order to have the environment variable set also under the postgres user

Step 3: create contour database (optional)

Optionally, create the main database and give them the GIS and OSM infrastructure to the $contours database:

# as database superuser in database machine, e.g. "su postgres" (as root)
. ./conf/ 
dropdb "$contours0"
# now we create the new database
createdb --encoding=UTF8 --owner="$osmdata" "$contours0"
psql --dbname="$contours0" --command="CREATE EXTENSION postgis;"
psql --dbname="$contours0" --command="CREATE EXTENSION postgis_topology;"
psql --dbname="$contours0" --command="ALTER TABLE geometry_columns OWNER TO $osmdata; ALTER TABLE spatial_ref_sys OWNER TO $osmdata;"
psql --dbname="$contours0" --command="GRANT CONNECT ON DATABASE $contours0 TO $osmrender;"
psql --dbname="$contours0" --command="GRANT SELECT ON ALL TABLES IN SCHEMA public TO $osmrender;"

Step 4: Optimizing PostgreSQL

see the nice slides in for tips on how to optimise the system for a faster import


  • For the initial import, we will use a different configuration, in order to speed up the import.
  • Default configuration is also enough good.

Build programs from sources


See also building osm2pgsql from sources

Checkout the sources (we use git, which is a copy of original development in svn):

# Getting sources (first time):
cd "$osmroot/src"
git clone git://
# to update the sources
cd "$osmroot/src/osm2pgsql"
git pull && git checkout

Now we build the program:

cd "$osmroot/src/osm2pgsql"
./configure --prefix="$osmroot" --with-gnu-ld --with-protobuf-c=yes CFLAGS="-g -O2 -march=native" CXXFLAGS="-g -O2 -march=native"
make install


  • install the needed packages. If it miss a library (e.g. libfoo), probably you should install the developer package libfoo-dev
  • The README tell us about:
    • (on Debian and Ubuntu) aptitude install autoconf automake libtool make g++ libboost-dev libboost-system-dev libboost-filesystem-dev libboost-thread-dev libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev protobuf-c-compiler libprotobuf-c0-dev lua5.2 liblua5.2-dev
    • the package/command apt-file helps finding what package provide the required file.
    • (on Fedora and RedHat) yum install gcc-c++ boost-devel libxml2-devel geos-devel postgresql-devel bzip2-devel proj-devel protobuf-compiler
  • If you have errors (missing file) during build, try to clean the working directory, with git clean -d -x -f and redo the autogen/configure/make commands.


See also Mapnik Installation, Mapnik Installation Troubleshooting

Checkout the sources, 2.3.x branch:

cd "$osmroot/src"
git clone git://
# to update the sources
cd "$osmroot/src/mapnik"
git pull && git checkout 2.3.x

Now we build the program

cd "$osmroot/src/mapnik"
./configure PREFIX="$osmroot" PYTHON_PREFIX="$osmroot" OPTIMIZATION=3 INPUT_PLUGINS=all
make install


  • Now we install the 2.3.x branch. Master (3.x) is not yet well supported in mod_tiles and tirex
  • install the needed packages. If it miss a library (e.g. libfoo), probably you should install the developer package libfoo-dev
  • some required packages: scon, libboost-all-dev, libboost-program-options-dev, libboost-python-dev, libfreetype6-dev, libharfbuzz-dev
  • harfbuzz may require backport repository, or self compiling (see )
  • for python3, see:

Standard OSM Mapnik style, CartoCSS version

See Also: CartoCSS, openstreetmap-carto style sources, carto sources
Note: we use this package also to get the script

We download also the reference OSM Mapnik style. Note, since August 2013 openstreetmap uses styles in CartoCSS format, and than converted in Mapnik .xml file.

Checkout the sources of cartoCSS styles:

cd "$osmroot/src"
git clone
# to update the sources
cd "$osmroot/src/openstreetmap-carto"
git pull && git checkout

Checkout also the sources of carto (need to compile the styles)

cd "$osmroot/src"
git clone
# to update the sources
cd "$osmroot/src/carto"
git pull && git checkout
npm install


  • The following fonts are recommended (for rendering the entire world): ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts (see notes in
  • python-yaml is needed to regenerate the styles.
  • carto needs the package/program: npm
  • we run carto using nodejs ./bin/carto (on Debian system, until 2015). On some distributions you can skip nodejs: the program node is the required node.js program (not a hamradio program).

phyghtmap (for contours)

See also phyghtmap, downloading phyghtmap
Note: Contours are optional and there are some more possibilities to do contours. See Contour pages.

Checkout the sources:

cd "$osmroot/src"
mkdir phyghtmap
cd phyghtmap
tar xzf hyghtmap_1.45.orig.tar.gz
cd hyghtmap-1.45
python build
python install --prefix="$osmroot"


  • some required packages: python-pkg-resources, python-matplotlib, python-beatifulsoup, python-numpy

mod_tile / Tirex

See also mod_tile
See as alternative to renderd part:Tirex

Checkout of mod_tile sources:

cd "$osmroot/src"
git clone git://
# to update the sources
cd "$osmroot/src/mod_tile"
git pull && git checkout
Buildinf mod_tile
./configure --prefix="$osmroot" --with-gnu-ld CFLAGS="-g -O3 -march=native" CXXFLAGS="-g -O3 -march=native -I$osmroot/include" --with-libmapnik="$osmroot/bin/mapnik-config"
make install
mv "$osmroot/etc/renderd.conf" "$osmroot/conf/"
rmdir "$osmroot/etc" || true


  • -std=c++11 is needed, in CXXFLAGS, with new mapnik, in case you select mod_tile to renders also the tiles.
  • You can use make all-local because we are interested only on apache module mod_tile. Really it is not necessary: the mod_tile is build again at install time (!).
  • some required packages: apache2-dev libmemcached-dev libcairomm-1.0-dev libipc-sharelite-perl libjson-perl libgd-gd2-perl libwww-perl libcxxtools-dev librados-dev

As superuser / sudo:

cd "$osmroot/src"
make install-mod_tile
mkdir /var/lib/mod_tile
chown www-data.www-data /var/lib/mod_tile
cp debian/tile.load /etc/apache2/mods-available/
a2enmod tile
cp debian/renderd.init /etc/init.d/renderd
update-rc.d renderd default

Then edit /etc/init.d/renderd, changing the following lines (adding the /home/osmdata paths):


and inserting the following line, just after RUNASUSER=:

export LD_LIBRARY_PATH="/home/osmdata/lib:$LD_LIBRARY_PATH"

We start the deamon later, when we have data in the database.

Import of OSM data into the database

Choose the data source and download data

Choose the right planet.osm file (check the link: "planet.osm").


  • Prefer the PBF format.
  • Verify that your source has the needed (to the required interval) change files.
  • Prefer downloading a file with the date (and not the generic alias). It help to trace when the snapshot was taken, and thus what changeset you need to apply
  • For the first try, download only a small region. Later, import the new data, without the "-a/--append" flag, will reset the old data before the new import.

We download the planet.osm (the wished region) in $osmroot/data/planet; e.g. planet-130130.osm.pbf in this example (file size: 18.9 GB, 30 January 2013)

If you want regular updates, you should set up the cronjob to download and update the database.

Import style (filter what to import)

See also import style (Osm2pgsql)

Copy the database "style" to the configuration and eventually edit it: remove what you will not use in Mapnik, in order to have a small database (and quicker import).

cp $osmroot/src/openstreetmap-carto/ $osmroot/conf/db/

In standard installation we didn't change the file.


  • Osm2pgsql has also an identical file (named

Initial import of OSM data

see also osm2pgsql optimization

Import the data:

osm2pgsql --create --input-reader pbf --slim --database "$osm0" --cache 12000 --number-processes 2 \
  --style "$osmroot/conf/db/" --flat-nodes=/path/to/flat_nodes"  "$osmroot/data/planet/planet-130130.osm.pbf" 


  • Use 75% of memory for --cache (numbers in MB).
  • Use --host, --port, --username and --password if you have a remote database or if the access to the database requires credentials.
  • Ev. you should add --port to select the right postgresql engine (also if you use the UNIX socket: the socket name depends on port). Run pg_lsclusters to find the port number
  • --flat-nodes is optional, but it could make import a lot faster, and decrease the disk usage. Put it in a osmdata writable directory, possibly on a fast disk. (the file need not to exists at the beginning))
  • At this phase we doesn't need the expire list (but if you are rebuilding the database). But we can store them anyway adding e.g.:
    --expire-tiles="$maxzoom" --expire-output "$osmroot/data/planet/planet-130130.osm.pbf.expired"
  • Use screen in order to detach the terminal: import could take days/weeks.
  • For the first try, download only a small region. Later, import the new data, without the "-a/--append" flag, will reset the old data before the new import.
  • With the above configuration, on a 16GB machine, world import as January 2013, it took 12 days and 15 hours to import all, 7 days for the first part:
    • Processing: Node(1755043k 156.4k/s) Way(167918k 0.46k/s) Relation(1782000 8.08/s) parse time: 597177s)
    most of the time it is used for disk IO access, so no need to increase much the --number-processes.
  • Keep (and adapt) two different postgresql configuration: one for the server running, one for the initial import, consider to:
    Increase: shared_buffers and maintenance_work_mem for both configurations
    Set: autovacuum = off, fsync=off, synchronous_commit=off for the initial import configuration
    Check also Osm2pgsql/benchmarks for other tips on different configurations.

Getting world boundaries and costlines, and styles

The OSM Mapnik style requires few other data, e.g. world boundaries for minimal zoom display, coastlines, etc. Now openstreetmap-carto package distribute a script to automate this step:

cd "$osmroot/data"


  • Shapefiles are nearly 500MB in download size.
  • This script already incorporate the ogr2ogr fix, and it also generate the shapeindex.

Generating the mapnik .xml style:

  • Edit or/and copy the file in conf directory.
  • in project.yaml you should change the dbname from gis to osm.
cd "$osmroot/src/openstreetmap-carto/"
scripts/ < project.yaml > osm.mml
nodejs "$osmroot/src/carto/bin/carto" "$osmroot/src/openstreetmap-carto/osm.mml" > "$osmroot/conf/osm.xml"
# links conf to data
cd "$osmroot/conf"
ln -s "$osmroot/data" data
ln -s "$osmroot/src/openstreetmap-carto/symbols" symbols


  • Adapt style to your needs.
  • Alternate style lists: , List of CartoCSS projects
  • we run carto using nodejs ./bin/carto (on Debian system, until 2015). On some distributions you can skip nodejs: the program node is the required node.js program (not a hamradio program).
  • Instead of linking data directory (the mml file, and thus the xml file point to that), you can edit the mml file, and provide the full path of the files

Creating and importing contours

I use phyghtmap instead of the shapefiles approach (gdal_contour). The shapefile approach is a lot slower, and it has some artifacts on the degree boundaries (at degree line there are erroneous horizontal and vertical lines). Additionally phyghtmap could use also other sources to create contours above 60 degrees, and where srtm lacks data.

Step 1: create the osm2psql style

I create the $osmroot/conf/db/ , setting them inimal data to store in contours database:

node,way   note         text         delete   # These tags can be long but are useless for rendering
node,way   source       text         delete   # This indicates that we shouldn't store them
node,way   created_by   text         delete
node,way   ele          text         linear   # contours are ways
node,way   contour_ext  text         linear
node,way   contour      text         delete

Step 2: download and create the contours

Then I create the contours from srtm data (putting hgt file in $osmroot/data/hgt as cache)

# create the cache (of the file automatically downloaded by phyghtmap
mkdir "$osmroot/data/hgt"
# destination of the resulting .osm.pbf files:
mkdir "$osmroot/data/hgt/pbf"
# Generate the .osm.pbf files
cd "$osmroot/data/hgt/pbf"
phyghtmap -a -180:-60:180:61 --step=20 --line-cat=500,100 --jobs=4 --pbf --source=srtm3 --hgtdir="$osmrootdata/hgt/" --start-node-id=1000 --start-way-id=1000
# import the contours in the database


  • It take less that 1 day to download hgt files (arounf 140000) and generate the .osm.pbf files (around 25000 files)
  • Adapt --step=20 --line-cat=500,100 if you need contours at other level (here: 20m for 'contour_minor, 100m for contour_medium and 500m for contour_major). Optimize the two larger categories for large tiles (lower zoom). You can in any case differentiate any contour line at any step multiple.
  • Adapt also -a -180:-60:180:61 to include only the needed part of the world.
  • Check also the additional sources, in order to include the missing data (e.g. in mountains, in most northern and southern latitudes, etc.), and the permission to use such sources for your specific project.

Step 3: import the contours

# initially import one (random) file, to create the tables.
cd "$osmroot/data/hgt/pbf"
ls | grep '\.osm\.pbf$' | head -1 | xargs osm2pgsql --create --input-reader pbf --database "$contours0" --cache 8192 --number-processes 3 --style "$osmroot/conf/db/"
# now we import of the pieces
find . -name '*.osm.pbf' | xargs -n 10 osm2pgsql --append --input-reader pbf --database "$contours0" --cache 8192 --number-processes 3 --style "$osmroot/conf/db/"


Test of manual rendering

First we test the installation:

cd "$osmroot/src/mapnik-stylesheets"

It should deliver (after some time) a map of the United Kingdom as image.png (24 MB). You should do this test every time you change the rendering configuration.


  • Did you use regularly . ~/conf/ The OSM style setup don't warn about missing names. Also before to run you need to source that file (in order to load the correct library and python module)
  • Check PostgreSQL configuration and method to authenticate to the database.

Customizing osm.xml

See also: using Mapnik with OpenStreetMap data, Mapnik

This depend on project. We work directly on the original git repository, doing there own branche, 3-way merge, etc.

cd "$osmroot/conf"
ln -s "$osmroot/src/mapnik-stylesheets" "stylesheet" 

Note: You can see some styles in action, in and their sources in

Rendering contours

See Phyghtmap/CartoCSS

Configuring the rendering engine and the web-server

Unfortunately now only Apache is is supported (and not e.g. lighttpd) by mod_tile (and also tirex), without good reasons. I'm working to implement a good and clean solution.

In /etc/renderd.conf I put:

tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET



Now you are ready to start the renderd deamon:

/etc/init.d/renderd restart

(it will automatically started at every reboot).

In /etc/apache2/sites-enabled/osmdata.conf I put:

<VirtualHost *:80>
   LoadTileConfigFile /etc/renderd.conf
   ModTileRenderdSocketName /var/run/renderd/renderd.sock
   # Timeout before giving up for a tile to be rendered
   ModTileRequestTimeout 15
   # Timeout before giving up for a tile to be rendered that is otherwise missing
   ModTileMissingRequestTimeout 30
<VirtualHost *:80>
   DocumentRoot /var/www/map
   <Directory /var/www/map>
       Options -Indexes FollowSymLinks MultiViews
       AllowOverride None
       Order allow,deny
       allow from all
   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined

in /etc/apache2/conf.d/mod_tile

LoadModule tile_module /usr/lib/apache2/modules/

Note: Yet make install-mod_tile install the module in /usr/lib... and not in /usr/local/lib.

Now we are ready to restart the webserver with the new configuration:

/etc/init.d/apache2 restart


and a lot of testing, research, code sources and documentation.