Openptmap/Installation

From OpenStreetMap Wiki

Jump to: navigation, search

This page describes how to install the software you need for creating a map like openptmap.de (see also openptmap Wiki page). The advanced approach, so-called diff-based rendering is presently being tested on a second server: openptmap.org.

1 Prerequisites
- 1.1 Hardware
- 1.2 Operating System
2 Prepare your System
3 Create the Map
- 3.1 Fill the Database
- 3.2 Render the Tiles
4 Test your new Map
5 Create the Map Automatically
- 5.1 Weekly Map Data Update
- 5.2 Daily Map Data Update
6 Using a Rendering Strategy
7 An Advanced Approach: Diff-based Rendering
- 7.1 Some Details to the Alternative Strategy
- 7.2 How-To Implement the Alternative Strategy
8 Further Strategies
9 Benchmarks
10 Troubleshooting

Prerequisites

Hardware

A weak CPU, e.g. Intel Atom or a virtual Internet server, will suffice if you limit the geographical region (e.g. France, Germany, Australia). 1 GB RAM is recommended, but 768 MB will be enough.

Operating System

We assume that you use Ubuntu >=8.04 as operating system. All of the following steps have been tested with version 10.04, most have been tested with 8.04 too. Meanwhile, short term versions >=10.10 are available. There, the procedure has not been tested yet, but it should work as well.

Prepare your System

This chapter describes all steps which are to be done once. After having run through this work, you will only need to update the OSM data for your map from time to time.

Create a User

For all openptmap purposes you can use your usual user account if you like. However, it is recommended to create a separate user for that. In this example, we decide to create a user account with the name pt. Please log in with a user with administrator rights and enter the following commands:

sudo adduser pt
sudo usermod -aG admin pt

You will have noticed that we gave admin rights to user pt. These rights should be removed after the whole installation process has been completed: sudo deluser pt admin Now logoff, then login with the new user name pt.

Debian Packages

You must install all packages necessary to run your own web server. Additionally, you will need some other packages. Ensure the installation of all necessary packages by performing these commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install apache2 php5 php5-pgsql libapache2-mod-php5 subversion autoconf autoconf2.13 unzip
sudo apt-get install automake build-essential libxml2-dev libgeos-dev libbz2-dev proj libtool
# for Ubuntu <=9.10:
  sudo apt-get install postgresql-8.3-postgis postgresql-contrib-8.3 postgresql-server-dev-8.3
# for Ubuntu >=10.04:
  sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4 postgresql-server-dev-8.4

PostgreSQL Database

The GIS database must be intitialized. Please follow these steps:

sudo -u postgres -i -H
createuser -SdR ptuser
createdb -E UTF8 -O ptuser ptgis
createlang plpgsql ptgis
# for Ubuntu <=9.10:
  psql -d ptgis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
# for Ubuntu >=10.04:
  psql -d ptgis -f /usr/share/postgresql/8.4/contrib/postgis.sql
# for Ubuntu 11.04:
  psql -d ptgis -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql ptgis -c "ALTER TABLE geometry_columns OWNER TO ptuser"
psql ptgis -c "ALTER TABLE spatial_ref_sys OWNER TO ptuser"
exit

To enable easy database login by user ptuser you must change some lines in one of the database configuration files. In case you are running Ubuntu with a graphical interface, you could a more comfortable editor, e.g. gedit instead of nano.

# for Ubuntu <=9.10:
  sudo nano /etc/postgresql/8.3/main/pg_hba.conf
# for Ubuntu >=10.04:
  sudo nano /etc/postgresql/8.4/main/pg_hba.conf

Near to the bottom of the file you will find these lines:

local   all         all                               ident
host    all         all         127.0.0.1/32          md5
host    all         all         ::1/128               md5

Change the words ident and md5 each to trust and close the editor (for nano: Ctrl-O, Enter, Ctrl-X). Now reload the database configuration:

# for Ubuntu <=9.10:
  sudo /etc/init.d/postgresql-8.3 reload
# for Ubuntu >=10.04:
  sudo /etc/init.d/postgresql-8.4 reload

For a short test, login to the database by using the previously created database user ptuser:

psql ptgis ptuser

Type \d to see a list with the two tables which we have created some steps above (geometry_columns and spatial_ref_sys). Then logout with \q

If you encounter any problems, you may find a solution here: PostGIS.

Choose a Directory

Our example directory for downloading OSM data and generating the data base contents will be the home directory of the user pt, "/home/pt", which can be abbreviated as "~" if being logged in with this user.

If you choose a different directory, please create it and grant read and write access rights to the user pt. You also will have to replace all cd /home/pt commands in the following examples with cd and the full path to your alternative directory.

Tool osm2pgsql

We will need the tool osm2pgsql to write OSM data into the database. Although provided by Ubuntu community, it's strongly recommended to build osm2pgsql from source, because we rely on getting the newest version. Please follow these steps:

cd /home/pt
svn export http://svn.openstreetmap.org/applications/utils/export/osm2pgsql
cd osm2pgsql
./autogen.sh
./configure
make

We need to copy and extend the file "default.style" which should be located in the directory /home/pt/osm2pgsql. You can do this with an editor or by executing these commands:

cd /home/pt
cp osm2pgsql/default.style osm2pgsql_pt.style
cat <<eof >> osm2pgsql_pt.style

# extensions for openptmap:
node,way line text linear
node,way ref_name text linear
node,way uic_ref text linear
node,way public_transport text linear
node,way train text linear
node,way wheelchair text linear
node,way website text linear
eof

Now the database needs to be initialized for Spherical Mercator projection:

cd /home/pt
psql ptgis ptuser -f osm2pgsql/900913.sql

If you encounter any problems, you may find a solution here: osm2pgsql.

Tools osmconvert and osmfilter

Both tools are used to accelerate the PostgreSQL import process. If we filter out all information we want to use and drop everything else, osm2pgsql will run faster. On top of this, database queries will be faster too, so the rendering process will be accelerated as well. You can download the programs as binary, however, it is recommended to download and compile the source code because the binary may be out of date. To do this – downloading and compiling – from the command line, enter these commands:

cd /home/pt
wget -O - http://m.m.i24.cc/osmconvert.c |cc -x c - -lz -o osmconvert
wget -O - http://m.m.i24.cc/osmfilter.c |cc -x c - -o osmfilter

For further details on both tools refer to osmconvert and osmfilter.

Mapnik Renderer

First, check if your package sources offer Mapnik version 0.7.1:

apt-cache show python-mapnik

If yes, install Mapnik package:

sudo apt-get install python-mapnik

If there is no Mapnik package available or it has a lower version number, you must install Mapnik from source. There is an installation guide which will help you: Ubuntu Installation Guide for Mapnik

For example the procedure for Ubuntu 8.04:

cd /home/pt
sudo apt-get purge python-mapnik
sudo apt-get autoremove
# boost dependencies
sudo apt-get install binutils cpp-3.3 g++-3.3 gcc-3.3 gcc-3.3-base libboost-dev libboost-filesystem-dev libboost-filesystem1.34.1 libboost-iostreams-dev libboost-iostreams1.34.1 libboost-program-options-dev libboost-program-options1.34.1 libboost-python-dev libboost-python1.34.1 libboost-regex-dev libboost-regex1.34.1 libboost-serialization-dev libboost-serialization1.34.1 libboost-thread-dev libboost-thread1.34.1 libicu-dev libicu38 libstdc++5 libstdc++5-3.3-dev python2.5-dev 
# remaining required dependencies
sudo apt-get install libfreetype6 libfreetype6-dev libjpeg62 libjpeg62-dev libltdl3 libltdl3-dev libpng12-0 libpng12-dev libtiff4 libtiff4-dev libtiffxx0c2 python-imaging python-imaging-dbg proj 
# optional Cairo Renderer dependencies
sudo apt-get install libcairo2 libcairo2-dev python-cairo python-cairo-dev libcairomm-1.0-1 libcairomm-1.0-dev libglib2.0-0 libpixman-1-0 libpixman-1-dev libpthread-stubs0 libpthread-stubs0-dev ttf-dejavu ttf-dejavu-core ttf-dejavu-extra
# all Optional GIS utilities
sudo apt-get install libgdal-dev python2.5-gdal postgresql-8.3-postgis postgresql-8.3 postgresql-server-dev-8.3 postgresql-contrib-8.3
# WMS Dependencies
sudo apt-get install libxslt1.1 libxslt1-dev libxml2-dev libxml2
 #sudo apt-get install python-pip
 #easy_install jonpy
 #easy_install lxml
# build Mapnik
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik_src
cd mapnik_src
python scons/scons.py
sudo python scons/scons.py install
ldconfig

For example the procedure for Ubuntu 10.04:

cd /home/pt
sudo apt-get purge python-mapnik
sudo apt-get autoremove
sudo apt-get install g++ cpp \
libboost1.40-dev libboost-filesystem1.40-dev \
libboost-iostreams1.40-dev libboost-program-options1.40-dev \
libboost-python1.40-dev libboost-regex1.40-dev \
libboost-thread1.40-dev \
python-dev libxml2 libxml2-dev \
libfreetype6 libfreetype6-dev \
libjpeg62 libjpeg62-dev \
libltdl7 libltdl-dev \
libpng12-0 libpng12-dev \
libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2 \
libcairo2 libcairo2-dev python-cairo python-cairo-dev \
libcairomm-1.0-1 libcairomm-1.0-dev \
ttf-unifont ttf-dejavu ttf-dejavu-core ttf-dejavu-extra \
subversion build-essential python-nose libcurl4-gnutls-dev
sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev
sudo apt-get install libgdal1-dev python-gdal \
postgresql-8.4 postgresql-server-dev-8.4 postgresql-contrib-8.4 \
postgresql-8.4-postgis libsqlite3-dev
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik_src
cd mapnik_src
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/
python scons/scons.py
sudo python scons/scons.py install

Be prepared that Mapnik may take an hour to be compiled. Of course, you can use Synaptic Packet Administration to accomplish this, but ensure to get at least Mapnik version 0.7.1.

Mapnik Tools

We already installed Mapnik. Now we should care about some additional files we will need in this context.

cd /home/pt
svn export http://svn.openstreetmap.org/applications/rendering/mapnik
svn export http://mapnik-utils.googlecode.com/svn/tags/nik2img/0.7.0/ mapnik-utils
cd mapnik-utils
sudo python setup.py install

Coastlines

Mapnik needs external shape-file data for coastlines at lower zoom levels; let's download them now. Afterwards we have to unpack the files and dissolve possible subdirectories. The whole procedure can be accomplished by one of the OSM scripts:

cd /home/pt
cd mapnik
./get-coastlines.sh
rm *.tar.bz2 *.tgz *.zip

In case this did not work, go the manual way:

cd /home/pt
mkdir mapnik
mkdir mapnik/world_boundaries
cd mapnik/world_boundaries
rm ../world_boundaries/*
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip
tar -xjf *.tar.bz2
tar -xzf *.tgz
bunzip2 *.bz2
unzip *.zip
rm *.tar.bz2 *.tgz
mv */* .
rmdir * 2>/dev/null

Mapnik Initialization

For Mapnik, some initializations are to be done. Afterwards, we can create our own Mapnik rendering styles. The file osm.xml will be used as a template for this. In this case, we must add some layers with style information about the representation of public transport routes.

cd /home/pt
cd mapnik
./generate_xml.py --host localhost --user ptuser --dbname ptgis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --port '' --password ''
cp osm.xml mapnik_pt.xml
nano mapnik_pt.xml

Prepare the Website

The web server Apache will expect our web content at "/var/www". Note that you will need write-access rights in this directory. If these rights have not been set automatically during Apache installation, you may want to do this by hand:

sudo chown root:www-data /var/www
sudo chmod g+rwx /var/www
sudo usermod -aG www-data pt

To get these changes work, you will need to relogin. If you had chosen another user account, not pt, than please change the last command accordingly.

Having done this, change to that directory and download all necessary files.

cd /var/www
mkdir img
wget http://openlayers.org/dev/theme/default/style.css
wget http://www.openlayers.org/api/OpenLayers.js
wget http://www.openstreetmap.org/openlayers/OpenStreetMap.js
wget -P img http://www.openstreetmap.org/openlayers/img/blank.gif
wget -P img http://www.openstreetmap.org/openlayers/img/cloud-popup-relative.png
wget -P img http://www.openstreetmap.org/openlayers/img/drag-rectangle-off.png
wget -P img http://www.openstreetmap.org/openlayers/img/drag-rectangle-on.png
wget -P img http://www.openstreetmap.org/openlayers/img/east-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/layer-switcher-maximize.png
wget -P img http://www.openstreetmap.org/openlayers/img/layer-switcher-minimize.png
wget -P img http://www.openstreetmap.org/openlayers/img/marker.png
wget -P img http://www.openstreetmap.org/openlayers/img/marker-blue.png
wget -P img http://www.openstreetmap.org/openlayers/img/marker-gold.png
wget -P img http://www.openstreetmap.org/openlayers/img/marker-green.png
wget -P img http://www.openstreetmap.org/openlayers/img/measuring-stick-off.png
wget -P img http://www.openstreetmap.org/openlayers/img/measuring-stick-on.png
wget -P img http://www.openstreetmap.org/openlayers/img/north-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/panning-hand-off.png
wget -P img http://www.openstreetmap.org/openlayers/img/panning-hand-on.png
wget -P img http://www.openstreetmap.org/openlayers/img/slider.png
wget -P img http://www.openstreetmap.org/openlayers/img/south-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/west-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/zoombar.png
wget -P img http://www.openstreetmap.org/openlayers/img/zoom-minus-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/zoom-plus-mini.png
wget -P img http://www.openstreetmap.org/openlayers/img/zoom-world-mini.png

Now delete the existing index file with sudo rm index.html and create a new one with nano index.html and insert this contents:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html lang="de">
<head>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" >
  <title>Public Transport Lines</title>
  <link rel="stylesheet" href="style.css" type="text/css">
  <script src="OpenLayers.js" type="text/javascript"></script>
  <script src="OpenStreetMap.js" type="text/javascript"></script>
  <script type="text/javascript">
    // Start position for the map (hardcoded here for simplicity)
    var lon=11;
    var lat=51;
    var zoom=7;
    var map; //complex object of type OpenLayers.Map
 
    OpenLayers.Protocol.HTTPex= new OpenLayers.Class(OpenLayers.Protocol.HTTP, {
      read: function(options) {
        OpenLayers.Protocol.prototype.read.apply(this,arguments);
        options= OpenLayers.Util.applyDefaults(options,this.options);
        options.params= OpenLayers.Util.applyDefaults(
        options.params,this.options.params);
        options.params.resolution= map.getResolution();
        options.params.zoom= map.getZoom();
        if(options.filter) {
          options.params= this.filterToParams(
            options.filter, options.params);
          }
        var readWithPOST= (options.readWithPOST!==undefined)?
          options.readWithPOST: this.readWithPOST;
        var resp= new OpenLayers.Protocol.Response({requestType: "read"});
        if(readWithPOST) {
          resp.priv= OpenLayers.Request.POST({
            url: options.url,
            callback: this.createCallback(this.handleRead,resp,options),
            data: OpenLayers.Util.getParameterString(options.params),
            headers: {"Content-Type": "application/x-www-form-urlencoded"}
            });
        } else {
          resp.priv= OpenLayers.Request.GET({
            url: options.url,
            callback: this.createCallback(this.handleRead,resp,options),
            params: options.params,
            headers: options.headers
            });
          }
        return resp;
        },
      CLASS_NAME: "OpenLayers.Protocol.HTTPex"
      });
 
    var popwin= null;
 
    function onPopupClose(evt) {
      popwin= null;
      selectControl.unselect(this.feature);
      }
 
    function init() {
      var args= OpenLayers.Util.getParameters(); ///
 
      OpenLayers.Util.onImageLoadError= function() {
        this.src= "emptytile.png"; };
 
      map= new OpenLayers.Map("map", {
        controls:[
          new OpenLayers.Control.Navigation(),
          new OpenLayers.Control.PanZoomBar(),
          new OpenLayers.Control.LayerSwitcher(), ///
          new OpenLayers.Control.Permalink(),
          new OpenLayers.Control.ScaleLine(),
          new OpenLayers.Control.Permalink('permalink'),
          new OpenLayers.Control.MousePosition(),                    
          new OpenLayers.Control.Attribution()],
        maxExtent: new OpenLayers.Bounds(-20037508.34,-20037508.34,20037508.34,20037508.34),
        maxResolution: 156543.0399,
        numZoomLevels: 17,
        units: 'm',
        projection: new OpenLayers.Projection("EPSG:900913"),
        displayProjection: new OpenLayers.Projection("EPSG:4326"),
        restrictedExtent: new OpenLayers.Bounds(550000,5700000,2000000,7450000)
        } );
 
    map.addLayer(new OpenLayers.Layer.OSM.Mapnik("Landkarte",
      { maxZoomLevel: 17, numZoomLevels: 18 }));
 
    map.addLayer(new OpenLayers.Layer.OSM.Mapnik("Landkarte, blass",
      { maxZoomLevel: 17, numZoomLevels: 18, opacity: 0.5 }));
 
    map.addLayer(new OpenLayers.Layer.OSM.CycleMap("Radkarte",
      { maxZoomLevel: 17, numZoomLevels: 18 }));
 
    map.addLayer(new OpenLayers.Layer.OSM.CycleMap("Radkarte, blass",
      { maxZoomLevel: 17, numZoomLevels: 18, opacity: 0.5 }));
 
    map.addLayer(new OpenLayers.Layer.OSM("Kein Hintergrund","img/blank.png",
      { maxZoomLevel: 17, numZoomLevels: 18 }));
 
    map.addLayer(new OpenLayers.Layer.OSM("&Ouml;V-Linien","tiles/${z}/${x}/${y}.png",
      { maxZoomLevel: 17, numZoomLevels: 18, alpha: true, isBaseLayer: false }));
 
    var lay= new OpenLayers.Layer.Vector("Rollstuhleignung", {
      visibility: false,
      strategies: [new OpenLayers.Strategy.BBOX({resFactor: 1.1})],
      protocol: new OpenLayers.Protocol.HTTPex({
        url: "ptfeatures.php?f=w",
        format: new OpenLayers.Format.Text()
        }),
      maxZoomLevel: 17, numZoomLevels: 18,
      transparent: true, isBaseLayer:false, maxResolution : 4.8  // i.e. zoom 15..18
      });
    map.addLayer(lay);
 
    lay= new OpenLayers.Layer.Vector("Fahrplan per Klick", {
      //visibility: false,
      strategies: [new OpenLayers.Strategy.BBOX({resFactor: 1.1})],
      protocol: new OpenLayers.Protocol.HTTPex({
        url: "ptfeatures.php?f=a",
        format: new OpenLayers.Format.Text()
        }),
      maxZoomLevel: 17, numZoomLevels: 18,
      transparent: true, isBaseLayer:false, maxResolution : 4.8  // i.e. zoom 15..18
      });
    map.addLayer(lay);
    lay.redraw();
 
    if(!map.getCenter()){
      var lonLat= new OpenLayers.LonLat(lon, lat).transform(new OpenLayers.Projection("EPSG:4326"),
        map.getProjectionObject());
      map.setCenter(lonLat,zoom);
      }
 
    }
  </script>
</head>
<body onload="init();">
<div style="width:100%; height:100%;" id="map"></div><br>
<div style="border:1px solid black; padding:10px;">
(Enter some map description here...)
</div>
</body>
</html>

Further information can be found here.

Create the Map

This chapter will show you how to create or update the tiles for your map. Tiles are small bitmaps which will be assembled to a whole map by the map browser. We assume that all tasks of the previous chapter have been completed successfully.

The process of map creation involves two steps – filling the database and rendering the map tiles.

Fill the Database

All information we need to create the map tiles must be written into our database. To do this, we need to develop a script which performs several tasks, step by step. Please do not try to create and run the whole script from scratch. It is better to run-through the process step by step, entering every single command at the command line terminal. That makes it much easier to find errors.

Get the Planet File

It is not recommended to use the file of the entire planet. Please choose the file of an area you are interested in, e.g. Germany. The first task of our script will be to download this file and to store it using .o5m file format.

cd /home/pt
wget -O - http://download.geofabrik.de/osm/europe/germany.osm.pbf |./osmconvert - --out-o5m >a.o5m

Filter the Planet File

We need to do a hierarchic filtering because there will be nodes in ways, ways in relations and relations in other relations. For performance reasons a pre-filtered file will be used for interrelational filtering as there is no need for considering nodes or ways in the first filtering stages. In this example we decide to filter public-transport specific information because we want to create a public transport map.

./osmfilter a.o5m --keep="amenity=bus_station highway=bus_stop =platform public_transport=platform railway=station =stop =tram_stop =platform =rail route=rail =train =light_rail =subway =tram route=bus =trolleybus =ferry =railway =funicular line=rail =light_rail =subway =tram =bus =trolleybus =ferry =railway type=public_transport" --drop="railway=platform" >gis.osm

Some seconds later – dependant of the size of the chosen area – we will get the file gis.osm, containing only that information we need.

Note: We are using .o5m file format, because this will accelerate the filter process significantly. See also osmconvert and Daily update an OSM XML file.

Transfer the Data to Postgres Database

Now we can transfer the OSM data from the file gis.osm into the Postgres database. The program osm2pgsql will do this job:

osm2pgsql/osm2pgsql -s -C 700 -d ptgis -U ptuser -S osm2pgsql_pt.style gis.osm

Because the .osm file had been filtered in the previous step, this transfer should take only a few minutes.

Render the Tiles

At first, we should test if the renderer works as expected. Thereby, the nik2img program will help us. Let's pick an area for which we have already imported GIS data – in this example: Nürnberg, Germany – generate a test image and examine it with the viewer Eye of Gnome.

cd /home/pt
cd mapnik
nik2img.py /home/pt/mapnik_pt.xml image.png -c 11 49.5 -z 10 -d 800 400 --no-open -f png256
eog image.png

If the test run was successful, the next step should be generating map tiles. For this purpose, we use the script generate_tiles.py. First, the bounding box must be adjusted to our map area. Open the script with gedit and delete all the lines below # Start with an overview.

cd /home/pt
cd mapnik
gedit generate_tiles.py

Replace the deleted lines with these (do not change the 4 spaces indent):

    force = int(os.environ['F'])
    x1 = float(os.environ['X1'])
    y1 = float(os.environ['Y1'])
    x2 = float(os.environ['X2'])
    y2 = float(os.environ['Y2'])
    minZoom = int(os.environ['Z1'])
    maxZoom = int(os.environ['Z2'])
    bbox = (x1,y1,x2,y2)
    render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom)

Then go up and replace the definition of the function loop (it starts with def loop(self): and ends with self.q.task_done()). Replace the whole definition by these lines (do not change the indents):

    def loop(self):
        while True:
            #Fetch a tile from the queue and render it
            r= self.q.get()
            if r==None:
                self.q.task_done()
                break
            (name,tile_uri,x,y,z)= r
            try:
                mtime= os.stat(tile_uri)[8]  # ST_MTIME
            except:
                mtime= 961063200  # June 2000
            if z<=12 or (mtime>=946681200 and mtime<=978303540) or force==1:
                    # low Zoom OR mtime between Jan 2000 and Dec 2000
                self.render_tile(tile_uri,x,y,z)
            self.q.task_done()

Now start the script:

cd /home/pt
cd mapnik
F=1 X1=5.8 Y1=47.2 X2=15.1 Y2=55.0 Z1=5 Z2=15 MAPNIK_MAP_FILE="/home/pt/mapnik_pt.xml" MAPNIK_TILE_DIR="/var/www/tiles/" ./generate_tiles.py

Depending of the hardware, it may take several hours to render all the tiles. In case you want to abort the rendering, you have to use the Gnome System Monitor or the command kill and kill the process because Ctrl-C does not work here.

Test your new Map

Open a web browser and try to access your new website. If your browser is installed on the same computer as the Apache server, type localhost as URL.

Create the Map Automatically

Did all previous steps work without any errors? Then it's time to pack them into a single script:

cd /home/pt
gedit ptgen.sh
chmod ug+x ptgen.sh

That content could be like this:

#!/bin/bash
cd /home/pt
mv gen.log gen.log_temp
tail -1000 gen.log_temp>gen.log
rm gen.log_temp
echo >>gen.log
date >>gen.log

# download section - start
rm a.o5m
wget -O - http://download.geofabrik.de/osm/europe/germany.osm.bz2 2>/dev/null |./osmconvert --out-o5m >a.o5m
date >>gen.log
# download section - end
if [ -z a.osm.gz"" -o $(stat -L -c%s a.osm.gz) -lt 1000000 ]; then
  echo "OSM file download error" >>upd.log
  date >>gen.log
  exit
fi
ls -lL a.osm.gz >>gen.log
date >>gen.log
./osmfilter a.o5m --keep="amenity=bus_station highway=bus_stop =platform public_transport=platform railway=station =stop =tram_stop =platform =rail route=rail =train =light_rail =subway =tram route=bus =trolleybus =ferry =railway =funicular line=rail =light_rail =subway =tram =bus =trolleybus =ferry =railway type=public_transport" --drop="railway=platform" >gis.osm
ls -lL gis.osm >>gen.log
date >>gen.log

echo Generating data base tables >>gen.log
osm2pgsql/osm2pgsql -s -C 700 -d ptgis -U ptuser -S osm2pgsql_pt.style gis.osm 2>/dev/null >/dev/null
date >>gen.log

echo Rendering the tiles >>gen.log
cd mapnik
F=1 X1=5.8 Y1=47.2 X2=15.1 Y2=55.0 Z1=5 Z2=15 MAPNIK_MAP_FILE="/home/pt/mapnik_pt.xml" MAPNIK_TILE_DIR="/var/www/tiles/" ./generate_tiles.py 2>&1>/dev/null
cd ..
date >>gen.log

Weekly Map Data Update

To get the map data updated automatically every week, an additional Cron entry will be mandatory:

sudo echo -e "0 4 * * 7 root /usr/share/ptgen/ptgen.sh 2>&1>/dev/null \n" > /etc/cron.d/ptgen

Daily Map Data Update

Instead of downloading the whole regional .osm file, you can also use the mechanism for a daily update. After you have followed the description in that page, create a symbolic link to the automatically updated .o5m file (insert the right path):

cd /home/pt
ln -fs /insert_here_the_actual_path_to_osmupdate_directory/a.o5m a.o5m

If you use the script for automatic map creation from above, you will no longer need the script's download part. Hence, remove the 3 lines which are included by the # download section markings.

Using a Rendering Strategy

You can improve the rendering speed significantly if you render only that tiles which have been requested at least once within the last rendering period. The idea behind this strategy is that only those tiles which are frequently used must be kept up-to-date. To activate this strategy you need to change the F=1 to F=0 at the beginning of the line with the generate_tiles.py rendering command.

Now, in lower zoom levels only those tile files will be rendered which do not yet exist or have a file modification time which lies in the year 2000. Of course, we must ensure now that every tile which has been served by Apache, is being marked accordingly. Therefore we will have to run this command every day some minutes after midnight:

LANG=en_US.utf8; cat /var/log/apache2/access.log.1 /var/log/apache2/access.log 2>/dev/null |grep "$(date -d yesterday +"\[%d/%b/%Y")" |cut -d" " -s -f 7 |grep -e "/tiles/13/" -e "/tiles/14/" -e "/tiles/15/" -e "/tiles/16/" |sort -u |sed s"/^/\/var\/www/" |xargs -I '{}' touch -c -t200006151200 '{}'

With this single command, the last Apache log files are searched for tile request entries which have been recorded yesterday. The modification time of all these tile files is set to June 2000.

We should pack this command into script, let's name it marktiles.sh, and add another Cron entry:

sudo echo -e "45 0 * * * root /usr/share/ptgen/marktiles.sh 2>&1>/dev/null \n" > /etc/cron.d/marktiles

Note that using generate_tiles.py incurs a significant performance penalty compared to the typical Tirex or renderd setups, as tirex and renderd will render "metatiles (8x8 tiles) in one go. Rendering a metatile takes on average 4 times as long as rendering a single tile, i.e. you achieve 16 times the tile throughput of generate_tiles.py - provided that the tiles you render are close to each other.

An Advanced Approach: Diff-based Rendering

In the previous chapter we updated map tiles which have been watched by at least one visitor of the map's website. This saves a lot of rendering effort but it is still far from being optimal.

Much more efficient would it be if we rendered only that tiles whose OSM data have been changed. With the right call, the program osm2pgsql will supply us with a list of these tiles. This is usually not very helpful for a thematic overlay map because 90% of the changed data will not affect the overlay image, but there is a way to get exactly the information we need: update the PostgreSQL database with OSM-Change files which have been retrieved from intensively filtered data.

Some Details to the Alternative Strategy

The idea is to compare the filtered data from before an update against the filtered data from after this update, and then to generate an OSM-Change file (.osc) on the basis of this difference.

This OSM-Change file should not bee very large because it is retrieved from filtered data, therefore the dirty_tiles list which is produced by osm2pgsql will not be long and the regular rendering process should not take days to conclude.

Getting a List of all Dirty Tiles

Speaking of the file dirty_tiles, the program osm2pgsql does not provide us with the names of all affected tiles because this list is stripped of all redundant information. If there is, for example, the tile 3/0/1 (zoom, x, y) in the list, the also affected tile 2/0/0 will be omitted. Furthermore, if all four subtiles – 3/0/0, 3/0/1, 3/1/0 and 3/1/1 – where affected, none of them would appear in the list, they would be replaced by the entry 2/0/0.

This is a very effective way to reduce the dirty-tiles list length but we still need a list of all effected tiles. For this reason, we decide to append one of osm2pgql's source files: "expire-tiles.c".

Omit Empty Tiles

Empty tiles on a transparent map layer are completely transparent graphics and not of any use for our map, they just waste disk space. Therefore it is better to refrain from saving them in the first place. This will be accomplished by a few additional lines in the Mapnik Python script.

OpenLayers must be informed about this intention also, otherwise it would post error messages upon missing tiles or trying to wait for them.

Schematic Diagram

Here is an overview of the programs and files we are going to use. This diagram is somewhat simplified; it has been created to show all main steps of the strategy in one picture.

                   hourly.osc
                        |
                        v
                  +-----------+
                  | osmupdate |
      a.osm ----> | -B=p.poly | ----> b.osm
        |         +-----------+         |
        v               ^               v
+---------------+       |       +---------------+
|   osmfilter   |     p.poly    |   osmfilter   |
| --keep=route= |               | --keep=route= |
+---------------+               +---------------+
        |                               |
        v         +------------+        v
      fa.osm ---> | osmconvert | <--- fb.osm
                  |   --diff   |
                  +------------+
                        |
                        v
                     gis.osc
                        |
                        v
                  +------------+
                  | osm2pgsql  | ---> database
                  | -e 5-12 -a |      update
                  +------------+
                        |
                        v
                   dirty_tiles
                        |
                        v
                +----------------+
                | mapnik_tile.py | ---> new tiles
                +----------------+

In the diagram some of the files are shown with conventional extensions (.osm and .osc) although compressed and binary formats are being used (.osc.gz, .o5m, .o5c).

Involved Files

hourly.osc: hourly OsmChange file, downloaded from planet.openstreetmap.org
p.poly: border polygon of our graphical region
a.osm: previous OSM data file (all data of our geographical region)
b.osm: updated OSM data file (all data of our geographical region)
fa.osm: previous filtered OSM data file (only thematic data, e.g. public transport)
fb.osm: filtered new OSM data file (only thematic data, e.g. public transport)
gis.osc: the OsmChange file (so-called diff file) you would need to update fa.osm to fb.osm
dirty_tiles: a list of all tiles which are affected by the gis.osc and therefore have to be rerendered

Used Programs

osmupdate: download .osc files and use them to update an .osm file
osmfilter: filter OSM data and discard all the information we do not need
osmconvert: compare the difference between two .osm files and create an .osc diff file
osm2pgsql: update a postgreSQL geo database and calculate a dirty-tiles list
mapnik_tile.py: read the dirty-tiles file and (re)render all listed tiles

Most of the task also can be done – maybe more reliable – with Osmosis. If you already have installed Osmosis, feel free to use it. Osmosis offers a much wider variety of functions but will be slower.

How-To Implement the Alternative Strategy

Please note that this strategy has not been tested yet. Please expect errors and unexpected behaviour. Nevertheless, the following sections will give you a brief description how one might implement the strategy.

The Usual Steps

Please follow the steps which are described by the chapter #Prepare your System, then continue with the following tasks.

Tool osmupdate

Following this strategy, there is no need to adhere to a fixed update timetable. Hence we decide to use not only weekly or daily downloads but also the hourly .osc files. The program osmupdate will help us to keep the local OSM data files up-to-date.

To download and compile osmupdate, enter these commands:

cd /home/pt
wget -O - http://m.m.i24.cc/osmupdate.c |cc -x c - -o osmupdate

For further details on this tool refer to osmupdate.

Adapt osm2pgsql

As described above, we will have to adapt one of osm2pgsql's source files. The following changes will have to be made (for osm2pgsql version 0.80.0). Please create a new file:

cd /home/pt
cd osm2pgsql
nano expire-tiles.diff

Now copy this contents into the new file:

*** expire-tiles_0_80_0.c    2011-05-21 15:13:49.000000000 +0200
--- expire-tiles.c    2011-08-10 16:23:05.000000000 +0200
*************** static int _mark_tile(struct tile ** tre
*** 105,114 ****
--- 105,146 ----
   */
  static int mark_tile(struct tile ** tree_head, int x, int y, int zoom) {
  	return _mark_tile(tree_head, x, y, zoom, 0);
  }
  
+ #if 1
+ static void output_dirty_tile(FILE * outfile, int x, int y, int zoom, int min_zoom) {
+   // writes a tile into output file, including all subsequent tiles
+   // with higher zoom levels;
+   int x_max, y_max;
+   int tile_size;
+ 
+   tile_size= 1;
+   while (zoom <= Options->expire_tiles_zoom) {
+     if(zoom>=min_zoom) {
+       x_max = x + tile_size;
+       while (x < x_max) {
+         y_max = y + tile_size;
+         while (y < y_max) {
+ 		      if ((outcount++ % 1000) == 0) {
+ 			      fprintf(stderr, "\rWriting dirty tile list (%ik)", outcount / 1000);
+ 			      fflush(stderr);
+ 		      }
+ 		      fprintf(outfile, "%i/%i/%i\n", zoom, x, y);
+           y++;
+         }
+         y-= tile_size;
+         x++;
+       }
+       x-= tile_size;
+     }
+     zoom++;
+     x<<= 1; y<<= 1;
+     tile_size<<= 1;
+   }
+ }
+ #else
  static void output_dirty_tile(FILE * outfile, int x, int y, int zoom, int min_zoom) {
  	int	y_min;
  	int	x_iter;
  	int	y_iter;
  	int	x_max;
*************** static void output_dirty_tile(FILE * out
*** 131,148 ****
--- 163,191 ----
  			}
  			fprintf(outfile, "%i/%i/%i\n", out_zoom, x_iter, y_iter);
  		}
  	}
  }
+ #endif
  
  static void _output_and_destroy_tree(FILE * outfile, struct tile * tree, int x, int y, int this_zoom, int min_zoom) {
  	int	sub_x = x << 1;
  	int	sub_y = y << 1;
  	FILE *	ofile;
  
  	if (! tree) return;
  
+ #if 1
+   if(this_zoom >= min_zoom) {
+     if ((outcount++ % 1000) == 0) {
+       fprintf(stderr, "\rWriting dirty tile list (%ik)", outcount / 1000);
+       fflush(stderr);
+     }
+     fprintf(outfile, "%i/%i/%i\n", this_zoom, x, y);
+   }
+ #endif
+ 
  	ofile = outfile;
  	if ((tree->complete[0][0]) && outfile) {
  		output_dirty_tile(outfile, sub_x + 0, sub_y + 0, this_zoom + 1, min_zoom);
  		ofile = NULL;
  	}

Use these commands to apply the changes and to recompile osm2pgsql's source:

patch -b expire-tiles.c expire-tiles.diff
make

Feed Mapnik with a Tiles List

Mapnik must be enabled to read and process dirty-tiles lists. The following python script will exactly do this. Enter /home/pt/mapnik directory and create a file with the name mapnik_tile.py and the following contents:

#!/usr/bin/python
# mapnik_tile 2011-08-20 13:40
# read dirty-tiles file and render each tile of this list;
# afterwards, delete this file;
# call: DTF="dirty_tiles" mapnik_tile.py
# parallel call: DTF="dirty_tiles" PID=123 mapnik_tile.py
# License: AGPL
# (c) Markus Weber, Nuernberg
from math import pi,cos,sin,log,exp,atan
from subprocess import call
import sys, os
from Queue import Queue
import mapnik
import threading
import shutil
 
DEG_TO_RAD = pi/180
RAD_TO_DEG = 180/pi
 
def minmax (a,b,c):
    a = max(a,b)
    a = min(a,c)
    return a
 
class GoogleProjection:
    def __init__(self,levels=18):
        self.Bc = []
        self.Cc = []
        self.zc = []
        self.Ac = []
        c = 256
        for d in range(0,levels):
            e = c/2;
            self.Bc.append(c/360.0)
            self.Cc.append(c/(2 * pi))
            self.zc.append((e,e))
            self.Ac.append(c)
            c *= 2
 
    def fromLLtoPixel(self,ll,zoom):
         d = self.zc[zoom]
         e = round(d[0] + ll[0] * self.Bc[zoom])
         f = minmax(sin(DEG_TO_RAD * ll[1]),-0.9999,0.9999)
         g = round(d[1] + 0.5*log((1+f)/(1-f))*-self.Cc[zoom])
         return (e,g)
 
    def fromPixelToLL(self,px,zoom):
         e = self.zc[zoom]
         f = (px[0] - e[0])/self.Bc[zoom]
         g = (px[1] - e[1])/-self.Cc[zoom]
         h = RAD_TO_DEG * ( 2 * atan(exp(g)) - 0.5 * pi)
         return (f,h)
 
if __name__ == "__main__":
 
  # read environment variables
  dt_file_name = os.environ['DTF']
  try:
    temp_file = "/dev/shm/tile" + os.environ['PID']
  except:
    temp_file = "/dev/shm/tile"
 
  # do some global initialization
  print "mapnik_tile " + dt_file_name + ": started."
  mapfile = "mapnik_pt.xml"
  tile_dir = "/var/www/tiles/"
  max_zoom = 18
  tile_count = 0
  empty_tile_count = 0
 
  # create tile directory and all possible zoom directories
  if not os.path.isdir(tile_dir):
    os.mkdir(tile_dir)
  for z in range(0, max_zoom):
    d = tile_dir + "%s" % z + "/"
    if not os.path.isdir(d):
      os.mkdir(d)
 
  # open the dirty-tiles file
  try:
    dt_file = file(dt_file_name, "r")
  except:
    dt_file = None
  if (dt_file != None):
 
    # do some Mapnik initialization
    mm = mapnik.Map(256, 256)
    mapnik.load_map(mm, mapfile, True)
    # Obtain <Map> projection
    prj = mapnik.Projection(mm.srs)
    # Projects between tile pixel co-ordinates and LatLong (EPSG:4326)
    tileproj = GoogleProjection(max_zoom + 1)
 
    # process the dirty-tiles file
    while True:
 
      # read a line of the dirty-tiles file
      line = dt_file.readline()
      if (line == ""):
        break
      line_part = line.strip().split('/')
 
      # create x coordinate's directory - if necessary
      d = tile_dir + line_part[0] + "/" + line_part[1] + "/"
      if not os.path.isdir(d):
        os.mkdir(d)
 
      # get parameters of the line
      z = int(line_part[0])
      x = int(line_part[1])
      y = int(line_part[2])
      tile_name= tile_dir + line[:-1] + ".png"
 
      # render this tile - start
 
      # Calculate pixel positions of bottom-left & top-right
      p0 = (x * 256, (y + 1) * 256)
      p1 = ((x + 1) * 256, y * 256)
 
      # Convert to LatLong (EPSG:4326)
      l0 = tileproj.fromPixelToLL(p0, z);
      l1 = tileproj.fromPixelToLL(p1, z);
 
      # Convert to map projection (e.g. mercator co-ords EPSG:900913)
      c0 = prj.forward(mapnik.Coord(l0[0], l0[1]))
      c1 = prj.forward(mapnik.Coord(l1[0], l1[1]))
 
      # Bounding box for the tile
      if hasattr(mapnik,'mapnik_version') and mapnik.mapnik_version() >= 800:
        bbox = mapnik.Box2d(c0.x,c0.y, c1.x,c1.y)
      else:
      	bbox = mapnik.Envelope(c0.x,c0.y, c1.x,c1.y)
      render_size = 256
      mm.resize(render_size, render_size)
      mm.zoom_to_box(bbox)
      mm.buffer_size = 256
      # (buffer size was 128)
 
      # render image with default Agg renderer
      im = mapnik.Image(render_size, render_size)
      mapnik.render(mm, im)
      im.save(temp_file, 'png256')
      tile_count = tile_count + 1
 
      # render this tile - end
 
      # copy this tile to tile tree
      l= os.stat(temp_file)[6]
      if l>116:
        shutil.copyfile(temp_file,tile_name)
      else:
        try:
          os.unlink(tile_name)
        except:
          l= l  # (tile did not exist)
        empty_tile_count = empty_tile_count + 1
 
      # print progress information
      # if (tile_count % 250) == 0:
        # print "mapnik_tile " + dt_file_name + ": Progress:", tile_count, "tiles."
        # sys.stdout.flush()
 
    # close and delete the dirty-tiles file
    dt_file.close()
    try:
      os.unlink(dt_file_name)
    except:
      l= l  # (file did not exist)
 
  # print some statistics
  print "mapnik_tile " + dt_file_name + ":", tile_count, "tiles (" + "%s" % empty_tile_count + " empty)."

Create an Empty Tile Image

OpenLayers must be advised what to do if an tile image is to be loaded which does not exist on the server. Fortunately, we already did the necessary changes to the file index.html (see above, look for emptytile.png). Now we must create such an "empty tile" and store it as .png file. Just enter this command:

echo -en "\x89PNG\r\n\x1a\n...\rIHDR..;...;.;\x03...f\xbc:"\
"%...\x03PLTE...\xa7z=\xda...;tRNS.@\xe6\xd8f..."\
"\x1fIDATh\x81\xed\xc1;\r...\xc2\xa0\xf7Om\x0e7\xa0........\xbe\r"\
"\x21..;\x9a\x60\xe1\xd5....IEND\xaeB\x60\x82" |\
tr ".;" "\000\001" >/var/www/emptytile.png

Toolchain for the Alternative Strategy

First prepare an osm filter parameter file. Open a new file with the name toolchain_filter with the editor and insert the following text:

--keep=
route=ferry =railway =train =light_rail =subway =tram =trolleybus =bus =funicular =aerialway
line=ferry =railway =rail =train =light_rail =subway =tram =trolleybus =bus =funicular
public_transport=stop_area =stop_position
railway=station =halt =tram_stop
highway=bus_stop
amenity=bus_station
aerialway=station

--drop=
railway=platform

--keep-tags=all
railway=station =halt =tram_stop
highway=bus_stop
amenity=bus_station
disused=yes
public_transport=stop_area
aerialway=station =yes
ferry=yes train=yes subway=yes
monorail=yes tram=yes trolleybus=yes bus=yes
ref= uic_ref= ref_name= name= website= wheelchair=

--keep-way-relation-tags=all
route= line= type=

Now use the same method to create the toolchain script, name it toolchain_pt.sh:

#!/bin/bash
# toolchain_pt.sh
# (c) Markus Weber, 2011-09-18 18:10, 2012-01-20
# License: AGPL
#
# This script cares about creating a thematical map.
# It loads and updates the map data and renders the tiles.
# The script will run endless in a loop. To terminate it,
# please delete the file "toolchain_running.txt".
 
PLANETURL=download.geofabrik.de/osm/europe/switzerland.osm.pbf
PLANETMINSIZE=60000000  # minimum size of OSM data file in .o5m format
BORDERS="-B=switzerland.poly"
MAXPROCESS=5  # maximum number of concurrent processes for rendering
OSM2PGSQLPARAM="-s -C 3000 -d ptgis -U ptuser -S osm2pgsql_pt.style"
  # main parameters to be passed to osm2pgsql
 
# enter working directory and do some initializations
cd /home/pt
echo >>tc.log
echo $(date)"  toolchain started." >>tc.log
PROCN=1000  # rendering-process number (range 1000..1999)
mkdir d_t 2>/dev/null
 
# publish that this script is now running
rm toolchain_ended.txt 2>/dev/null
echo -e "The toolchain script has been started at "$(date)"\n"\
"and is currently running. To terminate it, delete this file and\n"\
"wait until the file \"toolchain_ended.txt\" has been created.\n"\
"This may take some minutes." \
>toolchain_running.txt
 
# clean up previous Mapnik processes
killall "mapnik_tile.py" 2>/dev/null
while [ $(ls d_t/at* 2>/dev/null |wc -l) -gt 0 ]; do
    # there is at lease one incompleted rendering process
  AT=$(ls -1 d_t/at* 2>/dev/null|head -1)  # tile list
  echo $(date)"  Cleaning up incomplete rendering: "$AT >>tc.log
  DT=${AT:0:4}d${AT:5:99}  # new name of the tile list
  mv $AT $DT  # rename this tile list file;
    # now the tile ist is market as 'to be rendered'
  done
 
# download and process planet file - if necessary
if [ "0"$(stat --print %s a.o5m 2>/dev/null) -lt $PLANETMINSIZE ]; then
  echo $(date)"  Missing file a.o5m, downloading it." >>tc.log
  rm -f fa.o5m
  wget -nv $PLANETURL -O - 2>>tc.log |./osmconvert - \
    $BORDERS --out-o5m >a.o5m 2>>tc.log
  echo $(date)"  Updating the downloaded planet file." >>tc.log
  rm -f b.o5m
  ./osmupdate -v a.o5m b.o5m --keep-tempfiles \
    --max-merge=15 $BORDERS --drop-author >>tc.log 2>&1
  if [ "0"$(stat --print %s b.o5m 2>/dev/null) -gt $PLANETMINSIZE ]; then
    echo $(date)"  Update was successful." >>tc.log
    fi
  mv -f b.o5m a.o5m
  if [ "0"$(stat --print %s a.o5m 2>/dev/null) -lt $PLANETMINSIZE ]; then
    echo $(date)"  toolchain Error: could not download"\
      $PLANETURL >>tc.log
    exit 1
    fi
  fi
 
# refill the database - if necessary
if [ "0"$(stat --print %s fa.o5m 2>/dev/null) -lt 5 ]; then
  echo $(date)"  Missing file fa.o5m, creating it." >>tc.log
  rm dirty_tiles d_t/* 2>/dev/null
  echo $(date)"  Filtering the downloaded planet file." >>tc.log
  ./osmfilter a.o5m --parameter-file=toolchain_filter --out-o5m \
    >fa.o5m 2>>tc.log  # filter the planet file
  ./osmconvert fa.o5m --out-osm >gis.osm 2>>tc.log
    # convert to .osm format
  echo $(date)"  Writing filtered data into the database." >>tc.log
  ./osm2pgsql/osm2pgsql $OSM2PGSQLPARAM -c gis.osm -e 4-17 >/dev/null 2>&1
    # enter filtered planet data into the database
  echo $(date)"  All tiles need to be rerendered!" >>tc.log
  echo $(date)"  If the tile directory is not empty, please remove" >>tc.log
  echo $(date)"  all outdated tile files." >>tc.log
  fi
 
# main loop
while [ -e "toolchain_running.txt" ]; do
  echo $(date)"  Processing main loop." >>tc.log
 
  # limit log file size
  if [ "0"$(stat --print %s tc.log 2>/dev/null) -gt 1500000 ]; then
    echo $(date)"  Reducing logfile size." >>tc.log
    mv tc.log tc.log_temp
    tail -c +1000000 tc.log_temp |tail -n +2 >tc.log
    rm tc.log_temp 2>/dev/null
    fi
 
  #care about entries in dirty-tile list
  while [ $(ls dirty_tiles d_t/?t* 2>/dev/null |wc -l) -gt 0 -a \
      -e "toolchain_running.txt" ]; do
      # while still tiles to render
 
    # start as much render processes as allowed
    while [ $(ls d_t/dt* 2>/dev/null |wc -l) -gt 0 -a \
        $(ls -1 d_t/at* 2>/dev/null |wc -l) -lt $MAXPROCESS ]; do
        # while dirty tiles in list AND process slot(s) available
      DT=$(ls -1 d_t/dt* |head -1)  # tile list
      AT=${DT:0:4}a${DT:5:99}  # new name of the tile list
      mv $DT $AT  # rename this tile list file;
        # this is our way to mark a tile list as 'being rendered now'
      #echo $(date)"  Rendering "$DT >>tc.log
      PROCN=$(($PROCN + 1))
      if [ $PROCN -gt 1999 ]; then
        PROCN=1000;  # (force range to 1000..1999)
        fi
      N=${PROCN:1:99}  # get last 3 digits
      cd mapnik
      DTF=../$AT PID=$N nohup ./mapnik_tile.py >/dev/null 2>&1 &
        # render every tile in list
      cd ..
      echo $(date)"  Now rendering:"\
        $(ls -m d_t/at* 2>/dev/null|tr -d "d_t/a ") \
        >>tc.log
      sleep 2  # wait a bit
      done
 
    # determine if we have rendered all tiles of all lists
    if (ls d_t/?t* >/dev/null 2>&1); then  # still tiles to render
      sleep 11  # wait some seconds
  continue  # care about rendering again
      fi
    # here: we have rendered all tiles of all lists
 
    # care about dirty-tiles master list and split it into parts
    if (ls dirty_tiles >/dev/null 2>&1); then
        # there is a dirty-tiles master list
      echo $(date)"  Splitting dirty-tiles list" \
        "("$(cat dirty_tiles |wc -l)" tiles)" >>tc.log
      split -l 1000 -d -a 6 dirty_tiles d_t/dt
      echo "*** "$(date) >>dt.log
      cat dirty_tiles >>dt.log  # add list to dirty-tiles log
      rm dirty_tiles 2>/dev/null
      # limit dirty-tiles log file size
      if [ "0"$(stat --print %s dt.log 2>/dev/null) -gt 750000000 ]; then
        echo $(date)"  Reducing dirty-tiles logfile size." >>tc.log
        mv dt.log dt.log_temp
        tail -c +500000000 dt.log_temp |tail -n +2 >dt.log
        rm dt.log_temp 2>/dev/null
        fi
      fi
 
    done  # while still tiles to render
  if [ ! -e "toolchain_running.txt" ]; then  # script shall be terminated
continue;  # exit the main loop via while statement
    fi
  # here: all tiles have been rendered
 
  # update the local planet file
  echo $(date)"  Updating the local planet file." >>tc.log
  rm b.o5m fb.o5m a.o5c 2>/dev/null
  ./osmupdate a.o5m b.o5m --daily \
    --max-merge=15 $BORDERS --drop-author -v >>tc.log 2>&1
  if [ "0"$(stat --print %s b.o5m 2>/dev/null) -lt \
      $(expr "0"$(stat --print %s a.o5m 2>/dev/null) \* 9 / 10) ]; then
      # if new osm file smaller than 90% of the old file's length
    # wait a certain time and retry the update
    I=0
    while [ $I -lt 33 -a -e "toolchain_running.txt" ]; do  # (33 min)
      sleep 60
      I=$(( $I + 1 ))
      done
continue
    fi
 
  # filter the new planet file
  echo $(date)"  Filtering the new planet file." >>tc.log
  ./osmfilter b.o5m --parameter-file=toolchain_filter --out-o5m >fb.o5m \
    2>>tc.log
 
  # calculate difference between old and new filtered file
  echo $(date)"  Calculating diffs between old and new filtered file."\
    >>tc.log
  ./osmconvert fa.o5m fb.o5m --diff-contents --fake-lonlat --out-osc \
    >gis.osc 2>>tc.log
 
  # check if the diff file is too small
  if [ "0"$(stat --print %s gis.osc 2>/dev/null) -lt 50 ]; then
    echo $(date)"  Error: diff file is too small." >>tc.log
    exit 2
    fi
 
  # enter differences into the database
  echo $(date)"  Writing differential data into the database." >>tc.log
  ./osm2pgsql/osm2pgsql $OSM2PGSQLPARAM -a gis.osc -e 4-17 >/dev/null 2>&1
 
  # replace old files by the new ones
  echo $(date)"  Replacing old files by the new ones." >>tc.log
  ls -lL a.o5m fa.o5m b.o5m fb.o5m gis.osm gis.osc >>tc.log
  mv -f a.o5m a_old.o5m
  mv -f fa.o5m fa_old.o5m
  mv -f b.o5m a.o5m
  mv -f fb.o5m fa.o5m
 
  # check if there are any tiles affected by this update
  if [ "0"$(stat --print %s dirty_tiles 2>/dev/null) -lt 5 ]; then
    echo $(date)"  There are no tiles affected by this update." >>tc.log
    rm -f dirty_tiles
    fi
 
  done  # main loop
 
# wait until every rendering process has terminated
while (ls d_t/at* 2>/dev/null) ; do sleep 30; done
 
# publish that this script has ended
rm toolchain_running.txt 2>/dev/null
echo "The toolchain script ended at "$(date)"." \
  >toolchain_ended.txt

Further Strategies

Mapnik experts will have noticed that the rendering strategies which have been introduced on this page do not represent the standard solution. Most Slippy Map installations will use the more comfortable Tirex/Modtile tools to define which tiles when to render. Furthermore, of course, you usually do not render regularly sized tiles but so-called meta tiles instead, which are 16 or 64 times larger. On standard maps this increases rendering speed significantly. It would be interesting to know if there are advantages for diff based rendered thematic maps too because you would have to render a lot of areas which have not changed.

Please feel free to add your experiences here if you have tried different methods and had the opportunity to compare them.

Benchmarks

Be prepared that in the first run every affected tile of your thematic layer will be rendered. This may take between one hour and a few weeks, depending on the available hardware, the size of the chosen geographical region and the density of the thematic data.

The planet-wide public transport map, for example, took about four days to be initially rendered on a quad core CPU. The statistics in detail:

1 hour downloading and converting the planet file
30 minutes initially updating the planet file
20 minutes filtering the planet file
40 minutes writing the filtered data to the database
4 days rendering all the tiles of the thematic layer

Map updates run faster. An average updates takes about 15 hours. In detail (example):

20 minutes updating the planet file
20 minutes filtering the new planet file
8 seconds calculating the differences between old and new filtered data
10..20 minutes writing the differences of the filtered data to the database
10..30 hours rerendering the changed tiles of the thematic layer

Troubleshooting

Too many links

If this message occurs during rendering, you most likely are using a file system which supports only a limited number of subdirectories (e.g. ext2 or ext3). Please upgrade to ext4. The migration from ext3 to ext4 should be possible without any loss of data. Please refer to the users guide of your operating system.