Mapnik

From OpenStreetMap Wiki

Mapnik is the software we use to render the main Slippy Map layer for OSM, along with other layers such as the "cycle map" layer and "noname" layer. It is also the name given to the main layer, so things get a bit confusing sometimes.

Mapnik is a open source toolkit for rendering maps. It's written in C++ and includes high-level Python bindings. It uses the AGG library and offers anti-aliasing rendering with subpixel accuracy. It can read ESRI shapefiles, PostGIS, TIFF rasters, .osm files, any GDAL or OGR supported formats. Packages are available for most Linux distributions and binaries are available for Mac OS X and Windows.

The official homepage is mapnik.org, and new users can learn more about using Mapnik at the The Mapnik Wiki and by joining the #mapnik channel on freenode

Bugs and rendering suggestions

Please report any bugs, graphical suggestions, and requests for unrendered features on the OSM mapnik map to OSM trac under the "mapnik" component. Please check if the ticket has been filed before: trac tickets with component "mapnik".

Report problems, ideas, enhancements about Mapnik software in general (not the main OSM map layer) to the mapnik project at http://trac.mapnik.org

Overview

OSM uses Mapnik to render 256 × 256 pixels tiles, which are served from the tile server (tile.openstreetmap.org). The Slippy Map JavaScript references URLs such as e.g. http://tile.openstreetmap.org/7/63/42.png for this image:

See also Slippy map tilenames

Data Sources

OSM data is converted to a format usable by Mapnik by osm2pgsql, which loads derived data into a PostGIS database. In addition common usages of Mapnik also rely on shapefiles to render a few additional layers. Coastlines, for example, are created using shapefiles generated from the natural=coastline ways by the Coastline Error Checker. See Coastline#Main_Mapnik_Layer

Preparation

This is a work in progress.

This process will require quite a bit of diskspace if you plan on loading the entire world. Individual countries can be handled on any ordinary machine, but for the whole planet here are some indications of the requirements (as at 4 Apr 2009). This does not include the indexes:

Which totals about 30 GB. If you are using slim-mode in osm2pgsql you will also need around 20 GB for the temporary tables. Please mention you would probably need indexes as well, and the whole database with indexes amounts to 61 GB!

Fresh news: As of Apr 2010, europe.osm.bz2 in slim-mode will need around 90Go of disk space

If you have enough memory (>12 GB) then you can try running in normal mode. On a suitable machine the process should complete in 3-4 hours. If you don't have enough memory then you need to use the slim-mode which will work with any amount of memory you can give it. Configured for 3 GB RAM the process has been measured at around 24 hours, though it is strongly dependent on your disk subsystem.

Note: slim-mode is required if you want to apply the planet diff files (e.g., minutely diffs).

There is a graph explaining memory usage during the planet import, in a RAM mode:

Don't forget you count the 6.1 GB for the compressed planet dump itself!

Datasets

There are two large datasets needed to render tiles, so its probably a good idea to start downloading those first.

Planet.osm

Planet.osm is a snapshot of the OpenStreetMap database. See Planet.osm

$ wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Note: If you want to render a small area as a quick test then you can import any .osm file, you do not need to render the whole planet. This is recommended if you want to quickly prove that the rendering is working before you spend several hours downloading 3+ GB and importing the whole planet. You can get a suitable .osm file by downloading and saving an area in JOSM or use one of the smaller planet extracts listed at Planet.osm#Extracts.

World boundaries

Mapnik uses external shapefile data for coastlines at lower zoom levels. See Coastline#Main_Mapnik_Layer

$ wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz (50 MB)
$ wget http://tile.openstreetmap.org/processed_p.tar.bz2 (357 MB)
$ wget http://tile.openstreetmap.org/shoreline_300.tar.bz2 (48 MB)

After downloading them and getting a checkout of the Mapnik scripts (see details below), make a directory mapnik/world_boundaries and unpack the archives, but move all the files from any subdirectories that might exist directly in the world_boundaries directory.

Since June 2010, two additional files are needed:

$ wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip (1.5 MB)
$ wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip (38 KB)

PostgreSQL/PostGIS

Mapnik can use PostgreSQL/PostGIS as its spatial database. For instructions on how to install and configure PostgreSQL/PostGIS for use with Mapnik, see the Mapnik/PostGIS installation page.

osm2pgsql

osm2pgsql is a Openstreetmap data to PostgreSQL converter and loader.

For installation instructions, see osm2pgsql. Before you can use osm2pgsql for the first time with the Spherical Mercator projection (see below), you need to initialize configuration data for this projection. Do this by running the .sql file included with osm2pgsql:

[Syntax on Windows]
$ psql -d gis -f c:\osm2pgsql\900913.sql

[Syntax on Linux]
$ wget http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/900913.sql?format=raw -O 900913.sql
$ psql -d gis -f ./900913.sql

Enabled Intarray (if planning on using diffs)

psql gis < /usr/share/postgresql/8.3/contrib/_int.sql

Also you should make sure that the database schema used by osm2pgsql is in sync with the one Mapnik acts on. If they aren't, funny things will happen that aren't obvious right away.

The schema is described in the file default.style for osm2pgsql and osm.xml (resp. osm-template.xml) for Mapnik. default.style of your osm2pgsql package is very likely out of date when not building from svn source. So when fetching Mapnik OSM stuff from svn (see below) you should also update default.style e.g. via repository browser.

Installing Mapnik

To install Mapnik, please follow the instructions here: Mapnik/Installation.

For more specific installation steps on Fedora 10, follow these steps: Mapnik/Fedora Installation.

Loading Data

Simply call osm2pgsql (as the user who will be running mapnik):

$ osm2pgsql -m -d gis planet-latest.osm.bz2

This process requires significant amounts of free memory (by rough estimate, at least 512 MB). If it runs out of memory, it can fail in somewhat unpredictable ways (e.g. messages suggesting that errors exist in the .osm file). Make sure that you have sufficient free RAM and swap space available before running osm2pgsql.

The "-m" option enables the use of the Spherical Mercator projection which is recommended. This is the default since version 0.55, SVN r8320. Some older tools may still expect the data to be imported in a different projection such as "latlong" or the non-spherical Mercator. There are command line options available to select these and other projections.

The latest osm2pgsql code also supports a simple bounding box filter during the import which will make things a lot faster if you only want to render a small part of the planet. For example if you just wanted to import London from the UK planet extract:-

$ osm2pgsql --bbox -0.5,51.25,0.5,51.75 -m -d gis uk-080213.osm.bz2

Some issues you may encounter

Wrong projection

If all your map tiles appear to be off by 10 - 20 km in the North/South direction then you may have forgotten to add the "-m" option on the osm2pgsql import. Since osm2pgsql version 0.55 the spherical mercator (-m) is the default projection.

Cannot open data file

If you attempt to load a gzip'd planet file which is more than 7 GB you may see an error that it fails to open the file. This is due to a bug in some versions of zlib which fail to supply the O_LARGEFILE flag to the open(). The workaround is to use an external gzip process which does not suffer this bug. Specify the filename "-" will make osm2pgsql read the data from STDIN.

gzip -dc planet-080213.osm.gz | osm2pgsql -m -

Out of memory

If your load fails with an error similar to

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100

Reading in file: -
Entity: line 1: parser error : Extra content at the end of the document

^
- : failed to parse
Error occurred, cleaning up

You probably need to run osmpg2sql in slim mode, which stores intermediate objects on disk rather than in RAM. To do so run osm2pgsql with the '-s' switch:

osm2pgsql -m -s -d gis planet-080213.osm.gz

Invalid projection in pgSQL

You may receive this error right after starting:

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 );
  failed: ERROR:  AddGeometryColumns() - invalid SRID
CONTEXT:  SQL statement "SELECT AddGeometryColumn('','', $1 , $2 , $3 , 
$4 , $5
)"
PL/pgSQL function "addgeometrycolumn" line 4 at SQL statement

This means that the projection has not been intialized and you need to execute 900913.sql in the osm2pgsql folder. To do this :

wget http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/900913.sql && psql -f 900913.sql gis

to make it work after that, I had to restart the database daemon :

sudo /etc/init.d/postgresql restart

Locked out

After several hours the import may quit with an error like:-

ANALYZE planet_osm_line;
 failed: ERROR:  deadlock detected
DETAIL:  Process 28511 waits for AccessExclusiveLock on relation 1064115 of database 18309; blocked by process 12776.
Process 12776 waits for ShareLock on transaction 572766655; blocked by process 28511.

Error occurred, cleaning up

This seems to be a fault in some versions of PostgreSQL and is caused when an auto-vacuum is attempted during the ANALYZE. The solution is to disable all auto-vacuums on the database. The data is not updated after the import so the vacuum process does nothing useful. In the postgresql.conf file set the option:

autovacuum = off

Then restart the database server

sudo /etc/init.d/postgresql-8.1 restart

Note: In Debian/Ubuntu you also need to update /etc/cron.d/postgresql-common to comment out the two pg_maintenance tasks (add a '#' in front of them) which would otherwise cause vacuums to occur at regular intervals:-

# Run VACUUM ANALYSE on all databases every 5 hours if pg_autovacuum is not
# running
# 2 0,5,10,15,20 * * 1-6 root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --analyze >/dev/null; fi

# On Sunday you may wish to run a VACUUM FULL ANALYSE as well
# If you do not run a 24/7 site, you may want to uncomment the next line 
# so as to do a regular VACUUM FULL.  If you need 24/7 connectivity, save
# VACUUM FULL for when you think you really need it.
# 10 3 * * Sun root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --full --analyze >/dev/null; fi

Don't forget to reenable them if you're doing anything else with postgres, or performance will slowly degrade.

Access denied

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 );
 failed: ERREUR:  droit refusé pour la relation geometry_columns
CONTEXT:  instruction SQL « DELETE FROM geometry_columns WHERE
		f_table_catalog = '' AND f_table_schema = 'public' AND f_table_name = 'planet_osm_point' AND f_geometry_column = 'way' »
PL/pgSQL function "addgeometrycolumn" line 94 at EXECUTE statement
instruction SQL « SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4 , $5 ) »
PL/pgSQL function "addgeometrycolumn" line 4 at SQL statement

Error occurred, cleaning up

The user is not allowed to modify the gis database. You have to run osm2pgql as the user you created in the postgresql database setup.

Authentication failed

If you can't connect to postgres with the psql command (this happens, for example, with Ubuntu 9.10 and postgres 8.3):

 $ psql -W gis username
 psql: FATAL:  Ident authentication failed for user "username"

Even after you've set the password for the user:

 $ sudo -u postgres -s -H
 postgres@aoeu$ psql 
 Welcome to psql 8.3.5, the PostgreSQL interactive terminal.
 ALTER ROLE username PASSWORD 'mypassword';

You may have to append the following lines to your pg_hba.conf (found e.g. in /etc/postgresql/8.3/main):

 local   all         all                               trust   
 host    all         all         127.0.0.1/32          trust
 host    all         all         ::1/128               trust

So that you can use password authentication from localhost.

Note that you may have to comment out any existing lines beginning with local all or host all. I had to comment out the following section on Ubuntu 8.10:

# "local" is for Unix domain socket connections only 
#local   all         all                               ident sameuser
# IPv4 local connections:
#host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
#host    all         all         ::1/128               md5

Unknown element

If you receive the error message

Unknown Element name : changeset

then you need to update your osm2pgsql to a recent version.

Rendering with Mapnik

Unpack the lowzoom and coastline data you downloaded earlier, the latter two zip files needed since June 2010:

cd ~/mapnik
tar xzf world_boundaries-spherical.tgz
tar xjf processed_p.tar.bz2 -C world_boundaries
tar xjf shoreline_300.tar.bz2 -C world_boundaries
unzip 10m-populated-places.zip -d world_boundaries
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries

The above should leave you with all the shapefiles in the world_boundaries directory.

If you haven't already done so, export the Mapnik stuff from the OSM subversion repository:

svn export http://svn.openstreetmap.org/applications/rendering/mapnik
cd mapnik

The osm.xml file is built up out of the core osm.xml file and XML include files in the inc/ directory. The inc/ directory holds 3 .template files, which are used as inputs for the generate_xml.py script:

./generate_xml.py -h

Check to see if the defaults that the script provides are ok for you. You'll have to provide the values where there are no defaults. For instance:

./generate_xml.py --host localhost --user `whoami` --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --port 5432 --password 'something'

This command takes the .template files from inc/ and creates non-template versions of those files, with your settings. If you can not or do not want to run the script, you can do this by hand as well. Updating the stylesheet from SVN is now as easy as running an 'svn update'. SVN will never touch the generated settings files. If there is ever a new option added to the .template files, only then do you need to run the generate_xml.py script again.

After you have set up the stylesheet this way, you can point mapnik to the osm.xml file. The osm.xml file draws in the files in the inc/ directory, through the use of XML entities. You do not need to create another copy of the stylesheet.

You can decide which areas get rendered by setting bounding boxes and zoom levels in generate_(tiles|image).py

Optional: CJK font fallback support

Currently, mapnik includes a font to support CJK characters (Chinese, Japanese, Korean, et. al. or more generally East Asia). To add support you need to open the file inc/fontset-settings.xml.inc and uncomment the lines that have "unifont Medium".

Older mapnik versions did not have unifont included. There you need to:

1. Download the GNU Unifont Glyphs
2. Unpack and put the ttf file in your /usr/local/lib/mapnik/fonts (or appropriate path) directory with the other Mapnik fonts.

Rendering tiles

You can now call

MAPNIK_MAP_FILE="osm.xml" MAPNIK_TILE_DIR="tiles/" ./generate_tiles.py

generate_tiles.py generates tiles which can be served by a suitable server. (Look inside the file to find out what area of the planet it will render by default and at what zoom. Look for bbox and min,max zoom. If you want to select a different area go to openstreetmap.org and click the "export" tab; then "manually select a different area". Draw the area you want, and the four text boxes will show values you should provide in a sequence [left,bottom,right,top]). If you have multi-core machine, to speed-up generation of tiles, you can divide an area for e.g. four squares and run few instances in the same time. Further speed improvements are possible by reducing disk I/O bottlenecks (iowait time). These bottlenecks can for example be found with iofile.d; on Mac OS X this returns Spotlight, and Postgress statistics collector as parallel disk I/O consumers.

or

./generate_image.py

generate_image.py generates a single image for a given bounding box. That is, it essentially gets the tiles and stitches them into a single large image. This may be useful for people who want to just cover a limited area and thus want to save the hassle of handling tiles.

Troubleshooting

There are several possible problems you may encounter in your rendered tiles or while rendering tiles:

1. If the generated tiles or image show coastlines, but lack streets, street names, cities, and other features, please verify that the database user can indeed login. The scripts and Mapnik may quietly fail to read the data and thus render with coastlines, but no other data. Another cause is building Mapnik without PostGIS support.
2. The version of the last two scripts currently in svn is for use with Mapnik 0.5 and you will get some errors if you try to use them with an older version. (These scripts may try to invoke the 'convert' program. You may need to install ImageMagick or choose not to use 'convert'.)
3. If Asian place names appear as blocks instead of normal characters, then you haven't enabled the unifont font in the stylesheet. Edit inc/fontset-settings.xml.inc and uncomment the lines with unifont. The unifont font has been bundled with mapnik since 0.7.0.

   To make sure, that unifont is properly set up in the right folders, you can run the command line: ls `python -c "import mapnik;print mapnik.fontscollectionpath"`. You can then also check the font face names that are available with: > python -c "from mapnik import FontEngine as e;print '\n'.join(e.instance().face_names())"

4. If you are using the ext4 file system (the default in distributions such as openSUSE 11.2 and Ubuntu 9.10) and you are generating many tiles, it can occur that the script crashes because it can't create new tile files. This is a file system issue. Delete as many files as you can, force a file system check and/or switch to a more stable file system.
5. this error can happen is some unix distributions:

 untimeError: PSQL error:
 FATAL:  Ident authentication failed for user "username" (encountered during parsing of layer 'leisure')

On most Unix installs postgres is setup in the default configuration to work using the 'ident same user' authentication. To use this remove the following params from the config: user, port, host, password. Delete all the parameter lines for these, for example:

 ./generate_xml.py --accept-none --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/

Then mapnik will connect the same way as if you ran "psql gis" from the command line (without specifying a host, username or password).

Reducing disk space

If you are rendering the entire earth (that is, bbox = (-180.0, -90.0, 180.0, 90.0)) there are a few things you can do to dramatically reduce disk space usage.

1. Verify that you only have tiles you will use.
   1. For instance, the maximum x and y values for zoom 10 are 1023. So, the last tile should be TILEDIR/10/1023/1023.png.
   2. However, you may have tiles rendered with x and y values beyond that. (I had y values up around 1300)
   3. Delete those first for each zoom level.
2. At higher zoom levels (8 or so), many tiles are duplicated (e.g., empty oceans or lakes, and land areas with no features).
   1. On operating systems supporting it, you can create symbolic links to one set of blank water and land tiles.

Example savings (for me (athlon02)):

1. At zoom 9, the disk space used was 3.8 GB (if memory serves). It is now 1.4 GB.
2. At zoom 10, the disk space used was 12-13 GB to start. It is now 4.58 GB.
3. Your mileage may vary.

Customizing the rendering

Now you have the standard stylesheet rendering working you can start customizing your map.

Contours

It is possible to add contours to your map like the cyclemap has, the contours are based on the public domain SRTM (Shuttle Radar Topograhy Mission) from NASA. It is also possible to add hillshading and elevation colouring to your map based on the same data.

Stylesheet

The stylesheet is an XML document controlling most aspects of how features are rendered, such as color and widths of lines showing different types of roads. The format is not very compact or easy to understand, but it does give a lot of power/flexibility. The version we use on the live slippy map is probably the osm.xml file in the SVN head:applications/rendering/mapnik/osm.xml . See Mapnik Example for an example of a more simple stylesheet.

The stylesheet syntax is specified here: XMLConfigReference. MinScaleDenominator and MaxScaleDenominator specify the minimum and maximum scales at which each symbol/feature should be rendered.

You can play around with mapnik stylesheets online here: http://dev.openstreetmap.nl/~panman/styledit/

Look at Cascadenik and Spreadnik if you plan to develop complex stylesheets.

Regionalised rendering of names

To render name:* names instead of the standard name tags where to available you can either preprocess the data or modify the postgis database afterwards.

Customising default.style

If you're editing default.style, for instance to add cycle lanes or network data to the database, note that osm2pgsql does not use the default.style in the working directory, instead looking for /usr/share/osm2pgsql/default.style.

You will need to use the --style parameter to specify your own default.style file. Use .osm2pgsql -h to display more information.