Mapnik
From OpenStreetMap
| Other languages: | +/- |
|---|---|
Mapnik is an Open Source map renderer which we use to render the main Slippy Map layer for OSM.
The official homepage is mapnik.org. Mapnik is a free toolkit for rendering maps. It's written in C++ and there are Python bindings. It uses the AGG library and offers anti-aliasing rendering with subpixel accuracy. It can read ESRI shapes, PostGIS and TIFF raster. Currently it is easiest to build mapnik from source on linux because all the dependencies can be easily fetched with apt-get. However as of the 0.5 release binaries are available for windows and mapnik has been successfully built on Mac OS 10.4 and 10.5.
OSM data is converted to a format usable by Mapnik by osm2pgsql, which loads derived data into a PostGIS database. Based on this and some other data sources, OSM uses Mapnik to render 256 × 256 px tiles, which are served from the tile server (tile.openstreetmap.org). The Slippy Map JavaScript references URLs such as e.g. http://tile.openstreetmap.org/7/63/42.png for this image:
See also Slippy map tilenames
Contents |
Data Source
The main data source for the main Mapnik layer is the OSM data, converted using osm2pgsql. But Mapnik uses more than just OSM's data to create its maps. Coastlines, for example, are created using data from VMAP0. See Coastline#Main_Mapnik_Layer
Bugs and rendering suggestions
Please report any bugs, graphical suggestions and requests for unrendered features to trac under the "slippy_map" component. Please check if the ticket has been filed before: trac tickets with component "slippy_map".
Preparation
This is a work in progress.
This process will require quite a bit of diskspace if you plan on loading the entire world. Individual countries can be handled on any ordinary machine, but for the whole planet here are some indications of the requirements (as at 21 Apr 2008). This includes the indexes:
| Table | Size |
|---|---|
| roads | 575MB |
| line | 6.25GB |
| point | 154MB |
| polygon | 191MB |
Which totals about 7GB. If you are using slim-mode in osm2pgsql you will also need around 20GB for the temporary tables.
If you have enough memory (>3GB) then you can try running in normal mode. On a suitable machine the process should complete in 3-4 hours. If you don't have enough memory then you need to use the slim-mode which will work with any amount of memory you can give it. Configured for 1.5GB RAM the process has been measured at around 13-14 hours, though it is strongly dependant on your disk subsystem.
Don't forget you count the 5GB for the planet dump itself!
DataSets
There are two large datasets needed to render tiles, so its probably a good idea to start those downloading first.
Planet.osm
Planet.osm is a snapshot of the OpenStreetMap database. See Planet.osm
$ wget http://planet.openstreetmap.org/planet-latest.osm.bz2
Note: If you want to render a small area as a quick test then you can import any .osm file, you do not need to render the whole planet. This is recommended if you want to quickly prove that the rendering is working before you spend several hours downloading 3GB+ and importing the whole planet. You can get a suitable .osm file by downloading and saving an area in JOSM or use one of the smaller planet extracts listed at Planet.osm#Extracts.
World Boundaries
Mapnik uses external data for coastlines at lower zoom levels. See Coastline#Main_Mapnik_Layer
$ wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz (51MB) $ wget http://hypercube.telascience.org/~kleptog/processed_p.zip (~220MB)
PostgreSQL / PostGIS
The below assumes you will replace username with the name of the user who will be running mapnik
Install
For Debian Sarge (old stable)
As root, then following the etch instructions
# echo deb http://www.backports.org sarge-backports main >> /etc/apt/sources.list # apt-get -t sarge-backports install ...
For Debian etch (onwards)
To install PostGIS (as root):
# aptitude install postgresql-postgis
or with unstable
# aptitude install postgresql-8.3-postgis
or with Ubuntu:
# sudo apt-get install postgresql-8.3-postgis
For Fedora
To install PostGIS (as root):
# yum install postgis postgresql-server # service postgresql initdb # service postgresql start
Tuning the database
Since we are predominantly doing bulk loads and handling lots of data, it is advisable to tune Postgres a bit for this load; the default settings are generally fairly conservative. The following is just a brief overview of the settings you need to think about, read the documentation for more info. Remember at all times that the rest of your system (including a webserver for example) also needs memory to run, so don't starve them out just for PostgreSQL. You need to edit the file postgresql.conf which may be in /etc/postgresql/8.3/main or /var/lib/pgsql/data or similar depending on your distribution. The server needs to be restarted after the changes.
If you are using a version older than 8.2 you will need to use the numbers rather than the more descriptive units.
shared_buffers = 128MB # 16384 for 8.1 and earlier
This controls the amount of memory used for stored things like index pages, data pages directly into local cache. More is not necessarily better. Remember that osm2pgsql needs lots of memory too and this memory is permanently reserved for PostgreSQL.
If the server won't start after making this change, check for errors appearing in pgstartup.log about exceeding the kernel SHMMAX variable. If this occurs, increase the kernel.shmmax variable using the sysctl command sysctl -w kernel.shmmax=268435456 (increasing to 256MB) and editing /etc/sysctl.conf to make it permanent.
checkpoint_segments = 20
The default is far too low for bulk inserts. It costs more diskspace this way but the load will be smoother; it depends a bit on your disk subsystem. To determine the optimal setting you need to play a bit. Check your postgres logs for the following:
LOG: checkpoints are occurring too frequently (28 seconds apart) HINT: Consider increasing the configuration parameter "checkpoint_segments".
For advanced users: you can also tune the bgwriter to smooth performence.
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
This setting is the amount of memory used while building indexes. The default is far too small for the size of tables we're dealing with. This memory is released after completion so you don't need to be as stingy as above. Note that osm2pgsql is still large at this point so you can't overdo it.
Create Database
On most systems you need to be authenticated as the PostgreSQL super user in order to execute many of the commands below. Often you can not login as this user. Instead you should issue a command like this as root:-
# su - postgres $ <commands...>
or
# sudo -u postgres <command>
or on a system like Ubuntu which does not want to let you run as root either
user@machine$ sudo -u postgres -i postgres@machine$ <commands...>
Now as the postgres user run the following to setup the database:
$ createuser username $ createdb -E UTF8 -O username gis $ createlang plpgsql gis
PostGIS
For Debian Etch (stable)
As the postgres user or your postgresql super user:
$ psql -d gis -f /usr/share/postgresql-8.1-postgis/lwpostgis.sql $ echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
For Debian Lenny or Sid (testing or unstable)
As the postgres user or your postgresql super user:
$ psql -d gis -f /usr/share/postgresql-8.2-postgis/lwpostgis.sql
(debian sid) (and for ubuntu)
$ psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
And also for all of them
$ echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
For Fedora
As the postgres user or your postgresql super user:
$ psql -d gis -f/usr/share/pgsql/contrib/lwpostgis.sql $ echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
osm2pgsql
osm2pgsql is a Openstreetmap data to PostgreSQL converter and loader
For Debian Etch (stable)
Configure your system with access to the backports.org repository (see http://www.backports.org/).
# apt-get update # apt-get install osm2pgsql
Feel free to contact dom@earth.li with questions about these packages.
For Debian Lenny or Sid (testing or unstable)
osm2pgsql is already packaged so you can just install it!
aptitude install osm2pgsql
Fedora
Fetch the dependencies
yum install geos-devel proj-devel postgresql-devel
Follow "From source" instructions
From source
$ svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/ $ cd osm2pgsql $ make
When compiling under Ubuntu, you will need the following dependencies:
$ sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev
If using an old version of gcc then you may get an error: unrecognized option `-Wextra'. In this case simply amend the Makefile to remove this unnecssary option.
Mapnik
For Debian Etch (stable) (Mapnik 0.5)
Configure your system with access to the backports.org repository (see http://www.backports.org/).
# apt-get update # apt-get install python-mapnik
Feel free to contact dom@earth.li with problems about the Debian packages.
For Debian Lenny or Sid (testing or unstable)
Mapnik is already packaged so you can just install it!
# aptitude install python-mapnik
Compiling the new Mapnik 0.5 from source on Debian Etch (stable)
Besides the dependencies listed above for Mapnik 0.4, you need iostreams from the boost library
# aptitude install libboost-iostreams-dev
Next you need to create some symbolic links. The last one for GDAL is not needed for OSM, but if you have it installed on your system and want to use it with Mapnik, then you need that link too.
# cd /usr/lib # ln -s libboost_filesystem.so libboost_filesystem-mt.so # ln -s libboost_regex.so libboost_regex-mt.so # ln -s libboost_iostreams.so libboost_iostreams-mt.so # ln -s libboost_program_options.so libboost_program_options-mt.so # ln -s libboost_thread.so libboost_thread-mt.so # ln -s libboost_python.so libboost_python-mt.so # ln -s libgdal1.3.2.so libgdal.so
Next compile and install Mapnik. "XMLPARSER=libxml2" at the end of each line, is only needed if you want to use XML entities in your style sheets.
$ cd /path/where/you/unpacked/the/mapnik/0.5/source/ $ python scons/scons.py PGSQL_INCLUDES=/usr/include/postgresql PROJ_INCLUDES=/usr/include PROJ_LIBS=/usr/lib XMLPARSER=libxml2 # python scons/scons.py install PGSQL_INCLUDES=/usr/include/postgresql PROJ_INCLUDES=/usr/include PROJ_LIBS=/usr/lib XMLPARSER=libxml2
By default Mapnik is installed in /usr/local/lib so you need to add that path to /etc/ld.so.conf and run 'ldconfig' afterwards.
# echo "/usr/local/lib" >> /etc/ld.so.conf # ldconfig
For Fedora
At present there are twoRequests in the Redhat Bugzilla to get Mapnik included in Fedora. Until they make it you'll have to do a little leg work.
Get the mapnik deps using the following command (as root):
# yum install rpm-build rpmdevtools python python-devel boost-devel fribidi-devel libtool-ltdl-devel libpng-devel libjpeg-devel libtiff-devel zlib-devel freetype-devel proj-devel gdal-devel libpqxx-devel boost zlib freetype proj gdal
To build the SRPM for Fedora:
$ wget http://snecker.fedorapeople.org/mapnik/f8/mapnik-0.5.0-1.fc8.src.rpm $ rpmdev-setuptree $ rpm -i mapnik-0.5.0-1.fc8.src.rpm $ rpmbuild -bb ~/rpmbuild/SPECS/mapnik.spec
You can then install Mapnik (as root) replacing i386 with your arch:
# rpm -i [path-to-user-that-built-mapnik]/rpmbuild/RPMS/i386/mapnik-*
Loading Data
Simply call osm2pgsql (as the user who will be running mapnik):
$ osm2pgsql -m -d gis planet-latest.osm.bz2
This process requires significant amounts of free memory (by rough estimate, at least 512 MB). If it runs out of memory, it can fail in somewhat unpredictable ways (e.g. messages suggesting that errors exist in the .osm file). Make sure that you have sufficient free RAM and swap space available before running osm2pgsql.
The "-m" option enables the use of the Spherical Mercator projection which is recommended. This is the default since version 0.55, SVN r8320. Some older tools may still expect the data to be imported in a different projection such as "latlong" or the non-spherical Mercator. There are command line options available to select these and other projections.
The latest osm2pgsql code also supports a simple bounding box filter during the import which will make things a lot faster if you only want to render a small part of the planet. For example if you just wanted to import London from the UK planet extract:-
$ osm2pgsql --bbox -0.5,51.25,0.5,51.75 -m -d gis uk-080213.osm.bz2
Some issues you may encounter
If all your map tiles appear to be off by 10 - 20km in the North/South direction then you may have forgotten to add the "-m" option on the osm2pgsql import. Since osm2pgsql version 0.55 the spherical mercator (-m) is the default projection.
If you attempt to load a gzip'd planet file which is more than 2GB you may see an error that it fails to open the file. This is due to a bug in some versions of zlib which fail to supply the O_LARGEFILE flag to the open(). The workaround is to use an external gzip process which does not suffer this bug. Specify the filename "-" will make osm2pgsql read the data from STDIN.
$ gzip -dc planet-080213.osm.gz | osm2pgsql -m -
After several hours the import may quit with an error like:-
ANALYZE planet_osm_line; failed: ERROR: deadlock detected DETAIL: Process 28511 waits for AccessExclusiveLock on relation 1064115 of database 18309; blocked by process 12776. Process 12776 waits for ShareLock on transaction 572766655; blocked by process 28511. Error occurred, cleaning up
This seems to be a fault in some versions of PostgreSQL and is caused when an auto-vacuum is attempted during the ANALYZE. The solution is to disable all auto-vacuums on the database. The data is not updated after the import so the vacuum process does nothing useful. In the postgresql.conf file set the option:
autovacuum = off
Then restart the database server
# /etc/init.d/postgresql-8.1 restart
Note: In Debian/Ubuntu you also need to update /etc/cron.d/postgresql-common to comment out the two pg_maintenance tasks which would otherwise cause vacuums to occur at regular intervals:-
# Run VACUUM ANALYSE on all databases every 5 hours if pg_autovacuum is not # running # 2 0,5,10,15,20 * * 1-6 root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --analyze >/dev/null; fi # On Sunday you may wish to run a VACUUM FULL ANALYSE as well # If you do not run a 24/7 site, you may want to uncomment the next line # so as to do a regular VACUUM FULL. If you need 24/7 connectivity, save # VACUUM FULL for when you think you really need it. # 10 3 * * Sun root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --full --analyze >/dev/null; fi
Don't forget to reenable them if you're doing anything else with postgres, or performance will slowly degrade.
Rendering with Mapnik
Unpack the lowzoom and coastline data you downloaded earlier:
$ tar xzf world_boundaries-spherical.tgz $ unzip processed_p.zip $ mv coastlines/* world_boundaries/ $ rmdir coastlines
The above should leave you with all the shapefiles in the world_boundaries directory.
Checkout the Mapnik stuff from the OSM subversion repository:
$ svn checkout http://svn.openstreetmap.org/applications/rendering/mapnik $ cd mapnik
Create a local copy of the "set-mapnik-env" script and edit it to fit your local file names etc. All the settings are documented inside this script.
Create the osm.xml file from osm-template.xml by calling:
$ source ./set-mapnik-env $ ./customize-mapnik-map >$MAPNIK_MAP_FILE
You can decide which areas get rendered by setting bounding boxes and zoom levels in generate_(tiles|image).py
You can now call
$ ./generate_tiles.py
or
$ ./generate_image.py
to generate maps.
The version of the last two scripts currently in svn is for use with mapnik 0.5 and you will get some errors if you try to use them with an older version.
Stylesheet
The stylesheet is an XML document controlling most aspects of how features are rendered, such as color and widths of lines showing different types of roads. The format is not very compact or easy to understand, but it does give a lot of power/flexibility. The version we use on the live slippy map is probably the osm.xml file in the SVN head:applications/rendering/mapnik/osm.xml
You can play around with mapnik stylesheets online here: http://tile.openstreetmap.nl/~panman/styledit/
Old Stuff
Old, out of date information, and possibly stuff that simply hasn't been included above yet can be found at Mapnik/Old
