Mapnik
Mapnik
| |
|---|---|
| Website: | http://www.mapnik.org |
| Version: | 2.1.0 (2012-08-24) |
| License: | LGPL |
| Platform: | win ; macos ; linux |
Mapnik is an open source toolkit for rendering maps. Among other things, it is used to render the four main Slippy Map layers on the OpenStreetMap website. It supports a variety of geospatial data formats and provides flexible styling options for designing many different kinds of maps.
Mapnik is written in C++ and includes high-level Python bindings. It uses the AGG library and offers anti-aliasing rendering with subpixel accuracy. It can read ESRI shapefiles, PostGIS, TIFF rasters, .osm files, and any GDAL or OGR supported formats. Packages are available for most Linux distributions and binaries are available for Mac OS X and Windows.
You can learn more about using Mapnik at the The Mapnik Wiki and by joining the #mapnik channel on freenode
Note that Mapnik and the Standard OSM Mapnik style are separate projects. To report bugs or graphical suggestions for the main map on the OSM webside, do so in the 'mapnik' component of OSM's trac. Report problems, ideas, enhancements about the Mapnik software in general on GitHub. Please check that someone else hasn't reported the same issue first.
Contents |
Overview
Mapnik can output map images to a variety of graphics formats - PNG, JPEG, SVG, and PDF. OpenStreetMap's primary use of Mapnik involves rendering many thousands of 256 × 256 pixel tiles which are displayed in a JavaScript Slippy Map interface.
Mapnik Styles
Mapnik allows for customization of all the cartographic aspect of a map - data features, icons, fonts, colors, patterns, and even certain effects such as pseudo-3d buildings and drop shadows. This is all controlled by defining datasources and style rules, most commonly in an XML language specific to Mapnik.
The Mapnik style rules used for the Standard OpenStreetMap are open-source and can be used as the basis for custom renderings of OSM data (see OSM Standard Mapnik Style below). Other styles are available as well, such as those used to render MapQuest Open.
There are a number of external tools that can assist in the creation of Mapnik styles. Cascadenik, Spreadnik, and TileMill offer styling languages that are more compact and easier to read and write than Mapnik's built-in XML language. QGIS also has a plugin called Quantumnik for creating Mapnik styles using a more graphical interface.
Data Sources
Mapnik can use data from different sources: it can directly process OSM data, PostGIS databases, shapefiles and more.
PostGIS
PostGIS is the most common approach for rendering OSM data with Mapnik. OSM can be loaded by a tool such as osm2pgsql or Imposm and accessed via SQL queries and GIS functions defined in a Mapnik style. This approach can be used for more advanced renderings, and is the main datasource used by the Standard OpenStreetMap layer.
Shapefile
Shapefiles are a common storage and exchange format for geographic data. In addition to PostGIS, the Standard OpenStreetMap style uses several shapefiles to render the map. For example, land masses are drawn using shapefiles generated from the natural=coastline ways by the Coastline Error Checker. See Coastline#Rendering.
GeoTIFF
Mapnik can render GeoTIFF files as raster images. This is commonly used for relief maps. See Hillshading with Mapnik for a step-by-step tutorial.
Raster images
In a similar way to GeoTIFF, Mapnik can also render bitmap images which contain no geocoding information. The bounding box for these images has to be specified through separate parameters.
OSM XML
Mapnik has some limited capability for rendering directly from an OSM XML file, but this is not generally recommended. Still, it may be useful for testing Mapnik installations or for ad-hoc rendering of a map. For more information, see Mapnik: Rendering OSM XML data directly.
Installation
Detailed installation instructions for many platforms are maintained on the Mapnik Github wiki.
OSM Standard Mapnik Style
You can set up a copy of the Standard OpenStreetMap Mapnik style for your own rendering. You can either render it as-is, or make modifications for your own custom map.
Set up prerequisite software
Standard OSM Mapnik style requires Mapnik version 2.0 or higher. You will also need to set up a few other things:
- a PostGIS-enabled PostgreSQL database (setup instructions)
- osm2pgsql (setup instructions)
- the OSM Mapnik style files (see instructions below)
To get the OSM Mapnik style files you can download and unzip an archive of the project, or check out a copy in a version control system:
git clone git://github.com/openstreetmap/mapnik-stylesheets.git # or svn export http://svn.openstreetmap.org/applications/rendering/mapnik
Download required data
You can download a full copy of the OSM data, or a smaller extract. Smaller extracts are recommended for getting started as the full planet is around 24 GB. Smaller areas will import easily on an ordinary desktop or laptop computer and are still useful for testing. They can also be useful if you only need a map of a limited area.
Note: If you plan on loading the entire world, the data will require quite a bit of disk space. As of August 2012, a full 'slim mode' PostgreSQL import requires at least 330 GB for all of its tables and indexes.
Whether you download the entire planet or just a small area, using the PBF Format is recommended as it will be a smaller file size and process more quickly.
In addition to data that will be stored in a PostGIS database, the Standard OSM Mapnik style relies on several external shapefiles. A script is provided for getting these shapefiles:
./get-coastlines.shThis script downloads and extracts the following files:
- world_boundaries-spherical.tgz (53 MB)
- processed_p.tar.bz2 (410 MB)
- shoreline_300.tar.bz2 (46 MB)
- ne_10m_populated_places.zip (1.5 MB)
- ne_110m_admin_0_boundary_lines_land.zip (38 KB)
The script should leave you with all the shapefiles in the world_boundaries directory. You don't need the tgz, bz2 or zip files any more, maybe tuck them out of the way for now.
Populate the PostGIS Database
You need to transform the planet file or extract that you downloaded into a PostGIS database that Mapnik can render from. This is done from the terminal osm2pgsql (as the user who will be running mapnik):
osm2pgsql --database gis --username gisuser --slim planet-latest.osm.pbf
Adjust the --database (or -d) and --username (or -U) paramaters if you set up your PostgreSQL database differently.
If you installed osm2pgsql from a package, your import style may be out of date. make sure you have and use the latest version by downloading it to your mapnik-stylesheets directory and adding the --style (or -S) option to your osm2pgsql command to point to this up-to-date file. (Adjust the directory as necessary.)
osm2pgsql --style ~/mapnik-stylesheets/default.style --database gis --username gisuser --slim planet-latest.osm.pbf
Another potentially useful option is the --bbox (or -b) parameter, which you can use to limit your import to a specific bounding box smaller than your overall data file. For example, to just import London from a larger extract of Great Britain, the command might be:
osm2pgsql --bbox -0.5,51.25,0.5,51.75 --database gis --username gisuser --slim great_britain.osm.pbf
Configure the Mapnik style
The OSM Standard Mapnik style is built up out of a core osm.xml file and many additional XML include files in the inc/ directory. These files won't work out of the box - you'll need to check some configurations and run a build script first. (This is to account for different database setups and directory structures.)
The build script is generate_xml.py. You can see the options it takes with:
./generate_xml.py -h
Check to see if the defaults that the script provides are ok for you. You'll have to provide the values where there are no defaults. For instance:
./generate_xml.py --host localhost --user gisuser --dbname gis \ --symbols ./symbols/ --world_boundaries ./world_boundaries/
This command takes the .template files from inc/ and creates non-template versions of those files, with your settings. If you can not or do not want to run the script, you can do this by hand as well.
Example using default values (change user and password if necessary):
./generate_xml.py --host localhost --port 5432 --dbname gis --user `whoami` --password ''
Example for empty passwords and local connections (change user if necessary):
./generate_xml.py --dbname gis --user `whoami` --accept-none
If you've checked out the project from version control, you can safely update it (eg. 'git pull' or 'svn update') without disrupting your settings. If there is ever a new option added to the .template files, only then do you need to run the generate_xml.py script again.
Unifont fallback support
Mapnik includes a font to support a very wide range of writing systems, but the Standard OSM Stylesheet does not use this by default. To add support for this (and avoid broken labels in places like East Asia) you need to edit the file inc/fontset-settings.xml.inc to uncomment the lines that contain "unifont Medium".
Regionalised rendering of names
To render language-specific name:* tags instead of the standard name tags (where available) you can either preprocess the data or modify the PostGIS database afterwards. See Regionalisedmap for details.
Tile Rendering Scripts
Because Mapnik is a map rendering library, rendering images is usually handled by external programs. The OSM Standard Mapnik style includes a few Python scripts to help out with rendering. There are also external programs that can handle rendering Mapnik styles, such as Tirex, Mod tile, TileStache, and TileCache.
The generate_tiles.py' and generate_tiles_multiprocess.py scripts generates tiles which can be served by a suitable server and displayed by Leaflet, OpenLayers, and other things. To run one of them with its default settings:
MAPNIK_MAP_FILE="osm.xml" MAPNIK_TILE_DIR="tiles/" ./generate_tiles.py
The program will render tiles for a set of bounding boxes and zoom levels defined in the script. You can change these by editing the script.
Optimizing
Multiple processes
If you have multi-core machine, you can speed-up generate_tiles.py by dividing up an area for e.g. four squares and run few instances in the same time. Further speed improvements are possible by reducing disk I/O bottlenecks (iowait time). These bottlenecks can for example be found with iofile.d; on Mac OS X this returns Spotlight, and Postgress statistics collector as parallel disk I/O consumers.
Reducing disk space
If you are rendering the entire earth (that is, bbox = (-180.0, -90.0, 180.0, 90.0)) there are a few things you can do to dramatically reduce disk space usage.
- Verify that you only have tiles you will use.
- For instance, the maximum x and y values for zoom 10 are 1023. So, the last tile should be TILEDIR/10/1023/1023.png.
- However, you may have tiles rendered with x and y values beyond that. (I had y values up around 1300)
- Delete those first for each zoom level.
- At higher zoom levels (8 or so), many tiles are duplicated (e.g., empty oceans or lakes, and land areas with no features).
- On operating systems supporting it, you can create symbolic links to one set of blank water and land tiles.
- But, if your OS supports better use of hard links - this will give a much greater space savings on modern HDD, where the sector size of 4K, the size of the tile fill in the 103B.
Example savings (for me (athlon02)):
- At zoom 9, the disk space used was 3.8 GB (if memory serves). It is now 1.4 GB.
- At zoom 10, the disk space used was 12-13 GB to start. It is now 4.58 GB.
- Your mileage may vary.
Upgrading to Mapnik 2
If you have stylesheets that were created for versions of Mapnik older than 2.0, you will need to update them in order to render them Mapnik 2.0 or newer. Some changes in the specification of stylesheets were been made; these can be automatically applied with an upgrade script included with new versions of mapnik. See the Upgrade Guide section on the Mapnik 2 wiki page.
Troubleshooting
Import Issues
Locked out
After several hours the import may quit with an error like:-
ANALYZE planet_osm_line; failed: ERROR: deadlock detected DETAIL: Process 28511 waits for AccessExclusiveLock on relation 1064115 of database 18309; blocked by process 12776. Process 12776 waits for ShareLock on transaction 572766655; blocked by process 28511. Error occurred, cleaning up
This seems to be a fault in some versions of PostgreSQL and is caused when an auto-vacuum is attempted during the ANALYZE. The solution is to disable all auto-vacuums on the database. The data is not updated after the import so the vacuum process does nothing useful. In the postgresql.conf file set the option:
autovacuum = off
Then restart the database server
sudo /etc/init.d/postgresql-8.1 restart
Note: In Debian/Ubuntu you also need to update /etc/cron.d/postgresql-common to comment out the two pg_maintenance tasks (add a '#' in front of them) which would otherwise cause vacuums to occur at regular intervals:-
# Run VACUUM ANALYSE on all databases every 5 hours if pg_autovacuum is not # running # 2 0,5,10,15,20 * * 1-6 root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --analyze >/dev/null; fi # On Sunday you may wish to run a VACUUM FULL ANALYSE as well # If you do not run a 24/7 site, you may want to uncomment the next line # so as to do a regular VACUUM FULL. If you need 24/7 connectivity, save # VACUUM FULL for when you think you really need it. # 10 3 * * Sun root if [ -x /usr/sbin/pg_maintenance ]; then /usr/sbin/pg_maintenance --full --analyze >/dev/null; fi
Don't forget to reenable them if you're doing anything else with postgres, or performance will slowly degrade.
Access denied
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 );
failed: ERROR: permission denied for relation spatial_ref_sys
CONTEXT: SQL statement "SELECT SRID FROM spatial_ref_sys WHERE SRID = $1 "
PL/pgSQL function "addgeometrycolumn" line 74 at SQL statement
SQL statement "SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4 , $5 )"
PL/pgSQL function "addgeometrycolumn" line 4 at SQL statement
The user is not allowed to modify the gis database. You have to run osm2pgsql as the user you created in the postgresql database setup.
Rendering Issues
See also: the Mapnik wiki Troubleshooting page.
Map is empty
If the generated tiles or image show coastlines, but lack streets, street names, cities, and other features, please verify that the database user can indeed login. The scripts and Mapnik may quietly fail to read the data and thus render with coastlines, but no other data. Another cause is building Mapnik without PostGIS support.
Authentication failure
This error can happen is some unix distributions:
untimeError: PSQL error: FATAL: Ident authentication failed for user "username" (encountered during parsing of layer 'leisure')
On most Unix installs postgres is setup in the default configuration to work using the 'ident same user' authentication. To use this remove the following params from the config: user, port, host, password. Delete all the parameter lines for these, for example:
./generate_xml.py --accept-none --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/
Then mapnik will connect the same way as if you ran "psql gis" from the command line (without specifying a host, username or password).
See also
- Create a map from scratch: An introductory example
- Deutsche Anleitung User:Ajoessen/Mapnik
- Invoke Mapnik via Commandline nik2img
- Mapnik is used by the rendering framework TileMill.
- Renderd, Tirex as tile servers with Mapnik
- Mapnik2Geotools converts style for Geoserver
- Shaded relief maps using mapnik
- Keep the PostgreSQL database in sync with changes made on the OpenStreetMap server: Minutely Mapnik
