Mod tile

Mod tile is a system to serve raster tiles for example to use within a slippy map. It provides a dynamic combination of efficient caching and on the fly rendering. Due to its dynamic rendering, only a small fraction of overall tiles need to be kept on disk, reducing the resources required. At the same time, its caching strategy allows for a high performance serving and can support several thousand requests per second.

Mod_tile was originally written for serving the tiles of the main OSM map (Mapnik layer), but since is being used on a variety of different servers providing maps ontop of OpenStreetMap data.

Contents 1 Implementation 2 Features provided by mod_tile 3 Installation: 3.1 Mapnik rendering toolchain 3.2 mod_tile 3.3 renderd 3.4 tile expiry 3.5 HowTos 4 Configuration 5 Testing and debugging installation 6 Source code 7 Bug reports and feature requests 8 TODO 9 More ideas 10 TIPS

Implementation

The code is split into two parts:

• mod_tile - A custom Apache module which is responsible for serving tiles and requests the rendering of tiles if they aren't already available in cache or if they have changed since.
• A rendering back-end, that takes requests from mod_tile, and renders them to the file system. It is also responsible for queueing requests to ensure the server is not overloaded if many requests come in simultaneously. Currently there are two implementations of such a rendering back-end: The original renderd and tirex.

Features provided by mod_tile

• high performance serving of tiles
• management of dynamic rendering requests
• support for multiple different styles
• expiry and rerendering of individual tiles
• dynamic adjustment of cache_expiry headers
• a number of munin graphs to monitor the status and health of the server
• throttling of individual clients (by ip-address) to a fair share quota.

Installation:

Installing a tile server based on mod_tile requires a number of steps, some of which can be skipped depending on your needs:

TODO: Please expand the installation instructions

Mapnik rendering toolchain

Although not strictly necessary, nearly all installations of mod_tile use a rendering backend based on Mapnik. Therefore it is necessary to install it first and populate the PostGIS database with data first using osm2pgsql.

mod_tile

build and install the mod_tile apache module from source. If you use debian you can add this line in your source.list :

 deb http://repository.azae.net/debian/squeeze/ squeeze main

and install : libapache2-mod-tile and renderd

renderd

tile expiry

After rendering a tile, it gets cached on disk for fast serving. As OSM data is constantly updated, improved and changed, a mechanism is needed to correctly expire the cache to ensure updated tiles get re-rendered.

There are several possible mechanisms available for expiring and updating tiles (either through mod_tile or by directly re-rendering cached tiles), of which often a combination is used:

mod_tiles expiry works by comparing the time stamp of rendered tiles to the date of the last full data import into the database.

If the planet time stamp is updated, all tiles are automatically marked as in need for re-rendering (dirty) and will be submitted to the rendering backend on next visit.

For the diff-based updates of the database, expiring all tiles would be too costly, and so only those tiles for which the data was updated get expired by altering the time stamp of those tiles to date far in the past. Again there are two different scripts for doing this. One ruby script and one based on output from osm2pgsql.

Alternatively, there are a number of helper utilities that can directly talk to renderd or tirex, submitting tiles for rerendering. This is often used to re-render low-zoom tiles on a periodical basis in the background, as it would be too costly to update them through the normal diff-based expiry.

See also for more details: Tile expire methods.

HowTos

There are a number of HowTo's available to set up a tile server based upon mod_tile.

• DE:HowTo minutely hstore (German) - Describes the full setup of the mapnik rendering tool chain together with installing mod_tile, tirex and tile expiry
• Build your own OpenStreetMap Server - Describes the setup of the mapnik rendering toolchain
• HowTo mod tile - Describes real mod_tile install, it is a follow-up to the "build-your-own-openstreetmap-server" guide.

Configuration

Most config can be done in the apache config and in /etc/renderd.conf

Some special options like MAX_ZOOM can only be set on complile time in render_config.h

You will likely need to do some more preparation to load the data for rendering in mapnik.

The style sheet also needs adjusting

Testing and debugging installation

After installation and configuration, it is important to test that everything is working.

Here is a list of some of the common mistakes and problems and how to detect them:

• Check if mod_tile is loaded into Apache. First you should check if mod_tile is correctly loaded into your apache. One way of doing this is my verifying that you get the mod_tile status page under http://your.server/mod_tile . The status page should give you a number of tiles mod_tile has served, split by zoom-level, their status codes and how many requests it has sent to the rendering back end.

• Check if mod_tile can communicate with the rendering back end.
• Check that mod_tile has read permissions to the tile directory.

Additional help in trouble shooting might be found in the general help system of OSM

Source code

The code is in SVN at applications/utils/mod_tile, the readme.txt file contains other important information which you should read before trying to use this code.

Use svn co http://svn.openstreetmap.org/applications/utils/mod_tile/

Bug reports and feature requests

Mod_tile has been running stable on the OSM main page and many other sites for quite some time now with little trouble. It should thus be very stable and robust. Nevertheless, as will all software, it can have bugs. If you find any problems or instability using mod_tile, please report it on our bug tacker so that any remaining issues can be fixed:

Search for mod_tile tickets on trac

Do 'login' and 'new ticket'. Set the 'component' as 'mapnik' but mention 'mod_tile' in the description. In trac you can also report feature requests, however as always, there is no guarantee that any of the feature requests will get picked up.

Alternatively some issues and enhancements are listed below

TODO

This article or section contains out-of-date information. The information is no longer correct, or no longer has relevance. If you know about the current state of affairs, please help keep everyone informed by updating this information

This page has been suggested for clean-up. Please Discuss.

The current code has been running with few problems now for over a year but there are always some things which could be improved...

• Lots of parameters are set in the render_config.h file and compiled into the mod_tile & renderd code. Ideally mod_tile should get all these parameters from some suitable module config options in the Apache config file. The renderd process would also need a config file or some command line options. Update: Many parameters are now set at runtime thanks to [r12950] by Brian Quinion

Update: REQUEST_TIMEOUT, MAX_LOAD_OLD, MAX_LOAD_MISSING, and RENDER_SOCKET can now be set in Apache Config with ModTileRequestTimeout, ModTileMaxLoadOld, ModTileMaxLoadMissing, and ModTileRenderdSocketName. Update: socket name, number of threads, and some mapnik parameters can now be set in renderd.conf.

• It would be nice to support multiple layers using different osm.xml style files. Implemented in [r12950] by Brian Quinion
• Scalability of the list of tiles to render. Currently this is a linked list which needs to be scanned completely on each insertion to avoid duplicates. Maybe this should be a hash or tree to allow faster searching. This is mostly mitigated by using meta tiles. We currently only store a meta-tiles in the request queue instead of individual tiles. This effectively reduces the queue length by 64 (assuming METATILE=8). We also apply a hard limit on the queue lengths to prevent the search time growing excessively. Any popular tile will just get queued again when the queues have shrunk. Implemented in [r17338]
• There is a massive surge in rendering requests after the new planet dump has been imported. For several hours almost all tiles which are looked at will need to be re-rendered and this puts quite a strain on the rendering queue. We may need to implement another back-off algorithm to improve this behaviour.
• Bulk download tools can place an abnormal load on the system. They tend to download large numbers of tiles one after another with no break. They normally download all tiles within a given bounding box and can often download 100k+ tiles over a few hours. One of the existing backoff algorithms will return an error if the system is overloaded but the entries will still be added to the render queue. For bulk downloaders this has the unfortunate side effect of letting them fill the render queue faster than if they had to wait for each tile to be rendered and downloaded. I suspect this makes the Wednesday import load spike worse than it would otherwise be. The legacy ruby serving scripts imposed a time penalty for all requests which caused a new tile to be rendered and we may need to implement something similar in mod_tile Implemented in [r22778].

More ideas

The current server is managing to keep up fairly well with the rendering load. In Feb 2008 the server can re-render all the present tiles within the 7 day window between planet updates. It should take less than 14 days for any new data to appear on the map tiles (up to 7 days for the next planet dump and then up to 7 days for all the tiles to be re-rendered). Often it will be less, e.g. data added on Tuesday will often appear on the map on Thursday.

• Scalability of the rendering. The present scheme uses a single multithreaded rendering process. Ideally we want to be able to expand this across multiple rendering machines. Maybe there should be a 3rd tier which handles the request queue and acts as a broker between multiple mod_tile and rendering processes Implemented in [r17329].
• High availability - If we wanted to scale even further then maybe there could support for multiple queues. If each mod_tile supported making requests to multiple queues and each renderer could request from multiple queues then a redundant system could be developed.

TIPS

• Build mod_tile against SVN rather than mapnik 0.5.1 .. you'll have problem with multi-threading renderd otherwise. fribidi is not thread-safe, and TRUNK now uses libicu.
• While using debian, be sure to install either apache2-prefork-dev or apache2-threaded-dev , and modify the makefile in order to use /usr/bin/apxs2 instead of /usr/sbin/apxs.
• Check render_config.h for config parameters, like the number of renderd threads, the cut-off system loads, and the fonts directory. Renderd may crash while trying to use any font if this is wrongly set.

Unless specified, the configuration is read from the file /etc/renderd.conf installed with 'make install'

• The default configuration for stats file is '/var/run/renderd/renderd.stats', thus create manually the directory if it is missing.
• To see the syslog, execute renderd with the option -f (or use inet.d).