Mod tile

From OpenStreetMap

This is currently being used to render and serve tiles for the Mapnik layer on the Slippy_map. It is a custom C implementation to efficiently serve files from the disk and manage the rendering queue.

Implementation

The code is split into two parts:

• renderd - A rendering daemon which calls mapnik and writes the map tiles to the filesystem
• mod_tile - An Apache module which is responsible for requesting tiles to be rendered and setting the expiry headers

If the tiles are already rendered then Apache will serve them efficiently from the filesystem with minimum overhead (unlike the old scripts which required several MySQL queries for each tile served).

The code is in SVN at applications/utils/mod_tile the readme.txt file contains other important information which you should read before trying to use this code.

TODO

There are always some things which could be improved

• Scalability of the list of tiles to render. Currently this is a linked list which needs to be scanned completely on each insertion to avoid duplicates. Maybe this should be a hash or tree to allow faster searching. This is mitigated to some extent by using meta tiles. We currently only store a meta-tiles in the request queue instead of individual tiles. This effectively reduces the queue length by 64 (assuming METATILE=8).
• Lots of parameters are set in the render_config.h file and compiled into the mod_tile & renderd code. Ideally mod_tile should get all these parameters from some suitable module config options in the Apache config file. The renderd process would also need a config file or some command line options.
• It would be nice to support multiple layers using different osm.xml style files.
• There is a massive surge in rendering requests after the new planet dump has been imported. For several hours almost all tiles which are looked at will need to be re-rendered and this puts quite a strain on the rendering queue. We may need to implement another back-off algorithm to improve this behaviour.
• Bulk download tools can place an abnormal load on the system. They tend to download large numbers of tiles one after another with no break. They normally download all tiles within a given bounding box and can often download 100k+ tiles over a few hours. One of the existing backoff algorithms will return an error if the system is overloaded but the entries will still be added to the render queue. For bulk downloaders this has the unfortunate side effect of letting them fill the render queue faster than if they had to wait for each tile to be rendered and downloaded. I suspect this makes the Wednesday import load spike worse than it would otherwise be. The legacy ruby serving scripts imposed a time penalty for all requests which caused a new tile to be rendered and we may need to implement something similar in mod_tile.

More ideas

The current server is managing to keep up fairly well with the rendering load. In Feb 2008 the server can re-render all the present tiles within the 7 day window between planet updates. It should take less than 14 days for any new data to appear on the map tiles (up to 7 days for the next planet dump and then up to 7 days for all the tiles to be re-rendered). Often it will be less, e.g. data added on Tuesday will often appear on the map on Thursday.

• Scalability of the rendering. The present scheme uses a single multithreaded rendering process. Ideally we want to be able to expand this across multiple rendering machines. Maybe there should be a 3rd tier which handles the request queue and acts as a broker between multiple mod_tile and rendering processes.
• High availability - If we wanted to scale even further then maybe there could support for multiple queues. If each mod_tile supported making requests to multiple queues and each renderer could request from multiple queues then a redundant system could be developed.