Howto real time tiles rendering with mapnik and mod python

From OpenStreetMap Wiki
Jump to navigation Jump to search

In a project I was creating, based on my own rendering rules sets, I hit a problem that I solved by real-time rendering tiles. It is clearly not the fastest solution as mod_tile is better. But I didn't want to install mod_tile

Here is a live example of it : Yet another validation tool for osm data

What is this solution good for?

  • If you wish to save disk space
  • If you want to test or to have several different mapnik styles
  • If you want more flexibility on tile generation process with your own python code
  • If you want updates very often
  • if you can't or don't want to install mod_tile which is faster

What problems does that create?

  • If more than ~2 or 3 users access maps at the same time, it will put a lot of CPU load on the machine (for massive use, a cache mecanism is available, and you'll be able to serve more users )
  • Generation times are quite slow
  • On low zoom (3 to 10) tile generation time might be even worse (a cache is definitively useful for those zoom levels)

Hardware machine for my rendering

I'm using a fairly "standard" machine, and I expect things to be a lot lot better if you have a powerful server. Composed of :

  • Intel(R) Core(TM)2 Duo CPU E7200 @ 2.53GHz
  • 2 GO RAM
  • 512 Go disk space (I'm currently using 100 Mo of it for the permanent tiles cache)

software configuration

  • GNU/Linux Debian Lenny amd64
  • postgres & postgis
  • osm2pgsql

Starting up

I won't write everything again, so please have a look at the very good tutorials :

Then you have to try some generation with python of tiles to make sure every things work with mapnik (generate_image.py is a good one to test )

How to set up the real time rendering

Prerequist and line of installation

This method needs mod_python included with apache in order to limit loading and unloading the python interpreter

The idea is that you will define a "virtual" tile directory that will in fact be empty where the python script that does all the work is.

Preparing the directory

Create a directory of your choice ("tiles" in my example) and write a .htaccess file with : 

SetHandler python-program
PythonPath "['/home/sites/www.openhikingmaps.org/tiles/'] + sys.path"
PythonHandler renderer::handle
ExpiresActive on
ExpiresDefault "access plus 1 hours"
  • /home/sites/www.openhikingmaps.org/tiles/ is the path of you directory
  • ExpiresDefault "access plus 1 hours" will tell that tiles should be kept by the browser in cache for at least 1 hour
  • PythonHandler renderer::handle says that to do the tile rendering should be handled by a file called "renderer.py" with a method "handle"

Put the python script in

In that directory put a file called renderer.py that contains : 

#!/usr/bin/env python
# Source is GPL, credit goes to Nicolas Pouillon
# comments goes to sylvain letuffe org (" " are replaced by @ and . )
# fichier execute par mod_python. handle() est le point d'entree.

import os, os.path
from gen_tile import MapMaker

def get_renderers(path, zmax = 20):
    r = {}
    for filename in os.listdir(path):
        if filename.startswith("."):
            continue
        if not filename.endswith(".xml"):
            continue
        rname = filename[:-4]
        try:
            r[rname] = MapMaker(os.path.join(path, filename), zmax)
        except:
            pass
    return r
zmax=20
renderers = get_renderers("/home/sites/www.openhikingmaps.org/mapnik-styles/", zmax)

def handle(req):
    from mod_python import apache, util
    path = os.path.basename(req.filename)+req.path_info

    # strip .png
    script, right = path.split(".", 1)
    new_path, ext = right.split(".", 1)
    rien, style, z, x, y = new_path.split('/', 4)


    if style in renderers:
        req.status = 200
        req.content_type = 'image/png'
        z = int(z)
        x = int(x)
        y = int(y)
	#req.content_type = 'text/plain'
	#req.write(renderers[style])
	#return apache.OK
	if z<13:
	  cache=True
	else:
	  cache=False
        req.write(renderers[style].genTile(x, y, z, ext, cache))

    else:
        req.status = 404
        req.content_type = 'text/plain'
        req.write("No such style")
    return apache.OK

You have to change /home/sites/www.openhikingmaps.org/mapnik-styles/ to where you mapnik styles are

Note : If you are using an other webserver than Apache and you could not use the mod_python module, here is a script which works with lighttpd (fcgi and flup packages needed) : 

#!/usr/bin/env python
# Source is GPL, credit goes to Nicolas Pouillon
# comments goes for this version to pierre_dot_mauduit _at_ gmail _dot_ com

import os, os.path
from gen_tile import MapMaker

def get_renderers(path, zmax = 20):
    r = {}
    for filename in os.listdir(path):
        if filename.startswith("."):
            continue
        if not filename.endswith(".xml"):
            continue
        rname = filename[:-4]
        try:
            r[rname] = MapMaker(os.path.join(path, filename), zmax)
        except:
            pass
    return r
zmax=20
renderers = get_renderers("/var/www/rtmapnik/", zmax)

def handle(environ, start_response):
    newpath, ext = os.environ['PATH_INFO'].split('.', 1)
    rien, style, z, x, y = newpath.split('/', 4)

    if style in renderers:
        status = 200
        content_type = 'image/png'
        z = int(z)
        x = int(x)
        y = int(y)
        if z<13:
         cache=False
        else:
         cache=False
        start_response('200 OK', [('Content-Type', content_type)])
        print renderers[style].genTile(x, y, z, ext, cache)
        return []
    else:
       return ["Error occured"]


if __name__ == '__main__':
    from flup.server.fcgi import WSGIServer
    WSGIServer(handle).run()


And another one, in the same directory with a name "gen_tile.py" 

#!/usr/bin/env python

import os
import mapnik
import math

def minmax (a,b,c):
    a = max(a,b)
    a = min(a,c)
    return a

class GoogleProjection:
    def __init__(self,levels=18):
        self.Bc = []
        self.Cc = []
        self.zc = []
        self.Ac = []
        c = 256
        for d in range(0,levels):
            e = c/2;
            self.Bc.append(c/360.0)
            self.Cc.append(c/(2 * math.pi))
            self.zc.append((e,e))
            self.Ac.append(c)
            c *= 2
                
    def fromLLtoPixel(self,ll,zoom):
         d = self.zc[zoom]
         e = round(d[0] + ll[0] * self.Bc[zoom])
         f = minmax(math.sin(math.radians(ll[1])),-0.9999,0.9999)
         g = round(d[1] + 0.5*math.log((1+f)/(1-f))*-self.Cc[zoom])
         return (e,g)
     
    def fromPixelToLL(self,px,zoom):
         e = self.zc[zoom]
         f = (px[0] - e[0])/self.Bc[zoom]
         g = (px[1] - e[1])/-self.Cc[zoom]
         h = math.degrees( 2 * math.atan(math.exp(g)) - 0.5 * math.pi)
         return (f,h)

class MapMaker:
    sx = 256
    sy = 256
    prj = mapnik.Projection("+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over")
    def __init__(self, mapfile, max_zoom):
        self.m = mapnik.Map(2*self.sx, 2*self.sy)
        self.max_zoom = max_zoom
        self.gprj = GoogleProjection(max_zoom)
        try:
            mapnik.load_map(self.m,mapfile)
        except RuntimeError:
            raise ValueError("Bad file", mapfile)

        self.name = hex(hash(mapfile))
    def tileno2bbox(self, x, y, z):
        p0 = self.gprj.fromPixelToLL((self.sx*x, self.sy*(y+1)), z)
        p1 = self.gprj.fromPixelToLL((self.sx*(x+1), self.sy*y), z)
        c0 = self.prj.forward(mapnik.Coord(p0[0],p0[1]))
        c1 = self.prj.forward(mapnik.Coord(p1[0],p1[1]))
        return mapnik.Envelope(c0.x,c0.y,c1.x,c1.y)
    def genTile(self, x, y, z, ext="png", cache=False):
        if cache:
            outname = '/home/sites/www.openhikingmaps.org/tiles/cache/%d/%d/%d.%s'%( z, x, y, ext)
            if os.path.exists(outname):
                fd = open(outname, 'r')
                return fd.read()
            try:
                os.makedirs(os.path.dirname(outname))
            except:
                pass
        else:
	  outname = os.tmpnam()

        bbox = self.tileno2bbox(x, y, z)
        bbox.width(bbox.width() * 2)
        bbox.height(bbox.height() * 2)
        self.m.zoom_to_box(bbox)

        im = mapnik.Image(self.sx*2, self.sy*2)
        mapnik.render(self.m, im)
        view = im.view(self.sx/2, self.sy/2, self.sx, self.sy)
        
        view.save(outname, ext)
        
        fd = open(outname)
        out = fd.read()
        fd.close()

        if not cache:
            os.unlink(outname)
        return out

# Fonction de test, qui n'est appelee que si on execute le fichier
# directement.

def test():
    renderer = MapMaker('/home/sites/www.openhikingmaps.org/mapnik-styles/test.xml', 18)
    for filename, x, y, z in (
        ('test.png', 2074, 1409, 12),
        ):
        fd = open(filename, 'w')
        fd.write(renderer.genTile(x, y, z))
        fd.close()

if __name__ == '__main__':
    test()

Again replace /home/sites/www.openhikingmaps.org/tiles/ by your directory.

As you can see, if you read python fluently, here : 

        cache = z<13
        req.write(renderers[style].genTile(x, y, z, ext, cache))

This is a piece of code that says to keep cached files for zoom <13 you can adapt if you wish to "<0" for no cache at all or "<20" for a cache of anything.

<13 looks a good compromise because, low zoom tiles take a long time to create, while keeping everything will need to flush at some times (based on tiles dates, osm2pgsql diff expired tiles list or at defined dates )

Open Layers

I started form there : OpenLayers Simple Example

And added the relevant parts in the index : 

<script src="/libs/OpenLayers.js"></script>
<script src="/libs/styles.js"></script>

(...)

var layers = [];
  for ( var idx in all_available_styles ) {
      var name = all_available_styles[idx];
      var l = new OpenLayers.Layer.TMS( 
          name, 
          ["/tiles/renderer.py/"+idx+"/"],
          {type:'jpeg',
          getURL: get_osm_url, 
          displayOutsideMaxExtent: true }, {'buffer':1} );
      layers.push(l);
  }

  map.addLayers(layers);

My full index.html file for reference

With /libs/OpenLayers.js being a local copy.

And /libs/styles.js containing the different possible styles that you want to use. 


var all_available_styles= new Array();
all_available_styles["noname"]=["No name"];
all_available_styles["nooneway"]=["No Oneway"];
all_available_styles["noref"]=["No Ref on way"];
all_available_styles["fixme"]=["Fixme or note Tags"];
all_available_styles["for_wikipedia"]=["For Wikipedia (without POI)"];
all_available_styles["mapnik_copy"]=["Mapnik SVN Copy (11/2008)"];
all_available_styles["hiking"]=["Hiking"];
all_available_styles["test_zone"]=["test zone"];
all_available_styles["test_zone2"]=["test zone 2"];
  • the key is the name of your mapnik style file (without .xml )
  • The value is the name that is displayed on the layer switcher

Cron job

My simplified cron job, takes its data from http://download.geofabrik.de/osm/ and looks likes this : It must be launched by the postgres user or a user that has access to the DB :

populate_db.sh
#!/bin/bash
cd /home/sites/www.openhikingmaps.org/cron

#File name
FILE=france.osm.bz2

# URL where to find it
URL=http://download.geofabrik.de/osm/europe

#remove old
rm $FILE

#get the new
wget $URL/$FILE

/usr/local/bin/osm2pgsql -S ./default.style -m -d gis $FILE 2>>log >>/dev/null
  • "-S ./default.style" is optionnal but you can write your own if you wish

edit 2009-03-20 : This solution was the easy one but importing europe is long, very long, I now use diff imports as explained here Minutely_Mapnik

Tile URL acccess

In those examples, the URL for the tile access is constructed to be :

noname being the name of the style file you want to use for the rendering
12 being the zoom level
jpeg being the chosen extension, just change it to png and you will get png !

PS: you could remove the "renderer.py" as it is useless and adapt the split code of the url, but it at least tells people that this uses python and not mod_tile

Thanks for reading. -- Sletuffe 15:08, 25 November 2008 (UTC)

Tweeks to my setup

Questions / remark

On the talk page, or by e-mail please

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Howto_real_time_tiles_rendering_with_mapnik_and_mod_python&oldid=2735087"