Osmarender/Making the nestoria tiles
This page details how OpenStreetMap was integrated into Nestoria, the first commercial use of OSM maps. The process described below dates from October 2006.
The Brief
Nestoria uses Google Maps, via Mapstraction. They wanted to use GTileLayer class in Google Maps, which allows for a custom set of tiles; Nestoria was already using GTileLayer to overlay the London Undergound map. The tiles must exactly match the Google Maps tiling scheme, at a number of zoom levels. So the task was to use Osmarender to generate pretty maps from OpenStreetMap data, and tile in the GMap scheme.
Preparing the Data
For the tiling, it would be easiest if the starting maps' boundaries conformed precisely to tile boundaries in GMaps tiling scheme. Debug Google Maps is a nice utility to browse GMaps, with details of the visible tiles overlayed. At zoom level 11 (the lowest resolution map), the lat/lon boundary of the tiles that included all of the Isle of Wight was determined. This data was downloaded from the OpenStreetMap API.
Each higher resolution zoom level would reveal more detail. So a series of osmarender rules files were manually written, for each zoom level. Basically, this meant that certain render rules were commented out for the lower resolutions. For instance, at level 13, residential streets are first visible, and at level 15, residential street names are visible. Also at lower zoom levels the width of roads was increased to make them more prominent.
Also had to determine the dimensions of the generated map for each zoom level. At level 11, the Isle of Wight covers 4 x 3 tiles; each GMaps tile is 256 x 256, so the total map dimension is 1024 x 768. Each zoom level doubles the width and height, so at level 12 it's 2048 x 1536, etc.
Generating the Maps
Usually, osmarender images are produced by loading the rules file in firefox, which is transformed into SVG and rendered, and then a screenshot is made. Since we needed precision and some very large images (if possible, zoom level 18 would be 131072 x 98304) that method was out. Instead Xalan was used to transform IsleOfWight.osm into SVG, and that SVG was loaded into Inkscape for rendering. From there, the image could be saved as a PNG of desired size.
One tricky issue was selecting vectors which only lie within the desired area, since Osmarender draws some chrome and other elements outside the map. All of the Osmarender chrome except the border was switched off (including the zoom control and the grid). A <bounds> tag was included to make the border coincide with the exact boundaries of the required area. The image was then converted by loading the SVG file into Inkscape, selecting everything within the map border and then using Export Bitmap to create a .PNG of everything inside the border.
Inkscape's rendering is good, though not as good as Firefox. In particular neither it nor Firefox implement the baseline-shift attribute which is used to position the street names correctly along roads.
Due to memory constraints, the highest resolution produced was level 16 (a whopping 32768 x 24576).
The resulting PNG at zoom level 12
Coastlines
To fill the sea as blue, with white land, required that the coastline be present in the OSM data. Almien_coastlines_(PGS) has been importing a coastline database. Unfortunately there were some problems.
In order for SVG to fill in a area, the coastline would need to be present as a single ordered Way. The almien coastlines are broken up into arbitrary chunks, and either by the import script or the osm api, are not accessible in a continuous series. fixcoastlines.pl script was a quick hack to group all coastlines into a single way, but didn't order, and wasn't enough.
For the Isle of Wight, a coastline had been prepared for the IoW mapping weekend by tracing Landsat imagery. This Way was manually added to the OSM data. The generated tile boundaries also included parts of the mainland, Portsmouth. Since there was no reliable coastline there, all non-Isle of Wight data was removed. Anyway, including only the Isle of Wight was deemed to be more conceptually in line with the project.
Tiling
Nestoria provided a handy Python script to take an image, the x/y coordinate of the top left corner of the image, a zoom level, and generate all the tiles for that area.
This worked very well, but started to run into problems with the larger images at higher zoom levels. Zoom level 14 and 15 took so much memory and cpu that it started getting killed off on the shared host where it was running (dreamhost). Zoom level 16 image was so large that the Python Image Library simply crashed. Zoom level 15 was sufficient detail, so we stopped there.
Google Maps Code & Mapstraction
To test the tiles as they were being worked on, a quick GMaps API example was whipped up. This demonstrates all the steps necessary to add a GTileLayer.
This became the basis for the inital OpenStreetMap support in Mapstraction. If the "Openstreetmap" API is requested, a Google Map is created, the GTileLayer added, and then the API type switch to "Google" for the rest of Mapstraction processing.
Tweaking
After the first batch of tiles, Nestoria had three pieces of feedback .. 1) Could the sea colour be changed to match the one used by Google. 2) Could the areas with no data be made transparent 3) Run the tiles through pngcrush. The idea was to have Google Maps and OpenStreetMap display seamlessly. Regenerating the images from scratch wasn't an option at this point, so proceeded to work on transforming the current batch of tiles using ImageMagick.
The sea colour in the OSM tiles could not simply be made transparent for two reasons. The coastline did not exactly match Google's coastline, so in some spots GMaps which show through (for instance, in the interior harbor near Newport). And the ferry routes to the IoW did not exactly match those route drawn in Google Maps. So basically a buffer zone of sea would need to exist around the IoW, in the GMaps colour, with the ferry lines removed.
First, to change the sea colour, an OSM tile and Google tile were loaded into Gimp, and the hex value of those pixels determined; and then ran the following command (with other colour values, don't remember those at the moment).
find input/ -name *png -exec convert -fill 0x6666ff -opaque 0x0000ff {} output/{} \;
Tiles without data were a uniform colour, but varied in size and internal colour palette (not sure what those extra colours were there for). They did all have less than 15 colours in their palette, so the following script was executed by find, to replace those tiles with transparent png.
#!/usr/bin/perl
$img = $ARGV[0];
$num = `identify -verbose $img | grep Colors:`;
$num =~ s/[^\d]//g;
if ($num < 15) {
system("cp blank.png $img");
}
This left tiles with the ferry lines, or tiles at lower resolutions that overlapped both the IoW and mainland. So at this point, the tile were inspected in the GMaps API, and "bad" tiles loaded into Gimp for manual editing. This is why the transparency boundaries are so funky in the Mapstraction example.
Lessons
This was a useful exercise, in deploying OSM tools for "real-world" use, and provided lots of lessons, questions, and room for improvement.
- Breaks within rendered ways are caused by collected segments in multiple directions. These could be automatically corrected. JOSM and other tools could enforce such a rule, perhaps optionally.
- In Newport especially, things like Parking Places need to be repositioned so that the icons don't overlap the streets. Makes me wish for Osmarender level rendering in JOSM. The Osmarender view plugin is a good step towards this.
- Working with large images was one time consuming challenge. For the tiling, VIPS may be useful for large images -- it provides random access, and doesn't attempt to load the entire image into memory.
- cut up the svg (set the bounds) for a set of 9 tiles, render that, and select the center tile
- There are too many pubs! Within a town, pubs are not so useful for navigation, but many country pubs are definitely useful to have. Perhaps there should be different classifications for pubs useful to navigation.
- The SVG rendering of some labels are too long for the way (ie short street, long name). Is there a way to detect this in SVG? Or perhaps some osmarender rule could make the length of the way as parameter for whether the label is show or not. (There is some control using the svg:font-size and osmarender:renderName tags but this is very laborious and might need to be different for each zoom level).
- Sometimes text is rendered upside down. Is this simply because of the direction of the way? (osmarender:nameDirection=-1 would fix this, or reverse the direction and order of the segments in the way).
- also, if you look closely, the text is not rendered in the middle of the street, but off the center line, towards a border
- In render rules, perhaps the desired image resolution could be a parameter for each rule.
- Inkscape may not be appropriate for converting SVG into PNG, for use in automated tile generation. What about Batik?
- how about using a recent librsvg (that's supposed to do text properly) or fix it ?