User:Oliford

From OpenStreetMap Wiki
Jump to navigation Jump to search

Me

I live in Oxford and work at the Culham Science Centre near Abingdon.

Just started using OSM at the end of 2008.

I set out about 2 weeks ago trying to get an open-source app on my PDA which would give me searchable and routable maps for the whole of the UK based on the OSM data like many modern phones etc give. This page is, mostly for the benefit of anyone trying to do the same, an evaluation of the apps I found and of the various tools to handle and convert the OSM data.


PDA Map Display Apps

I've been working for the last ~8 months porting the linux kernel to the new HP iPAQ 21x series and have got it almost fully up and running. Part of the plan was to use it and my bluetooth GPS unit for car, bike and foot navigation (also, later, for flying gliders).

I want the data in vector format as I'd like place/street name lookup and routing to be done on the fly on the PDA. I want to be able to load ~entire countries and produce the data in a reasonable amount of time (an hour at most).

My brief impression of the various bits of relevant software I found:

GpsDrive

This is fairly good but is raster only. I used it while driving around Spain with google map tiles loaded. (I'll upload all the GPS traces of our travels one day).

Pros:

  • It is already packaged in openEmbedded which helps.
  • Works reasonably fast.

Cons:

  • Can only view within one tile at a time so there's no possibility of slippy map style.
  • Raster only.
  • I found the flat-C GTK code too messy and hard to follow for me to want to work on it.

Nav-It

Pros:

  • Vector based
  • Packaged under openEmbedded.
  • Has a built-in convertor directly from OSM XML.

Cons:

  • Rendering doesn't look as good as the others.
  • Can't work out how to move the map.
  • Not fast even on my core 2 duo.
  • Segfaults on the PDA (even the repo version).
  • Converter crashes on large extracts (e.g. Southern England).

Rana

Pros:

  • I really like the PDA-friendly interface for this.
  • Appears that it will one day support place name lookup.
  • Uses the tiledata2 convertor from the OSM SVN repository which did parse the whole UK.
  • Apparently does slippy moving the map but I can't make it work.

Cons:

  • Seems to be fairly early in development, has lots of bugs and crashes.
  • Written in python (I'm not a fan).
  • Uses far too much CPU on the PDA while not doing anything.
  • The tiledata2 convertor falls over on non-UTF8 characters.

Roadmap

Pros:

  • Slippy map style scrolling and continuous zooming functionality.
  • Even rotates the map.
  • Can deal with the whole of the UK in it's head even on the PDA.
  • It seems to think it can do name lookups (but I haven't got this working yet)/
  • Code seems fairly well structured (the buildmap_osm doesn't report useful errors though).
  • No seg faults so far even with the whole of the UK loaded.

Cons:

  • Lots of effort to reliably convert the OSM data.
  • I can't even work out if it is supposed to be able to find routes.


If you know of any PDA relevant apps I've missed, let me know.

Possibly a new OSM specific one?

I'm considering writing my own open source PDA app which would work directly from the OSMMB tiled vector data, give a slippy/scrollable and zoomable interface like Roadmap's but also provide the place/street name lookup and routing on all loaded tile data. The newer proposal on the OSM wiki for the OSMMB seems to be well suited to this. Currently the whole of the UK ends up in a 100MB OSSMB file which is quick to parse. I can slice it into tiles, linearly search it for street names etc in a few seconds on a machine it takes several minutes to do the same on the XML.

Handling the Data

The main problem with Roadmap is importing from OSM. It's own tool (buildmap_osm) can theoretically read the OSM XML directly but this doesn't work well at all. It seg-faults on even city-sized extracts and produces some crazy outputs at random. It can however convert from the OSM_Mobile_Binary_Protocol format very well (I'm calling it OSMMB). (NB: It can't handle > 64K polygons per tile).

So my next job was to get the OSMMB data for the whole UK, or for fairly large tiles of it...


The OSMMB PHP proxy

The PHP proxy that normally does this job can't at all cope with even fairly large data sets (e.g London) due to the memory issues of loading >7M nodes into PHP objects. It also uses PHP's basic XML parser which throws a temper tantrum on invalid UTF-8 data in strings and quite often randomly exits with no errors for no apparent reason. The newer(?) version of the OSMMB PHP proxy (from Symbian's SVN) works from a postgres DB which took ages to create and doesn't work (no errors, it just gives a single way and exits). I don't have the patience to work out why.

Osmosis

I tried to work around this by giving the older version data for small tiles. Asking for data directly from the OSM server is way too slow and it refuses any decent sized area (fair enough). Using Osmosis is incredibly slow and it dropped out due to running out of java heap when extracting medium sized areas from large data sets (eg greater London from the UK). Interstingly it can deal with very large areas from large data sets (Southern england from the UK) or very small areas (e.g. Oxford from the UK).

The_Rails_Port

I tried putting the 'ruby on rails' thing from the OSM svn on my local machine (Ubuntu Intrepid) which was hard work but I got there eventually. I loaded the UK extract XML into the mysql database with osmosis which took a few hours. Requesting data from the API (with the ruby server running) still takes a good long while and the ruby sever seg-faults on large areas. Again, I've not got the patience to work out why. Interestingly, querying mysql directly e.g. asking for all ways with nodes in a whole county sized box is fairly fast.

My own converter

In the end, I got so lost, confused and annoyed I spent a few hours writing my own XML -> OSMMB converter in C. It parses the whole of the UK into tiles in a couple of minutes (just ways, I've almost finished the POIs). This gives me Roadmap working fairly well with all the UK OSM data in ~50 tiles totaling 100MB.

Conclusions

In general, OpenStreetMap is truely awesome but the multitude of formats, utilities, editors, servers and protocols is really confusing for a newbie. I've now compiled and used half the things in the OSM svn tree, and have seven seperate copies of the UK database on my computer: XML, OSMMB, mysql, postgresql (which I don't think is the form used by Mapnik), Roadmap's own, Navit's own and tiledata2 form for Rana. Good job I have a big HDD! :)

I think there is definately a case for something which works on the OSMMB format given that it is simple to work with, almost directly relates to the main OSM data and can be parsed quickly.