OSM Mobile Binary Protocol

From OpenStreetMap Wiki
Jump to navigation Jump to search
OSM Mobile Binary Protocol - Other languages

Open Street Map Binary Protocol

This page is about additions to permit a particular way of using the OSM Binary Format for mobile devices in a tiled way that permits caching, omission of duplicate features and intelligent diffs (todo).

The idea is to provide mobile applications with the minimum download possible to get enough data to draw maps and provide routing functions.

The API will take three queries: tile, ts and have (optional). For routing requests an additional tileto and detaillevel queries will be added.

eg: bmap.php?tile=tileid&ts=31&have=neighboursdownloaded

Where the tileid is 32bit int representing the bbox required (as calculated by lonLatToTileId function below), using a form of quadtileing (see QuadTiles). We default to using 31 of the available bits to avoid problems with negative numbers with broken things like PHP. This gives quite small tiles, so the ts parameter is used to specify a different tile size.

Have is a 8bit int representing neighbouring tiles already downloaded {nw,n,ne,w,e,sw,s,se}

eg: have downloaded ne,e,se => 00101001 = 0x29 = 41

Giving a url looking like: 

bmap.php?tile=257882462&ts=29&have=41

Providing a list of the surrounding tiles already downloaded means that ways that overlap with them can be omitted from the data for this tile (this is optional on the caller, a default value of 0 will indicate no surrounding tiles downloaded). A list of omitted ways will be included in the output.

The output will just include 'interesting nodes', and ways. Routing information is to be calculate from comparing all the ways within the tile and any with identical coordinates should be considered a junction node.

The protocol is being developed as a php caching proxy, it is also hoped to become an official OSM API direct from the database. Features listed on this page will each individually indicate whether they are implemented in each of these forms.

Usage and intended use

The OSM Mobile Binary Protocol is intended for the following types of clients:

  • mobile moving maps
  • mobile routing applications

It is not intended for

  • mobile editors

It is optimized for:

  • small bandwidth in transmission

This protocol is supported by the following clients:

Performance

Downloading a large tile over the west and middle of Edinburgh using both the XML API and the PHPProxy gives the following sizes:

Version Tile TS Protocol Size Compressed By Additional
0.5 1007353 21 OSM XML API 7260292
OSM XML gz 791627 89%
OSM XML 7z 584595 91%
Binary API 349571 95%
Binary API gz 227218 97% 35%
Binary API 7z 180941 98% 20%

This gives a 69% reduction in download size from 7ziped xml file to 7ziped binary protocol file. If client doesn't support 7zip then the reduction is 71% by using the binary protocol. Clients without compression experience a 98% reduction.

Downloading several adjacent (neighboring) tiles probably further increases performance, as ways passing several tiles are only downloaded once. A test for such case (eg a small city) would also be interesting, as well as a full planet dump (reducing compressed planet from 900MB to an estimated 180MB). --Stefanb 14:13, 2 October 2007 (BST)

Binary output format

The formats base structure is a series of binary blocks, which specify a type followed by encoded data. To avoid parsing issues with block separator values each block is preceded by its size.

  • Length - 32bit unsigned integer length (length does not include the 4 bytes used to specify the length)
  • Type - 8bit unsigned integer type (as defined in Types)
  • Data - binary data any format

To save space long/lat values are encoded as 32bit signed integers from the floating point values to six decimal places (eg lat/lon * 1000000). This gives ~11cm accuracy which is plenty for any mapping project. Time/Date fields are the number of seconds since the Unix Epoch (January 1 1970 00:00:00 GMT). Strings are truncated to 255 and are given as a length followed by characters utf8 encoded.

OSM nodes are not transmitted in their entirety, only interesting nodes are included, (nodes with some additional meta data - poi's).

Ways reduce the data needed to be transmitted by only sending the positions of nodes as long/lat pairs, the first long/lat pair using 32bits then subsequent ones are 16bit diffs from the previous node position, if the diff is too big then fake nodes filling the gap are given.

Ways that cross more than one tile would end up being downloaded more than once, to avoid this the 'have' parameter is used, ways that have been omitted are listed so that deleted ways can be detected.

Tile Definition (optional)

This can be used at the beginning of each tile definition if multiple tiles are packed into the same stream.

  • Length (32bit)
  • Type - 'a' (8bit)
  • Tile Size (8bit)
  • Tile Id (32bit)
  • Tile Contents (8bit) see Content Type
  • Tile Format Version (8bit) see Version Numbers

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Nodes (pre version upto v1)

  • Length (32bit)
  • Type - 'n' (8bit)
  • Node Id (32bit)
  • Longitude (32bit)
  • Latitude (32bit)
  • Primary property (8bit) see Node Property
  • Feature Tags (optional) see Feature Tags

Implementation Status: PHPProxy(Yes), OSMAPI(No)

POIs (version v2 onwards)

  • Length (32bit)
  • Type - 'i' (8bit)
  • Node Id (64bit)
  • Longitude (32bit)
  • Latitude (32bit)
  • Primary property (8bit) see Node Property
  • Feature Tags (optional) see Feature Tags

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Ways

  • Length (32bit)
  • Type - 'w' (8bit)
  • Way Id (32bit)
  • Node Count (32bit)
  • Longitude (32bit)
  • Latitude (32bit)
  • Followed by:
    • Longitude Diff (16bit)
    • Latitude Diff (16bit)
  • Feature Tags see Feature Tags

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Routeable Ways (possible for V3)

  • Length (32bit)
  • Type - 'v' (8bit)
  • Way Id (32bit)
  • Node Count (32bit)
  • Longitude (32bit)
  • Latitude (32bit)
  • Followed by:
    • Longitude Diff (16bit)
    • Latitude Diff (16bit)
  • Ways Joined Count (16bit)
  • Followed by:
    • Joined Way ID (32bit)
    • Joined At Node Index (16bit)
  • Feature Tags see Feature Tags

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Relation

  • Length (32bit)
  • Type - 'r' (8bit)
  • Relation Id (32bit)
  • Relation Type (8bit) see Relations Types
  • Members Count (32bit)
  • Followed by:
    • Member Type/Role (8bit) see Relations Roles
    • Member Id (32bit way id/64bit poi id) (repeated as necessary)
  • Feature Tags see Feature Tags

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Omitted Ways (V1 only)

  • Length (32bit)
  • Type - 'o' (8bit)
    • Way Id (32bit) (repeated as necessary)

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Tile Creation Time

  • Length (32bit)
  • Type - 't' (8bit)
  • Time (32bit)

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Failure

  • Length (32bit)
  • Type - 'f' (8bit)
  • Failure Code (8bit)
  • Description Length (8bit) (optional)
  • Description (utf8 encoded text)

Implementation Status: PHPProxy(Yes), OSMAPI(No)

OSM Feature Tags

OSM properties are encoded in a binary form, see: Feature Tags

Translations

OSM's name:xx is encoded using: Language Codes

Client specific tagging

Specific client applications will be able to specify which tags they are interested in (no point sending tags to an app if it doesn't do anything with that tag).

Implementation Status: PHPProxy(Yes), OSMAPI(No)

Compression

Because processing power on a mobile device might not be such a big deal as bandwidth (satellite, GPRS roaming...), it is suggested (=optional, not mandatory) to use HTTP compression using mod_gzip or alternatives.

Client support of this is detected by use of the HTTP header: Content-Encoding: gzip

Implementation Status: PHPProxy(Yes), OSMAPI(Yes)

Diffs

A more advanced mechanism could be employed to indicate the date the oldest out of the requested and neighbouring tiles was last downloaded.

Giving tile requests of the form: 

bmap.php?tile=257882462&ts=29&have=41&since=1190994262

Implementation Status: PHPProxy(No), OSMAPI(No)

Routing

Finding out which tiles one needs on his mobile device might be tricky without knowing the route and vice versa - it is impossible to compute a route (optimal, suboptimal, any) without having all the tiles locally.

Options:

  • Client could request all the tiles in the ellipse between the start and destination point (focus points of ellipse) and try finding a route in there, gradually increasing the ellipse area until a satisfactory route is found.
  • Server side Routing determines which tiles are needed.

Implementation Status: PHPProxy(No), OSMAPI(No)

Resolution - Level of Detail

On a way from A to B the traveller might only be interested in certain kinds of roads (paved, motorways, trunks, not residential, not pathways, not service roads....). Travelling on a motorway trough a big, well mapped city between A and B might result in great deal of unnecessary transfer.

Levels of detail for downloading routing information:

  • Full detail (default, done :) )
  • Navigable Ways only (would need to specify if route is foot/cycle/car, no museums, buildings, restaurants...gas stations?)
  • Route plus x km left and right (for finding detours, snack, or finding a way back to tour if forced to make a wrong turn. full detail?)
  • Route plus connecting ways only (just beginnings, good for seeing street names as you pass them)
  • Route only

Implementation Status: PHPProxy(No), OSMAPI(No)

Filter levels of detail for zoomed out map viewing:

  • Zoomed Out (8) coastline/trunk/motorway/rail/peak/place-subburb
  • Full OSM (15) default

Request by a: f=8 query on API call.

Implementation Status: PHPProxy(In progress), OSMAPI(No)

Example code - PHP

Some example code to demonstrate how to convert between lat/lon, tileid and bbox. Note the OSM db uses a slightly different (more elegant - fixed size) mechanism to calculate its tiles (32bit tile ids from scaled to 16bit x/y values): https://git.openstreetmap.org/rails.git/blob/HEAD:/lib/quad_tile/quad_tile.h

The reference implementation as a PHP proxy converting the 0.5 XML protocol to this one is available at: https://svn.symbianos.org/osmtocsv/trunk/ (osmgetbmap.php - PHPProxy)

The proxy code is split into a common core, a PHP web interface and a command line version.

Note: lat/lon are represented as integers to six decimal places 

$KMaxLon = 180000000;
$KMaxLat = 90000000;
$KMaxBits = 31;
$KLLMultiplier = 1000000;

function lonLatToTileId($lon, $lat) {
	global $KMaxLon,$KMaxLat,$KMaxBits;
	$out = 0;
	for ($index=0; $index<$KMaxBits; $index++) {
		if ($index%2) {
			if ($lat >= ($KMaxLat-($KMaxLat>>($index>>1)))) {
				$out += 1<<($KMaxBits-1-$index);
			} else {
				$lat += ($KMaxLat>>($index>>1));
			}
		} else {
			if ($lon >= ($KMaxLon-($KMaxLon>>($index>>1)))) {
				$out += 1<<($KMaxBits-1-$index);
			} else {
				$lon += ($KMaxLon>>($index>>1));
			}
		}
	}
	return $out;
}

function tileIdToBBoxArg($tileid,&$lonmin,&$latmin,&$lonmax,&$latmax) {
	global $KMaxLon,$KMaxLat,$KMaxBits;
	$lonmin = -$KMaxLon;
	$latmin = -$KMaxLat;
	for ($index=($KMaxBits-1); $index>=0; $index--) {
		$set = ((($tileid>>(($KMaxBits-1)-$index)) %2)==1);
		if ($set) {
			if ($index%2) {
				$latmin += ($KMaxLat>>($index>>1));
			} else {
				$lonmin += ($KMaxLon>>($index>>1));
			}
		}
	}
	$lonmax = $lonmin+($KMaxLon>>(($KMaxBits-1)>>1));
	$latmax = $latmin+($KMaxLat>>(($KMaxBits-2)>>1));
}

function tileIdToBBox($tileid) {
	global $KLLMultiplier;
	$lonmin = 0;
	$latmin = 0;
	$lonmax = 0;
	$latmax = 0;
	tileIdToBBoxArg($tileid,$lonmin,$latmin,$lonmax,$latmax);
	$out = ($lonmin/$KLLMultiplier).",".($latmin/$KLLMultiplier).",".($lonmax/$KLLMultiplier).",".($latmax/$KLLMultiplier);
	return $out;
}
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=OSM_Mobile_Binary_Protocol&oldid=2669924"