Read Only Map API

From OpenStreetMap Wiki
Jump to navigation Jump to search
Not to be confused with Rome

"Read only map API" or short ROMA is a subset of the main API only supporting the "map" call, read - only return all objects for a given bounding box. This API is mostly used by the Tiles@home client which needs to fetch the z12 bbox as raw data.

ROMA only defines the usage/API but not the specific implementations. There are multiple implementations available or in development.

Accessing ROMA servers

The ROMA servers can be currently accessed using a load balancer at

http://api1.osm.absolight.net/api/0.6/map?bbox=

Note: This is actually a Trapi-Server and not a Roma-Server.

Postgresql/Postgis implementation

Resource usage as of 2008-10-23:

  • 150GB Disk space
  • Minimum 4GB Ram
  • Webserver+Perl

The Database will heavily profit from more memory as the index on the node table itself is ~11GB so every request even for a single node will most likely hit the disk in this setup. The disks are 4x72GB SCA 10K in a Hardware Stripe - the more spindles the more requests the database will be able to handle at once.

The Database schema used is the Osmosis 0.5 api simple postgres schema with one slight modification. Osmosis tries to build a bbox for the way after importing. This does never really finish as the tables have not been ANALYZED by postgres and thus it does a full table scan for every single way. As the code itself does not make use of the bbox i simple dropped the bbox column which needs some osmosis modifications to not bail out on the different schema. See here http://lists.openstreetmap.org/pipermail/dev/2008-September/011640.html for the patch against Osmosis 0.29.

This implementation has some security measurements by not handing out requests but rather return 50x when the data in the database is older than 15 minutes. For this reason the cgi scripts reads the osmosis timestamp.txt. Another safety measure is to not allow an arbitrary number of parallel requests. By this the cgi scripts writes its pid into a directory and counts pids before handling the requests - if above a certain amount it simple returns a 50x. Both number may need tweaking based on the needs and hardware limitation.

Building osmosis

First install some dependencys - I guess these are not all - just the ones I remember: 

 apt-get install ant openjdk-6-jdk openjdk-6-jre openjdk-6-jre-headless

Then unpack osmosis, patch and build it: 

 unzip osmosis-20080828.zip
 [...]
 cd osmosis-0.29
 cat ../osmosis-0.29-patch | patch -p1
   patching file src/com/bretth/osmosis/core/pgsql/v0_5/impl/ChangeWriter.java
   patching file src/com/bretth/osmosis/core/pgsql/v0_5/PostgreSqlWriter.java
 ant
 [...]

Note: The freshly built osmosis will be inside the build/dist sub-directory. The osmosis.jar in the root directory is distributed with the package and will fail because it is un-patched. Therefore you need to execute: 

osmosis-0.29/build/dist/bin/osmosis

Database initialization

 apt-get install postgresql-8.3 postgresql-8.3-postgis

Follow instructions in /usr/share/doc/postgresql-8.3-postgis/README.Debian.gz on how to enable the GIS extensions in your database.

Then put the osmosis simple shema into your database: 

cat osmosis-0.29/script/pgsql_simple_schema.sql | psql osm

But make sure the line concerning the bbox on the ways is not there. Now get the current planet export from https://planet.openstreetmap.org/ Then you may import the initial planet into your postgres: 

 osmosis --read-xml file=planet-XXXXX.bz2 --write-pgsql user=dbuser database=osm password=dbpass

This will take at least a day until the import finishes. Make sure you use the patched osmosis otherwise the import will basically never finish.

If you experience slow requests after importing the database, you may need to build two extra indexes with the following commands: 

 create index idx_ways_big on way_nodes using btree (way_id);
 create index idx_nodes_big on way_nodes using btree (node_id);

These may take an hour or so to complete, please be patient.

Keeping the Database uptodate

To update the database via omsosis I run this script permanently in the background e.g screen session : 

 while true; do
         timeout 180 \
                 osmosis \
                         --read-change-interval \
                                 workingDirectory=/home/flo/update/ \
                         --write-pgsql-change \
                                 user=flo \
                                 database=osm \
                                 password=password;
         sleep 60
 done

this will run osmosis every minute which - given the right config - will download the osc changes files from planet.openstreetmap.org and apply the changes to the database.

Here is the osmosis configuration.txt file: 

 baseUrl=https://planet.openstreetmap.org/minute
 intervalLength=60
 changeFileBeginFormat=yyyyMMddHHmm
 changeFileEndFormat=yyyyMMddHHmm
 maxDownloadCount = 15

Apache config

       Alias /api/0.5/map /home/flo/cgi-bin/map.fcgi
       <Directory "/home/*/cgi-bin/">
               Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
               AllowOverride None
               Order deny,allow
               Allow from all
               AddHandler cgi-script .cgi
               AddHandler fcgid-script .fcgi
       </Directory>

CGI script

The CGI script is rewritten for mod-apache2-fcgid but should work as a simple cgi with some little modifications too. As the cgi itself is mostly long running the fcgi mod makes no real difference.

The current implementation as of 2008-10-23 can be found here: http://silicon-verl.de/home/flo/projects/osm/roma/

Debian packages used

Right now the ROMA uses Osmosis 0.29 and these Debian/Lenny packages: 

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-============================================================
ii  apache2                2.2.9-7                Apache HTTP Server metapackage
ii  libapache2-mod-fcgid   1:2.2-1                an alternative module compat with mod_fastcgi
ii  libcgi-fast-perl       5.10.0-16              CGI::Fast Perl module
ii  libdbd-pg-perl         2.8.7-1                Perl DBI driver for the PostgreSQL database server
ii  libdbi-perl            1.605-1                Perl5 database interface by Tim Bunce
ii  libfcgi-perl           0.67-2.1+b1            FastCGI Perl module
ii  libtimedate-perl       1.1600-9               Time and date functions for Perl
ii  openjdk-6-jre          6b11-6                 OpenJDK Java runtime, using Hotspot JIT
ii  perl                   5.10.0-16              Larry Wall's Practical Extraction and Report Language
ii  postgresql-8.3         8.3.3-1                object-relational SQL database, version 8.3 server
ii  postgresql-8.3-postgis 1.3.3-3                geographic objects support for PostgreSQL 8.3
ii  postgresql-client-8.3  8.3.3-1                front-end programs for PostgreSQL 8.3

Trapi

Trapi is documented on it's own wiki page.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Read_Only_Map_API&oldid=1550386"