Overpass API/install

From OpenStreetMap Wiki
Jump to: navigation, search

This page tells you how to install the OSM3S server such that you can use it as a local OSM mirror. Additional functionality like management of areas and the line diagram utils aren't covered yet.

System Requirements

It is highly recommended that you have at least the following hardware resources available for an OSM planet server:

  • 1 GB of RAM (less is acceptable if you have more processor resources)
  • 40 GB of hard disk space (80-100 GB if you want minutely updates, less if you use a smaller extract file)

It is required that you have the following resources:

  • Access to Expat and a C++ compiler
  • An OSM file in XML format compressed in bzip format (CloudMade is an excellent resource for this. Another good resource is located on the Planet.osm page.)

NOTE: You do not need a database engine (e.g. MySQL or PostgreSQL); the database back-end is included in the OSM3S package.

You will need to identify or create:

  • $EXEC_DIR: The root directory in which executable files should be installed (/bin/ suffix removed). (~100 MB). For example, a good place might be: /srv/osm3s
  • $DB_DIR: a directory to store the database
  • $PLANET_FILE: a place for the compressed (bzip) OSM planet/extract file (up to 10 GB)
  • $REPLICATE_DIR: a directory to store minutely (or otherwise) diffs (only necessary if you decide to configure minutely updates below)

Installation

Ubuntu or Debian 6.0 (squeeze)

1. Install the following packages: g++, make, expat, and libexpat1-dev.

sudo aptitude install g++ make expat libexpat1-dev

2. Download the latest tarball, prepared with GNU autoconf. For example:

wget http://www.overpass-api.de/misc/osm-3s_v[latest_version].tar.gz

Alternatively, if you want the bleeding edge latest dev version, you can get it from here on github, check the README for instructions

3. Unpack the tarball:

tar -zxvf osm-3s_v*.tar.gz

4. Compile the OSM3S package:

cd osm-3s_v*/build/
../src/configure --prefix=$EXEC_DIR
make install

NOTE: If you encounter a message like this: configure: error: cannot find install-sh or install.sh in "../src" "../src/.." "../src/../.." it may indicate that the symbolic link(s) in the "../src/" directory are broken. Before you can continue you will need to delete and recreate the links to your system's proper files, for example:

ln -s /usr/share/automake-1.11/missing ./missing
ln -s /usr/share/automake-1.11/install-sh ./install-sh
ln -s /usr/share/automake-1.11/depcomp ./depcomp

NOTE: If you encounter an error of this format during compiling: make: *** [...] Error 1 it means that something unexpected occurred and this is an opportunity to help make the OSM3S package more robust. To help you will need to capture the compile-time output and email it to the package's current maintainer: Roland Olbricht For example, the following command will capture the output and put it in a file called error.log:

make install >&error.log

Populating the DB

Populate the database with:

nohup ../src/bin/init_osm3s.sh $PLANET_FILE $DB_DIR $EXEC_DIR &
tail -f nohup.out

If you want to user metadata, add the --meta- parameter:

nohup ../src/bin/init_osm3s.sh $PLANET_FILE $DB_DIR $EXEC_DIR --meta &
tail -f nohup.out

The nohup together with & makes the process detached from your console, so you can log off without accidently stopping it. The tail -f nohup.out allows to nontheless read the output of the process (which is written into nohup.out).

NOTE: This step can take a very long time to complete. In the case of smaller OSM extract files less than 1 hour, but in the case of a full planet file this step could take on the order of 24 hours or more, depending on available memory and processor resources. When the process has finished successfully the file nohup.out will indicate this with "Update complete" at the very end.

(As a side note, this also works for applying OSC files onto an existing database. Thus you can make daily updates by applying these diffs with a cronjob. This method takes fewer disk loads than minute updates, and the data is still pretty timely.)

Static Usage

OSM3S is now ready to answer queries. To run a query, run

$EXEC_DIR/bin/osm3s_query --db-dir=$DB_DIR

and enter your query on the standard input. If typing directly into the console, you need to press Ctrl+D in the end to signal the end of input. Answers will appear on standard output.

If you've imported the entire planet, try the example query:

<query type="node"><bbox-query n="51.0" s="50.9" w="6.9" e="7.0"/><has-kv k="amenity" v="pub"/></query><print/>

This one returns all pubs in Cologne (the city with the best beer in Germany :) ).

Check the full introduction to OSM3S query language on the Web or at $EXEC_DIR/html/index.html (installed as part of OSM3S) for more information.

Lastly, if you're using the dispatcher daemon, osm3s_query can connect to it and find $DB_DIR by itself:

$EXEC_DIR/bin/osm3s_query

Starting the dispatcher daemon

If you wish to automatically apply diff updates or run the Web API, you need to start the dispatcher daemon (this is otherwise optional).

nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR &

For meta data you need to add a parameter:

nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta &

Ubuntu Upstart script

If using Ubuntu or other system that uses Upstart, you can use it to start Overpass' dispatcher. For example, into /etc/init/overpass.conf, put:

description 'Overpass API dispatcher daemon'

env DB_DIR=/path/to/db
env EXEC_DIR=/path/to/Overpass/installation

start on (local-filesystems and net-device-up)
stop on runlevel [!2345]

pre-start script
        rm $DB_DIR/osm3s* || true
        rm /dev/shm/osm3s* || true
end script

exec $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR

Overpass' dispatcher will restart itself on reboot. Modify DB_DIR and EXEC_DIR appropriately, and add --meta to the exec line if you also want to serve metadata.

Applying minutely (or hourly, or daily) diffs

Note: The dispatcher daemon must be running for diff application to work.

First, decide the maximum tolerable lag for your DB:

From these, you need to find replicate sequence number, which will become $FIRST_MIN_DIFF in the instructions below. To find it:

  1. Browse through the replicate directory hierarchy (e.g. http://planet.openstreetmap.org/minute-replicate/) and find the diff that has a date before the starting point of the planet dump. The planet dump starts at 00:00 UTC; because the server shows local time, this is equivalent to 01:00 BST during summer and 00:00 BST during winter in the file listing.
  2. Verify you have the right file by checking the respective *.state.txt file. The timestamp should show a date (here always UTC) slightly before midnight. sequenceNumber in this file (also present in the filename) is your replicant sequence number, and $FIRST_MIN_DIFF.

From $EXEC_DIR/bin, run:

nohup ./fetch_osc.sh $FIRST_MINDIFF_ID http://planet.openstreetmap.org/minute-replicate $REPLICATE_DIR/ &

This starts a daemon that will download all diffs from $FIRST_MINDIFF_ID to the present into your replicate directory. When new diffs are made available, if this is kept running, it will download them automatically. If you get diffs on another way, you can omit this command.

Next, apply changes to your DB:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID &

This starts the daemon that keeps the database up to date. Latest versions require an additional parameter augmented_diffs:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID --augmented_diffs=no &

To add metadata, you must add a parameter to the second command. Instead of the above, run:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID --meta &

To see what's going on, watch these log files:

  • $DB_DIR/transactions.log
  • $DB_DIR/apply_osc_to_db.log
  • $REPLICATE_DIR/fetch_osc.log

Setting up the Web API

Note: The dispatcher daemon must be running for the Web API to work.

This section describes one way to setup a basic read-only HTTP based API with OSM3S.

1. Install Apache2

sudo apt-get install apache2

2. Configure Apache2

cd /etc/apache2/sites-available
nano default

Make your default file look something like this:

<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	ExtFilterDefine gzip mode=output cmd=/bin/gzip
	DocumentRoot [YOUR_HTML_ROOT_DIR]
 
	# This directive indicates that whenever someone types http://www.mydomain.com/api/ 
	# Apache2 should refer to what is in the local directory [YOUR_EXEC_DIR]/cgi-bin/
	ScriptAlias /api/ [YOUR_EXEC_DIR]/cgi-bin/
 
 
	# This specifies some directives specific to the directory: [YOUR_EXEC_DIR]/cgi-bin/
	<Directory "[YOUR_EXEC_DIR]/cgi-bin/">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
                #SetOutputFilter gzip
                #Header set Content-Encoding gzip
	</Directory>
 
	ErrorLog /var/log/apache2/error.log
 
	# Possible values include: debug, info, notice, warn, error, crit, alert, emerg
	LogLevel warn
 
	CustomLog /var/log/apache2/access.log combined
 
</VirtualHost>

3. Restart Apache2:

sudo /etc/init.d/apache2 restart

4. Start the dispatcher process and point it to your database directory:

sudo nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR &

With meta data:

sudo nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta &

Note: to convert this process to a service that starts up when your system boots do this (... in progress)


5. Test your Web-API by sending it the following command:

wget --output-document=test.xml http://[your_domain_or_IP_address]/api/interpreter?data=%3Cprint%20mode=%22body%22/%3E

The xml output document should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<osm-derived>
  <note>
    The data included in this document is from www.openstreetmap.org. It has there been collected 
    by a large group of contributors. For individual attribution of each item please refer to 
    http://www.openstreetmap.org/api/0.6/[node|way|relation]/#id/history 
  </note>
  <meta osm_base=""/>
 
</osm-derived>

Trouble shooting

runtime error: open64: 2 /osm3s_v0.6.91_osm_base Dispatcher_Client

Note: if you get an output doc that looks more like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>
 
<p>
   The data included in this document is from www.openstreetmap.org. It has there been collected
   by a large group of contributors. For individual attribution of each item please refer to 
   http://www.openstreetmap.org/api/0.6/[node|way|relation]/#id/history 
</p>
 
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 2 /osm3s_v0.6.91_osm_base Dispatcher_Client::1 </p>
 
</body>
</html>

Then it may indicate that the dispatcher process is not running or not configured correctly.

File_Error 17 /osm3s_v0.6.94_osm_base Dispatcher_Server::1

If you killed (or crashed) the dispatcher daemon and wish to restart it, you might encounter this error (unless you reboot) : There is a lock file : /dev/shm/osm3s_v0.6.94_osm_base that prevent other dispatchers to run while one is allready running. Remove that file (and check that no dispatcher is running) and restart it.

Apache config fails

If you encounter some message like this one when (re)starting the apache server:

# apache2ctl graceful 
Syntax error on line 12 of /etc/apache2/httpd.conf:
Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration
Action 'graceful' failed.

then apache doesnt use mod_headers. you can activate mod_headers by running:

# a2enmod headers
Enabling module headers.
To activate the new configuration, you need to run:
service apache2 restart
# apache2ctl graceful 

After this, apache should start up correctly.