Overpass API/install

From OpenStreetMap Wiki
Jump to: navigation, search

This page tells you how to install the OSM3S server such that you can use it as a local OSM mirror. Additional functionality like management of areas and the line diagram utils aren't covered yet.

System Requirements

It is highly recommended that you have at least the following hardware resources available for an OSM planet server:

  • 1 GB of RAM (less is acceptable if you have more processor resources)
  • 40 GB of hard disk space (80-100 GB if you want minutely updates, less if you use a smaller extract file)

It is required that you have the following resources:

  • Access to Expat and a C++ compiler
  • An OSM file in XML format compressed in bzip format (Geofabrik is an excellent resource for this. Another good resource is located on the Planet.osm page.)

NOTE: You do not need a database engine (e.g. MySQL or PostgreSQL); the database back-end is included in the OSM3S package.

You will need to identify or create:

  • $EXEC_DIR: The root directory in which executable files should be installed (/bin/ suffix removed). (~100 MB). For example, a good place might be: /srv/osm3s
  • $DB_DIR: a directory to store the database
  • $PLANET_FILE: a place for the compressed (bzip) OSM planet/extract file (up to 10 GB)
  • $REPLICATE_DIR: a directory to store minutely (or otherwise) diffs (only necessary if you decide to configure minutely updates below)

NOTE: If you run into 403 forbidden errors on apache, double check your configuration in /etc/apache2/apache2.conf if your directory is explicitly allowed.

Installation

Ubuntu or Debian 6.0 (squeeze) or Debian 7.0 (wheezy)

1. Install the following packages: g++, make, expat, and libexpat1-dev.

sudo aptitude install g++ make expat libexpat1-dev

or if 'aptitude not recognised'

sudo apt-get install g++ make expat libexpat1-dev

NOTE: if you receive an libexpat1-dev not found error, try running the following:

sudo apt-get update && sudo apt-get install expat libexpat1-dev
 
or just 
 
apt-get update 
apt-get upgrade

Option 1: Installation via tarball

2. Download the latest tarball, prepared with GNU autoconf. For example:

wget http://www.overpass-api.de/misc/osm-3s_v[latest_version].tar.gz

3. Unpack the tarball:

tar -zxvf osm-3s_v*.tar.gz

Continue with step 4 below.

Option 2: Installation via bleeding edge dev version (expert use)

2. Alternatively, if you want the bleeding edge latest dev version, you can get it from here on github

sudo apt-get install git libtool autoconf automake
git clone https://github.com/drolbr/Overpass-API.git osm-3s-dev-version

Depending on your Ubuntu version you may need to explicitly tell apt-get to install version 1.11:

sudo apt-get install automake1.11

3. Update build system

When using the latest dev version from github, the build system has to be updated first. The following steps were successfully tested on Ubuntu 14.04 and debian 7.0:

cd osm-3s-dev-version
pushd src/
autoscan
aclocal-1.11
autoheader
libtoolize
automake-1.11 --add-missing
autoconf
popd

Note: aclocal and automake version 1.11 are required, newer versions like 1.14 don't seem to work at this time.


4. Compile the OSM3S package:

cd osm-3s_v*/build/
../src/configure CXXFLAGS="-O3" --prefix=$EXEC_DIR
make install

NOTE: If you encounter a message like this: configure: error: cannot find install-sh or install.sh in "../src" "../src/.." "../src/../.." it may indicate that the symbolic link(s) in the "../src/" directory are broken. Before you can continue you will need to delete and recreate the links to your system's proper files, for example:

ln -s /usr/share/automake-1.11/missing ./missing
ln -s /usr/share/automake-1.11/install-sh ./install-sh
ln -s /usr/share/automake-1.11/depcomp ./depcomp

NOTE: If you encounter an error of this format during compiling: make: *** [...] Error 1 it means that something unexpected occurred and this is an opportunity to help make the OSM3S package more robust. To help you will need to capture the compile-time output and email it to the package's current maintainer: Roland Olbricht For example, the following command will capture the output and put it in a file called error.log:

make install >&error.log

Populating the DB

To populate the overpass database, you need to download a planet file to a known directory. e.g.

wget -O $PLANET_FILE "http://ftp.heanet.ie/mirrors/openstreetmap.org/planet-latest.osm.bz2"

Populate the database with:

nohup ../src/bin/init_osm3s.sh $PLANET_FILE $DB_DIR $EXEC_DIR &
tail -f nohup.out

If you want to user metadata, add the --meta- parameter:

nohup ../src/bin/init_osm3s.sh $PLANET_FILE $DB_DIR $EXEC_DIR --meta &
tail -f nohup.out

The nohup together with & makes the process detached from your console, so you can log off without accidently stopping it. The tail -f nohup.out allows to nontheless read the output of the process (which is written into nohup.out).

NOTE: This step can take a very long time to complete. In the case of smaller OSM extract files less than 1 hour, but in the case of a full planet file this step could take on the order of 24 hours or more, depending on available memory and processor resources. When the process has finished successfully the file nohup.out will indicate this with "Update complete" at the very end.

(As a side note, this also works for applying OSC files onto an existing database. Thus you can make daily updates by applying these diffs with a cronjob. This method takes fewer disk loads than minute updates, and the data is still pretty timely.)

Populating the DB with attic data (0.7.50 and newer)

(This section was moved to a user page due to highly experimental state, see User Page).

Official documentation will be available soon...

Static Usage

OSM3S is now ready to answer queries. To run a query, run

$EXEC_DIR/bin/osm3s_query --db-dir=$DB_DIR

and enter your query on the standard input. If typing directly into the console, you need to press Ctrl+D in the end to signal the end of input. Answers will appear on standard output.

If you've imported the entire planet, try the example query:

<query type="node"><bbox-query n="51.0" s="50.9" w="6.9" e="7.0"/><has-kv k="amenity" v="pub"/></query><print/>

This one returns all pubs in Cologne (the city with the best beer in Germany :) ).

Check the full introduction to OSM3S query language on the Web or at $EXEC_DIR/html/index.html (installed as part of OSM3S) for more information.

Lastly, if you're using the dispatcher daemon, osm3s_query can connect to it and find $DB_DIR by itself:

$EXEC_DIR/bin/osm3s_query

If you can make conversion requests to osm3s_query without specifying the db dir, then the dispatcher daemon is running correctly.

Starting the dispatcher daemon

If you wish to automatically apply diff updates or run the Web API, you need to start the dispatcher daemon (this is otherwise optional).

nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR &

For meta data you need to add a parameter:

nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta &

Ubuntu Upstart script

If using Ubuntu or other system that uses Upstart, you can use it to start Overpass' dispatcher. For example, into /etc/init/overpass.conf, put:

description 'Overpass API dispatcher daemon'

env DB_DIR=/path/to/db
env EXEC_DIR=/path/to/Overpass/installation

start on (local-filesystems and net-device-up)
stop on runlevel [!2345]

pre-start script
        rm $DB_DIR/osm3s* || true
        rm /dev/shm/osm3s* || true
end script

exec $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR

Overpass' dispatcher will restart itself on reboot. Modify DB_DIR and EXEC_DIR appropriately, and add --meta to the exec line if you also want to serve metadata.

Applying minutely (or hourly, or daily) diffs

Note: The dispatcher daemon must be running for diff application to work.

First, decide the maximum tolerable lag for your DB:

From these, you need to find replicate sequence number, which will become $FIRST_MIN_DIFF in the instructions below. To find it:

  1. Browse through the replicate directory hierarchy (e.g. http://planet.openstreetmap.org/minute-replicate/) and find the diff that has a date before the starting point of the planet dump. The planet dump starts at 00:00 UTC; because the server shows local time, this is equivalent to 01:00 BST during summer and 00:00 BST during winter in the file listing.
  2. Verify you have the right file by checking the respective *.state.txt file. The timestamp should show a date (here always UTC) slightly before midnight. sequenceNumber in this file (also present in the filename) is your replicant sequence number, and $FIRST_MIN_DIFF.

From $EXEC_DIR/bin, run:

nohup ./fetch_osc.sh $FIRST_MINDIFF_ID http://planet.openstreetmap.org/minute-replicate $REPLICATE_DIR/ &

This starts a daemon that will download all diffs from $FIRST_MINDIFF_ID to the present into your replicate directory. When new diffs are made available, if this is kept running, it will download them automatically. If you get diffs on another way, you can omit this command.

Next, apply changes to your DB:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID &

This starts the daemon that keeps the database up to date. Latest versions require an additional parameter augmented_diffs:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID --augmented_diffs=no &

To add metadata, you must add a parameter to the second command. Instead of the above, run:

nohup ./apply_osc_to_db.sh $REPLICATE_DIR/ $FIRST_MINDIFF_ID --meta &

To see what's going on, watch these log files:

  • $DB_DIR/transactions.log
  • $DB_DIR/apply_osc_to_db.log
  • $REPLICATE_DIR/fetch_osc.log

Setting up the Web API

Note: The dispatcher daemon must be running for the Web API to work.

This section describes one way to setup a basic read-only HTTP based API with OSM3S.

1. Install Apache2

sudo apt-get install apache2

2. Configure Apache2

cd /etc/apache2/sites-available
nano default

Make your default file look something like this:

<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	ExtFilterDefine gzip mode=output cmd=/bin/gzip
	DocumentRoot [YOUR_HTML_ROOT_DIR]
 
	# This directive indicates that whenever someone types http://www.mydomain.com/api/ 
	# Apache2 should refer to what is in the local directory [YOUR_EXEC_DIR]/cgi-bin/
	ScriptAlias /api/ [YOUR_EXEC_DIR]/cgi-bin/
 
 
	# This specifies some directives specific to the directory: [YOUR_EXEC_DIR]/cgi-bin/
	<Directory "[YOUR_EXEC_DIR]/cgi-bin/">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
                #SetOutputFilter gzip
                #Header set Content-Encoding gzip
	</Directory>
 
	ErrorLog /var/log/apache2/error.log
 
	# Possible values include: debug, info, notice, warn, error, crit, alert, emerg
	LogLevel warn
 
	CustomLog /var/log/apache2/access.log combined
 
</VirtualHost>

3. Restart Apache2:

sudo /etc/init.d/apache2 restart

NOTE: If when you restart apache, you receive an error message such as "ExtFilterDefine invalid command", you need to tell apache to install the correct filter module:

a2enmod ext_filter

4. Start the dispatcher process and point it to your database directory:

sudo nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR &

With meta data:

sudo nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta &

Note: to convert this process to a service that starts up when your system boots do this (... in progress)


5. Test your Web-API by sending it the following command:

wget --output-document=test.xml http://[your_domain_or_IP_address]/api/interpreter?data=%3Cprint%20mode=%22body%22/%3E

The xml output document should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<osm-derived>
  <note>
    The data included in this document is from www.openstreetmap.org. It has there been collected 
    by a large group of contributors. For individual attribution of each item please refer to 
    http://www.openstreetmap.org/api/0.6/[node|way|relation]/#id/history 
  </note>
  <meta osm_base=""/>
 
</osm-derived>

Area creation

This section was taken over from http://overpass-api.de/full_installation.html and may need some revision. Please also check the discussion page and add those details which are worth mentioning here.

To use areas with Overpass API, you essentially need another permanent running process that generates the current areas from the existing data in batch runs.

First, you need to copy the rules directory into a subdirectory of the database directory:

cp -pR "../rules" $DB_DIR

The next step is to start a second dispatcher that coordinates read and write operations for the areas related files in the database:

nohup ../bin/dispatcher --areas --db-dir=$DB_DIR &
 
chmod 666 "../db/osm3s_v0.7.*_areas"

The dispatcher has been successfully started if you find a line "Dispatcher just started." in the file transactions.log in the database directory with correct date (in UTC).

The third step then is to start the rule batch processor as a daemon:

nohup ../bin/rules_loop.sh $DB_DIR &

Now we don't want this process to impede the real business of the server. Therefore, I strongly suggest to priorize this process down. To do this, you need to find with

ps -ef | grep rules

the PIDs belonging to the processes rules_loop.sh and ./osm3s_query --progress --rules. Run for each of the two PIDs the commands:

renice -n 19 -p PID
ionice -c 2 -n 7 -p PID

The second command is not available on FreeBSD. This is not at big problem, because this rescheduling just means giving hints to the operating system.

When the batch process has completed its first cycle, all areas get accessible via the database at once. This may take up to 24 hours.

Troubleshooting

runtime error: open64: 2 /osm3s_v0.6.91_osm_base Dispatcher_Client

Note: if you get an output doc that looks more like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>
 
<p>
   The data included in this document is from www.openstreetmap.org. It has there been collected
   by a large group of contributors. For individual attribution of each item please refer to 
   http://www.openstreetmap.org/api/0.6/[node|way|relation]/#id/history 
</p>
 
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 2 /osm3s_v0.6.91_osm_base Dispatcher_Client::1 </p>
 
</body>
</html>

Then it may indicate that the dispatcher process is not running or not configured correctly.

File_Error Address already in use 98 /srv/osm3s/db_dir//osm3s_v0.7.3_osm_base Dispatcher_Server::4

Check for stale lock files in the following two locations before restarting a crashed/killed dispatcher

  • /dev/shm
  • your db directory (a file named osm3s_v*_osb_base).


File_Error 17 /osm3s_v0.6.94_osm_base Dispatcher_Server::1

If you killed (or crashed) the dispatcher daemon and wish to restart it, you might encounter this error (unless you reboot) : There is a lock file : /dev/shm/osm3s_v0.6.94_osm_base that prevent other dispatchers to run while one is allready running. Remove that file (and check that no dispatcher is running) and restart it.

Apache config fails

If you encounter some message like this one when (re)starting the apache server:

# apache2ctl graceful 
Syntax error on line 12 of /etc/apache2/httpd.conf:
Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration
Action 'graceful' failed.

then apache doesnt use mod_headers. you can activate mod_headers by running:

# a2enmod headers
Enabling module headers.
To activate the new configuration, you need to run:
service apache2 restart
# apache2ctl graceful 

After this, apache should start up correctly.