User:Breki/Overpass API Installation

From OpenStreetMap Wiki
Jump to: navigation, search

Introduction

This document tries to give a step-by-step guidance on installing Overpass API on your own computer. The installation procedure described was done on Xubuntu 16.04.2 (64 bit desktop image version) using Overpass API v0.7.54.

The procedure is "opinionated" and follows a specific order of installation steps and is not mean to be a general "how-to" on Overpass API installations. I just wanted to describe in detail how I set up Overpass API (after a long and painful process of learning and troubleshooting) that actually works, without many alternative ways of configuring the system and Overpass API.

Since I am a Linux newbie, I cannot guarantee all the steps in this procedure are the "right way" of doing things on Linux. If you spot any issues, please make a note in the Talk page. And please do not modify the contents of the main page. Thanks!

Installing Prerequisite Packages

These packages are needed so you can compile Overpass API package and run it. Apache web server package is also included.

sudo apt-get update
sudo apt-get install g++ make expat libexpat1-dev zlib1g-dev apache2 -y

Preparing Overpass API Binaries

Downloading Overpass API Tarball And Compiling The Source Code

We will put all Overpass files into the /home/igor/overpass (~/overpass) directory. For simplicity reasons, Overpass binaries will be built inside the sources directory.

cd Downloads
wget http://dev.overpass-api.de/releases/osm-3s_v0.7.54.tar.gz
cd ..
mkdir overpass
cd overpass
mkdir db
tar -zxvf ../Downloads/osm-3s_v0.7.54.tar.gz
cd osm-3s_v0.7.54
./configure --prefix="`pwd`"
make

Enabling Access To Overpass API Package Files

After the compilation, we make sure the scripts will have the necessary access to the Overpass API package files:

chmod -R 755 ~/overpass

Trying It Out With A Sample OSM Dataset (Optional)

This section describes how you can try Overpass API with a smaller sample OSM dataset (if you do not intend to use the whole Planet or just want to see if the pieces of the puzzle fit together).

Downloading Sample OSM Data

We are using a Isle of Man sample extract from Geofabrik since it is small and (mostly) self-contained.

cd ~/Downloads
wget http://download.geofabrik.de/europe/isle-of-man-latest.osm.bz2
cd ..

Populating Overpass API Database With The Small Sample

For testing reasons, we will do the populating in-process since it should not take too long for such a small dataset:

cd overpass
osm-3s_v0.7.54/bin/init_osm3s.sh ../Downloads/isle-of-man-latest.osm.bz2 db osm-3s_v0.7.54

Running A Sample Query

Now we are ready to run a sample query on the imported data. Enter command

osm-3s_v0.7.54/bin/osm3s_query --db-dir=db

... and then copy & paste this sample query into the prompt:

node["amenity"="post_box"];
out skel;

... and press Ctrl+D to start the query. It should return skeleton OSM data about all post box nodes on Isle of Man.

Populating Overpass API Database (Whole Planet)

Cleaning Up The db Directory

Clean the existing db directory (if you populated it using the sample dataset in the previous section, otherwise skip to the next subsection)

cd ~/overpass
rm -rf db
mkdir db

Populating The Database Via Cloning

Now we need to populate the database via cloning from the dev server. Note that this can take a long time (from several hours to several days, depending on your hardware, internet connection and other things).

You can do this directly in-process (which is easier to follow since the output/progress of the command is immediately visible in the terminal):

cd ~/overpass
osm-3s_v0.7.54/bin/download_clone.sh --db-dir=db --source=http://dev.overpass-api.de/api_drolbr/ --meta=no

... or out-of-process using nohup command so the process can continue to run even if the user logs out:

cd ~/overpass
nohup osm-3s_v0.7.54/bin/download_clone.sh --db-dir=db --source=http://dev.overpass-api.de/api_drolbr/ --meta=no &

(note that the ending & character means the command will run in the background).

The database will be stored in the ~/overpass/db directory.

Configuring The Dispatcher Daemon

Configuring systemd

Before configuring the systemd service, we need to tell systemd not to delete shared memory files created by non-system user accounts that are not logged in (since we will use a non-system user account igor to run system service(s)). To to that, first open /etc/systemd/logind.conf file in the editor:

sudo nano /etc/systemd/logind.conf

and uncomment the line RemoveIPC=yes, change it to RemoveIPC=no, save the file and reboot the system.

Preparing The Dispatcher Daemon Configuration

First we need to prepare a systemd configuration file (called overpass.service) for the dispatcher daemon:

sudo nano /lib/systemd/system/overpass.service

Paste the following into the text editor once it opens and save the file. Note that you need to change the user account the service will be running as (the User directive) and the various parameters that point to the /home/igor/overpass directory:

[Unit]
Description=Overpass init agent

[Service]
Type=simple
User=igor
ExecStartPre=/bin/rm /home/igor/overpass/db/osm3s* -f --verbose
ExecStartPre=/bin/rm /dev/shm/osm3s* -f --verbose
ExecStart=/home/igor/overpass/osm-3s_v0.7.54/bin/dispatcher --osm-base --db-dir=/home/igor/overpass/db

[Install]
WantedBy=multi-user.target

Enabling The Dispatcher Daemon

Now enable and start the Overpass service:

sudo systemctl enable overpass
sudo systemctl start overpass

Troubleshooting The Dispatcher Daemon Problems

You can check the status of the service with the following command:

sudo systemctl status overpass | more

It should report the service as loaded and active (running), and it should also report ExecStartPre as successfully executed, something like the following output:

overpass.service - Overpass init agent
   Loaded: loaded (/lib/systemd/system/overpass.service; enabled; vendor preset: enabled)
   Active: active (running) since ned 2017-07-23 10:14:27 CEST; 1h 47min ago
  Process: 4282 ExecStartPre=/bin/rm /dev/shm/osm3s* -f --verbose (code=exited, status=0/SUCCESS)
  Process: 4280 ExecStartPre=/bin/rm /home/igor/overpass/db/osm3s* -f --verbose (code=exited, status=0/SUCCESS)
 Main PID: 4285 (dispatcher)
   CGroup: /system.slice/overpass.service
           └─4285 /home/igor/overpass/osm-3s_v0.7.54/bin/dispatcher --osm-base --db-dir=/home/igor/overpass/db

Querying The systemd Journal

Also, you can query the systemd journal:

journalctl -xe

(-x means log lines with be augmented with explanation texts from the message catalog, while -e means the end of the journal will be shown).

You can also display the journal entries just for the overpass service:

journalctl -xe -u overpass

To follow the journal, use

journalctl -xf -u overpass

Restarting The Overpass Service

f you made a mistake in the overpass.service configuration file, update the file and then stop and start the service:

sudo systemctl stop overpass
sudo systemctl start overpass

Getting Information Status Information From The Running Dispatcher

You can get status info about from the running dispatcher:

~/overpass/osm-3s_v0.7.54/bin/dispatcher --status

For the list of all available command line parameters, execute

~/overpass/osm-3s_v0.7.54/bin/dispatcher --help

Configuring Diffs

We will register another systemd service that will perform the task of continually updating the database with the latest changes. First we need to prepare a systemd configuration file (called overpass-diff.service) for the dispatcher daemon:

sudo nano /lib/systemd/system/overpass-diff.service

Paste the following into the text editor once it opens and save the file. Note that you need to change the user account the service will be running as (the User directive) and the various parameters that point to the /home/igor/overpass directory:

[Unit]
Description=Overpass diff init agent
Requires=overpass.service
After=overpass.service

[Service]
Type=simple
User=igor
ExecStart=/home/igor/overpass/osm-3s_v0.7.54/bin/fetch_osc_and_apply.sh http://planet.osm.org/replication/minute/ --meta=no

[Install]
WantedBy=multi-user.target

Also note we configured this service to be dependent on the main dispatcher overpass.service (Requires directive) and that it should be started after that service (After directive). You can use the same instructions as for the main service to register, start and troubleshoot it. You can also use

tail -f apply_osc_to_db.log

to follow what is going on with the service.

Setting Up The Web API

The goal of this section is to set up the web access to Overpass API using Apache web server. In my case I wanted to keep the default virtual host (on port 80) as it is, so I created a new virtual host on port 8091 just for Overpass API (as described below).

Preparing Apache Global Configuration

This step is highly dependent on your specific server/Apache configuration, but I'm writing it nonetheless: make sure you have the ServerName directive specified in the Apache's global configuration file (/etc/apache2/apache2.conf):

echo "ServerName localhost" | sudo tee /etc/apache2/conf-available/servername.conf && sudo a2enconf servername

(in my case, my server is called "jazz" and I will be using that name later in the virtual host configuration file).

Telling Apache To Listen To Port 8091

Since I will be using port 8091 for the Overpass API virtual host, we first need to tell Apache to listen to that port. This is accomplished by adding a new line in the listening ports configuration file (/etc/apache2/ports.conf):

Listen 8091

Opening Port 8091 In The Firewall

In order for the Overpass API website (and the web service) to be accessible remotely, we need to open the 8091 port on the firewall:

sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 8091 -j ACCEPT

Enabling Required Apache Modules

We need to enable the CGI and ext_filter support in Apache2:

sudo a2enmod cgi
sudo a2enmod ext_filter

Preparing Virtual Host Configuration

Now we need to prepare the configuration for the new virtual host, by copying the configuration of the default virtual host and using it as a template:

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/overpass.conf

Now open the newly created overpass.conf file to edit it:

Paste the following into the text editor once it opens and save the file (note the various parameters point to the /home/igor/overpass directory, you need to change them for your own installation):

<VirtualHost *:8091>
   ServerAdmin webmaster@localhost
   ExtFilterDefine gzip mode=output cmd=/bin/gzip

   DocumentRoot /home/igor/overpass/osm-3s_v0.7.54/html

   ScriptAlias /api/ /home/igor/overpass/osm-3s_v0.7.54/cgi-bin/

   <Directory "/home/igor/overpass/osm-3s_v0.7.54">
      AllowOverride None
      Options Indexes FollowSymLinks
      Require all granted
   </Directory>

   <Directory "/home/igor/overpass/osm-3s_v0.7.54/cgi-bin/">
      AllowOverride None
      Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
      Require all granted
   </Directory>

   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Enabing The Virtual Host

Now we tell Apache to enable the virtual host we prepared:

sudo a2ensite overpass.conf

Reloading Apache Configuration

Now we tell Apache to reload the whole configuration so we can test Overpass API's website:

sudo service apache2 reload

Testing Overpass API's Website

In your browser, navigate to http://jazz:8091. It should display the entry page of Overpass API website with various links. You can navigate to the "Query and Convert Forms" link and enter a sample query there:

<query type="node"><bbox-query n="51.0" s="50.9" w="6.9" e="7.0"/><has-kv k="amenity" v="pub"/></query><print/>

Troubleshooting Overpass API Web Access

In case of problems, check Apache's error and access logs:

tail -f /var/log/apache2/error.log
tail -f /var/log/apache2/access.log

You can also analyze Apache's configuration using the following command:

sudo /usr/sbin/apache2 -S

Links And Resources

For Future Reference