OpenGastroMap/Installation

From OpenStreetMap Wiki
Jump to navigation Jump to search

This page describes how to install the software you need for creating a map like OpenGastroMap.de (see also OSM Wiki on OpenGastroMap.de).

It's not too complicated, feel free to try it yourself! As a result you will get a specialized map running on you own internet server. You can program this map to display every OpenStreetMap item you want in an extra layer. Furthermore, these items can be made clickable, so extra properties may be shown on the user's request.

Many thanks to Stephan Knauß who developed the first example and licensed it under GPL!

Prerequisites

Hardware

A weak CPU, e.g. Intel Atom or a virtual Internet server, will suffice, 1 GB RAM are recommended.

Operating System

We assume that you use Ubuntu >=8.04 as operating system. You must install all programs necessary to run your own web server. Additionally, you will need some other packages. Ensure the installation of all necessary packages by performing these commands:

sudo apt-get install apache2 php5 libapache2-mod-php5 subversion autoconf
sudo apt-get install build-essential libxml2-dev libgeos-dev libbz2-dev proj libtool
# for Ubuntu <=9.10:
  sudo apt-get install postgresql-8.3-postgis postgresql-contrib-8.3 postgresql-server-dev-8.3
# for Ubuntu >=10.04:
  sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4 postgresql-server-dev-8.4

Database

The GIS databases must be initialized. Please follow these steps:

sudo -u postgres -i -H
createuser -SdR gisuser
createdb -E UTF8 -O gisuser gis0
createdb -E UTF8 -O gisuser gis1
createlang plpgsql gis0
createlang plpgsql gis1
# for Ubuntu <=9.10:
  psql -d gis0 -f /usr/share/postgresql/8.3/contrib/_int.sql
  psql -d gis1 -f /usr/share/postgresql/8.3/contrib/_int.sql
  psql -d gis0 -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
  psql -d gis1 -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
# for Ubuntu 10.04:
  psql -d gis0 -f /usr/share/postgresql/8.4/contrib/_int.sql
  psql -d gis1 -f /usr/share/postgresql/8.4/contrib/_int.sql
  psql -d gis0 -f /usr/share/postgresql/8.4/contrib/postgis.sql
  psql -d gis1 -f /usr/share/postgresql/8.4/contrib/postgis.sql
# for Ubuntu >=10.10:
  psql -d gis0 -f /usr/share/postgresql/8.4/contrib/_int.sql
  psql -d gis1 -f /usr/share/postgresql/8.4/contrib/_int.sql
  psql -d gis0 -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
  psql -d gis1 -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql gis0 -c "ALTER TABLE geometry_columns OWNER TO gisuser"
psql gis1 -c "ALTER TABLE geometry_columns OWNER TO gisuser"
psql gis0 -c "ALTER TABLE spatial_ref_sys OWNER TO gisuser"
psql gis1 -c "ALTER TABLE spatial_ref_sys OWNER TO gisuser"
exit

To enable easy database login by user gisuser you must change some lines in one of the database configuration files. In case you are running Ubuntu without a graphical interface, please use another editor, e.g. nano instead of gedit.

# for Ubuntu <=9.10:
  sudo gedit /etc/postgresql/8.3/main/pg_hba.conf
# for Ubuntu >=10.04:
  sudo gedit /etc/postgresql/8.4/main/pg_hba.conf

Near to the bottom of the file you will find these lines:

local   all         all                               ident
host    all         all         127.0.0.1/32          md5

Change the words ident and md5 each to trust and close the editor. Now reload the database configuration:

# for Ubuntu <=9.10:
  sudo /etc/init.d/postgresql-8.3 reload
# for Ubuntu >=10.04:
  sudo /etc/init.d/postgresql-8.4 reload
# for Ubuntu 10.10:
  sudo /etc/init.d/postgresql reload

For a short test, login to the database by using the previously created database user gisuser:

psql gis0 gisuser

Type \d to see a list with the two tables which we have created some steps above (geometry_columns and spatial_ref_sys). Then logout with \q and try the same test with gis1 instead of gis0.

As you have noticed, we use two databases: gis0 and gis1. That is, because we do not want to interrupt web user requests during the daily running process of reconstructing the data base contents. So we use the gis0 database at even numbered days and the gis1 at odd ones. To calculate if a day is even or odd, the UNIX time is retrieved by a "date +%s" command and divided by 86400.

A second database consumes additional memory, of course. But this should not be a serious problem as we are going to condense the OSM planet data to an absolute minimum before we fill the databases.

If you encounter any problems, you may find a solution here: PostGIS.

Choose a Directory

Our example directory for downloading OSM data and generating the data base contents will be "/usr/share/osmgen". So, create this directory and jump into it:

sudo mkdir /usr/share/osmgen
cd /usr/share/osmgen

Afterwards, change the access rights of this directory so you have write access with your regular user:

sudo chmod a+rwx .

For security reasons, do not forget to revoke write access as soon as the installation is complete:

sudo chmod -R go-w /usr/share/osmgen   # do not run this command now, run it later!
sudo chown -R root:root /usr/share/osmgen   # do not run this command now, run it later!

Tool osm2pgsql

We will need the tool osm2pgsql to write OSM data into the database. Although provided by Ubuntu community, it's strongly recommended to build osm2pgsql from source, because we rely on getting the newest version. Please follow these steps:

cd /usr/share/osmgen
svn export https://svn.openstreetmap.org/applications/utils/export/osm2pgsql
cd osm2pgsql
./autogen.sh
./configure
make

Now the databases need to be initialized for Spherical Mercator projection:

cd /usr/share/osmgen
psql gis0 gisuser -f osm2pgsql/900913.sql
psql gis1 gisuser -f osm2pgsql/900913.sql

If you encounter any problems, you may find a solution here: osm2pgsql.

Tool osmfilter

This tool is used to speed-up the PostgreSQL import process. If we filter out all information we want to use and drop everything else, osm2pgsql will run much faster. On top of this, database queries will be faster too, so the map's user will get the page loaded more fluently.

Downloading the binary of osmfilter (to get the 64 bit version replace "32" by "64"):

cd /usr/share/osmgen
wget -O - http://m.m.i24.cc/osmfilter32 >osmfilter
chmod ug+x osmfilter

For further details on this tool refer to osmfilter.

Fill the Database

We do not need to store map tiles on our server because we fetch all tiles from https://www.openstreetmap.org/ for performance reasons, but all feature data we want to display on an extra layer must be stored in our database. To do this, we need to develop a script which performs several tasks, step by step. Please do not try to create and run the whole script from scratch. It's better to run-through the process step by step, entering every single command at the command line terminal. That makes it much easier to find errors.

Get the Planet File

It is not recommended to use the file of the entire planet. Please choose the file of an area you are interested in, e.g. Germany. The first task of our script will be to download this file:

cd /usr/share/osmgen
wget -O - http://download.geofabrik.de/osm/europe/germany.osm.bz2 2>/dev/null |bunzip2 |gzip -1 >a.osm.gz

Although the file is already available in compressed form, we decided to recompress it, because gzip works a lot faster then bzip2.

Filter the Planet File

./osmfilter a.osm.gz --keep="amenity=restaurant =bar =pub =cafe =fast_food =food_court =nightclub" -o=gis.osm

A few minutes later – dependent of the size of the chosen area – we will get the file gis.osm, containing only that information we need.

Transfer the Data to Postgres Database

The program osm2pgsql will do this job. First you have to ensure that the executable osm2pgsql is available. It should exist in this directory "/usr/share/osmgen/osm2pgsql". Then we need to edit the file "default.style" which should be found in the same directory as the executable.

gedit /usr/share/osmgen/osm2pgsql/default.style

This default file must be customized to fulfill our needs: all unused tags should be removed by preceding them with a #, and our special tags must be added to this file. As a result, only these lines remain valid, resp. have been added to the file:

# OsmType Tag DataType Flags
node,way amenity text nocache,polygon
node,way disused text linear
node,way name text linear
node,way cuisine text linear
node,way smokefree text linear
node,way smoking text linear
node,way smoking_hours text linear
node,way smoking:outside text linear

Now we can transfer the OSM data from the file gis.osm into the Postgres database:

osm2pgsql/osm2pgsql -s -C 700 -d gis0 -U gisuser -S osm2pgsql/default.style gis.osm

Shell Script

Did all previous steps work without any errors? Then it's time to pack them into a single script:

cd /usr/share/osmgen
gedit gen.sh

That content could be like this:

#!/bin/bash
cd /usr/share/osmgen

# rotate the log and write a new headline
mv gen.log gen.log_temp
tail -2000 gen.log_temp>gen.log
rm gen.log_temp
echo >>gen.log
echo Starting update script >>gen.log
date >>gen.log

# get the latest .osm file
# download section - start
rm a.osm.gz
echo Downloading .osm file >>gen.log
wget -O - http://download.geofabrik.de/osm/europe/germany.osm.bz2 2>/dev/null |bunzip2 |gzip -1 >a.osm.gz
# download section - end
ls -lL a.osm.gz >>gen.log
if [ "0"$(stat -L -c%s a.osm.gz) -lt 900000 ]; then
  echo "Downloaded file too small (implausible)" >>gen.log
  date >>gen.log
  exit
fi
date >>gen.log

# filter the .osm file
./osmfilter a.osm.gz --keep="amenity=restaurant =bar =pub =cafe =fast_food =food_court =nightclub" -o=gis.osm
echo File has been filtered. >>gen.log
date >>gen.log

# generate database contents
d=$(date +%s)
echo Unix time: $d >>gen.log
d=$(expr $d + 39600 + 300) # 11*3600 means: at about 11:00 UT;
  # i.e., this script should be started at 11:00 UT (or later)
d=$(expr $d / 86400 % 2)
  #d=$(expr 1 - $d)
echo Determined database name: gis$d >>gen.log
echo Generating data base tables >>gen.log
./osm2pgsql -s -C 700 -d gis$d -U gisuser -S gastro.style gis.osm 2>/dev/null >/dev/null
date >>gen.log
echo >>gen.log

Performance considerations

The typical process duration time for this script is 7 minutes on a slow machine (virtual web server with 1 GB RAM assigned):

  • 10 minute for getting the planet file (Germany),
  • 5 minutes for unpacking/filtering, and
  • 1 minute for writing the result into the database.

Weekly Map Data Update

To get the map data updated automatically every week, an additional Cron entry will be mandatory. For example, execute this command to start the update process every Sunday at 3 p.m.:

sudo echo -e "0 15 * * 7 root /usr/share/osmgen/gen.sh 2>&1>/dev/null \n" > /etc/cron.d/osmgen

Daily Map Data Update

Instead of downloading the whole regional .osm file, you can also use the mechanism for a daily update. After you have followed the description in that page, create a symbolic link to the automatically updated .osm file (insert the right path):

cd /usr/share/osmgen
ln -fs /insert_here_the_actual_path_to_osmupdate_directory/a.osm.gz a.osm.gz

If you use the shell script from above, you will no longer need the script's download part. Hence, remove the 3 lines which are included by the # download section markings.

Supply Web Content

The web server Apache will expect our web content at "/var/www". So, change to that directory and install all necessary files:

16pxb.png
16pxg.png
16pxr.png
16pxy.png
ajax-loader.gif
b.png
cb.png
cg.png
cr.png
cui_afghan.png
...
cui_vietnamese.png
cy.png
features.php
g.png
index.html
r.png
y.png

The most important files can be found here:

Note that you will need write access rights in the www directory. If these rights have not been set automatically during Apache installation, you may want to do this by hand:

sudo chown root:www-data /var/www
sudo chmod g+rwx /var/www
sudo usermod -aG www-data $USER

To get these changes work, please relogin.

Because we are going to use some other libraries, download copies of their files and place them into your www directory. Afterwards delete all library files we do not need:

cd /var/www
wget http://code.jquery.com/jquery.min.js
wget https://www.openstreetmap.org/openlayers/OpenStreetMap.js
wget http://openlayers.org/download/OpenLayers-2.9.zip
unzip OpenLayers-2.9.zip
mv OpenLayers-2.9 ol
cd ol
rm -r apidoc_config build doc doc_config examples lib tests tools

Test your new Map

Open a web browser and try to access your new website. If your browser is installed on the same computer as the Apache server, type localhost as URL.

Benchmarks

(Please add comments.)

Troubleshooting

(chapter needs to be written...)

Clustering

Stephan developed a nice server sided clustering system