Nominatim/Installation

From OpenStreetMap Wiki
Jump to: navigation, search
Available languages
English français

These instructions are for the installation of Nominatim V2, which can be found in http://github.com/twain47/Nominatim.git.

For instructions for the older Nominatim V1 that was part of osm2pgsql, see Nominatim/Installation_V1.

Prerequisites

Software

Ubuntu/Debian

A bash script to install Nominatim is available at: https://github.com/cyclestreets/nominatim-install

In standard Debian/Ubuntu distributions these should be available as packages.

 apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool automake libproj-dev
 apt-get install gcc proj-bin libgeos-c1 git osmosis
 apt-get install php5 php-pear php5-pgsql php5-json php-db
 apt-get install postgresql postgis postgresql-contrib postgresql-9.1-postgis postgresql-server-dev-9.1

For PBF support you will also need

 apt-get install libprotobuf-c0-dev protobuf-c-compiler

Note that the protobuf library that comes with Ubunutu 10.04LTS is too old. You will need to compile and install your own version. More information on the Osm2pgsql page.

On either some Debian versions and Ubuntu 14.04 LTS, geos requires a second package to be installed. If configure complains about a missing geos/version.h, try

 apt-get install libgeos++-dev

CentOS

Required packages should be installed from the EPEL repositories. To enable it on CentOS, get and install the epel RPM for your version of CentOS by running

 yum install epel-release
 yum update

You need Postgresql 9.x. CentOS still ships with 8.4, so you should enable the postgres repositories and get the latest version from there. Install the required packages:

 yum install postgresql93-server postgresql93-contrib postgresql93-devel postgis2_93-devel

Create a cluster and start the server:

 service postgresql-9.3 initdb
 service postgresql-9.3 start

Note that all commands are dependent on the postgresql version, so adjust as required.

Now install the remaining dependencies:

 yum install git make automake gcc gcc-c++ libtool
 yum install php-pgsql php php-pear php-pear-DB libpqxx-devel proj-epsg
 yum install bzip2-devel proj-devel geos-devel libxml2-devel protobuf-c-devel lua-devel

PostgreSQL

Currently, PostgresSQL 9.x is required. 8.x versions are no longer supported. PostGIS should be version 1.5 or later.

You might want to tune your PostgreSQL installation so that the later steps make best use of your hardware. You should tune the following parameters in your postgresql.conf file (e.g. at /etc/postgresql/9.1/main/postgresql.conf in Ubuntu):

  • shared_buffers (4GB)
  • maintenance_work_mem (16GB/10GB)
  • work_mem (50MB)
  • effective_cache_size (24GB)
  • synchronous_commit = off
  • checkpoint_segments = 100
  • checkpoint_timeout = 10min
  • checkpoint_completion_target = 0.9

The numbers in brackets behind some parameters seem to work fine for 32GB RAM machine. Adjust to your setup. Setting maintenance_work_mem to a relatively high value during the initial import will speed up index creation significantly. However, it should be reduced again afterwards to avoid swapping when autovacuum runs.

For the initial import, you should also set:

  • fsync = off
  • full_page_writes = off

Don't forget to switch them on again afterwards or you risk database corruption.

Autovacuum must not be switched off because it ensures that the tables are frequently analysed.

Hardware

A minimum of 1GB of RAM is required or installation will fail. For a full planet import 32GB or more are recommended.

For a full planet install you will need about 700GB of hard disk space (as of August 2013, take into account that the OSM database is growing fast). SSD disks will help considerably to speed up import and queries but you will need to hack the source code to make use of them.

On poldi the complete initial import required around 2 days. On a 12-core machine with 32GB RAM and standard SATA disks, the initial import (osm2pgsql) takes around 20 hours and the indexing process another 250 hours. Only, 8 parallel threads were used for this setup because I/O speed was the limiting factor. The same machine is able to import the Germany extract in around 4 hours.

First Installation

Getting the source code

You may choose to either use the stable released version (2.3.0) or join the bleeding edge for the latest features and bugs.

Obtaining 2.3.0 (stable release)

Please download and extract the source

 wget http://www.nominatim.org/release/Nominatim-2.3.0.tar.bz2
 tar xvf Nominatim-2.3.0.tar.bz2

Obtaining the Latest Version

Get the latest source code from Github

 git clone --recursive git://github.com/twain47/Nominatim.git
 cd Nominatim
 ./autogen.sh

This will pull in the correct version of osm2pgsql as well. If you do not clone recursively, you will need to manually pull the source with git submodule update --init.

Compiling the Required Software

Compile the source:

 cd Nominatim
 ./configure
 make

On CentOS, if you used the postgres repositories, you need to give configure the location of pg_config like this: ./configure --with-postgresql=/usr/pgsql-9.3/bin/pg_config

Customization of the Installation

You can customize Nominatim by creating a local configuration file settings/local.php. Have a look at settings/settings.php for available configuration settings.

Here is an example of a local.php:

 <?php
   // Paths
   @define('CONST_Postgresql_Version', '9.1');
   // Website settings
   @define('CONST_Website_BaseURL', 'http://mysite/nominatim/');

Again, CentOS has non-standard locations for postgresql, so a minimum configuration should look like this:

 <?php
   // Paths
   @define('CONST_Postgresql_Version', '9.3');
   @define('CONST_Postgis_Version', '2.1');
   @define('CONST_Path_Postgresql_Contrib', '/usr/pgsql-9.3/share/contrib');
   // Website settings
   @define('CONST_Website_BaseURL', 'http://mysite/nominatim/');

With Ubuntu and PostgreSQL 9.3, you need to configure a local.php file as follow:

 <?php
   // Paths
   @define('CONST_Postgresql_Version', '9.3');
   @define('CONST_Postgis_Version', '2.1');

Download (optional) data

Wikipedia rankings

Wikipedia can be used as an optional auxiliary data source to help indicate the importance of osm features. Nominatim will work without this information but it will improve the quality of the results if this is installed. This data is available as a binary download.

 wget --output-document=data/wikipedia_article.sql.bin http://www.nominatim.org/data/wikipedia_article.sql.bin
 wget --output-document=data/wikipedia_redirect.sql.bin http://www.nominatim.org/data/wikipedia_redirect.sql.bin

Combined the 2 files are around 1.5GB and add around 30GB to the install size of nominatim. They also increase the install time by an hour or so.

UK postcodes

Nominatim can use postcodes from an external source to improve searches that involve a UK postcode. This data can be optionally downloaded:

 wget --output-document=data/gb_postcode_data.sql.gz http://www.nominatim.org/data/gb_postcode_data.sql.gz

Creating postgres accounts

Creating the importer account

The import needs to be done with a postgres superuser with the same name as the account doing the import. You can create such a postgres superuser account by running:

 sudo -u postgres createuser -s <your user name>

Where <your user name> is the name of the account that will be used to perform the installation. You should ensure that this user can log in to the database without requiring a password (e.g. using ident authentication). This is the default on most distributions. See trust authentication for more information.

Create website user

Create the website user www-data:as a PostgreSQL database role

 createuser -SDR www-data

For the installation process, you must have this user. If you want to run the website under another user, see comment in section Set up the website.

You must not run the import as user www-data or root.

Nominatim module reading permissions

Some Nominatim Postgres functions are implemented in the nominatim.so C module that was compiled in one of the earlier steps. In order for these functions to be successfully created, PostgreSQL server process has to be able to read the module file. Make sure that directory and file permissions allow the file to be read. For example, if you downloaded and compiled Nominatim in your home directory, you will need to issue the following commands:

 chmod +x ~/src
 chmod +x ~/src/Nominatim
 chmod +x ~/src/Nominatim/module

Import and index OSM data

First download a Planet File or a planet extract, for example from Geofabrik. Using a file in PBF format is recommended.

Now start the import:

 ./utils/setup.php --osm-file <your planet file> --all [--osm2pgsql-cache 18000] 2>&1 | tee setup.log

The --osm2pgsql-cache parameter is optional but strongly recommended for planet imports. It sets the node cache size for the osm2pgsql import part (see -C parameter in osm2pgsql help). 18GB are recommended for a full planet import, for excerpts you can use less. Adapt to your available RAM to avoid swapping.

The import will take as little as an hour for a small country extract and as much as 10 days for a full-scale planet import. It produces a lot of log messages, which you should carefully examine. The last part of the command ensures that all output is logged into a file. Make sure that you have this log file ready when asking for support for the installation. We recommend running an import of a small osm/pdf file (e.g. Luxembourg) before attempting a full planet import to verify that everything is working.

Add special phrases

Add country codes and country names to the search index:

 ./utils/specialphrases.php --countries > specialphrases_countries.sql
 psql -d nominatim -f specialphrases_countries.sql

If you want to be able to search for special amenities like pubs in Dublin, you need to import special phrases from this wiki like this:

 ./utils/specialphrases.php --wiki-import > specialphrases.sql
 psql -d nominatim -f specialphrases.sql

This may be repeated from time to time when there are changes in the wiki. There is no need to repeat it after each update.

If you do not need phrases for all languages, edit utils/specialphrases.php and delete unneeded languages at the beginning of the file.

Set up the website

The following instructions will make Nominatim available at http://localhost/nominatim.

Create the directory for the website and make sure it is writable by you and readable by apache:

 sudo mkdir -m 755 /var/www/nominatim
 sudo chown <your username> /var/www/nominatim

Populate the website directory with the necessary symlinks:

 ./utils/setup.php --create-website /var/www/nominatim

You will need to make sure settings/local.php is configured with correct values for CONST_Website_BaseURL. e.g.

 @define('CONST_Website_BaseURL', 'http://localhost/nominatim');

Configure for use with Apache

Make sure your Apache configuration (/etc/apache2/sites-enabled/000-default in a standard Ubuntu/Debian installation) contains the following settings for the directory:

 <Directory "/var/www/nominatim/">
    Options FollowSymLinks MultiViews
    AddType text/html   .php     
 </Directory>

After making changes in the apache config you need to restart apache.

 sudo apache2ctl graceful

Note: The name of the website user is hard-coded into Nominatim. In most Linux distributions, apache will run as www-data, so this will work without any further modifications. If your web server runs under a different name (e.g. in Fedora and CentOS apache runs as user apache), simply alter the name of the www-data user in postgresql after the import has finished, e.g. psql -d nominatim -c 'ALTER USER "www-data" RENAME TO "apache".

Configure for use with Nginx

Install nginx and php-fpm as server-side, HTML-embedded scripting language (FPM-CGI binary) that runs as a daemon and receives Fast/CGI requests passed from nginx.

 apt-get install nginx php5-fpm

If you want to change the daemon to listen on unix socket instead configure the pool listener (/etc/php5/fpm/pool.d/www.conf in a standard Ubuntu/Debian installation)

   ; Comment out the tcp listener and add the unix socket
   ;listen = 127.0.0.1:9000
   listen = /var/run/php5-fpm.sock
   : Ensure that the daemon runs as the correct user
   listen.owner = www-data
   listen.group = www-data
   listen.mode = 0666

Tell nginx that php files are special and to fastcgi_pass to the php-fpm unix socket by adding the location definition to the default configuration. (/etc/nginx/sites-available/default in a standard Ubuntu/Debian installation)

   location ~ [^/]\.php(/|$) {
           fastcgi_split_path_info ^(.+?\.php)(/.*)$;
           if (!-f $document_root$fastcgi_script_name) {
                   return 404;
           }
           fastcgi_pass unix:/var/run/php5-fpm.sock;
           fastcgi_index index.php;
           include fastcgi_params;
   }

Restart the nginx and php5-fpm services and view your home brew Nominatim indexed Open Street Map with your favourite browser.

Updates

There are many different possibilities to update your Nominatim database. The following section describes how to keep it up-to-date with osmosis. For a list of other methods see the output of ./utils/update.php --help.

Installing the newest version of osmosis

The version of osmosis supplied with most distributions is too old for running diffs other than minutely diffs. Get the newest version as described on the Osmosis page and unpack it somewhere. Then tell Nominatim to use this version by adding the following line to your settings/local.php:

 @define('CONST_Osmosis_Binary', '<path to osmosis>/bin/osmosis')

Setting up the update process

Next the update needs to be initialised. By default nominatim is configured to update using the global minutely diffs.

If you want a different update source you will need to add some settings to settings/local.php. For example, to use the daily country extracts diffs for Ireland from geofabrik add the following:

 @define('CONST_Replication_Url', 'http://download.geofabrik.de/europe/ireland-and-northern-ireland-updates');
 @define('CONST_Replication_MaxInterval', '40000');     // Process each update separately, osmosis cannot merge multiple updates
 @define('CONST_Replication_Update_Interval', '86400');  // How often upstream publishes diffs
 @define('CONST_Replication_Recheck_Interval', '900');   // How long to sleep if no update found yet


First you have to delete existing 'configuration.txt' then run the following command to create the osmosis configuration files:

 ./utils/setup.php --osmosis-init

Enabling hierarchical updates

When a place is updated in the database, all places that contain this place in their address need to be updated as well. These hierarchical updates are disabled by default because they slow down the initial import. Enable them with the following command:

 ./utils/setup.php --create-functions --enable-diff-updates

Updating Nominatim

The following command will keep your database constantly up to date:

 ./utils/update.php --import-osmosis-all --no-npi

If you have imported multiple country extracts and want to keep them up to date, have a look at the script in this issue.

Troubleshooting

When running the setup.php script I get a warning: PHP Warning: file_get_contents(): open_basedir restriction in effect.

You need to adjust the open_basedir setting in your PHP configuration (php.ini file). By default this setting may look like this:

 open_basedir = /srv/http/:/home/:/tmp/:/usr/share/pear/

Either add reported directories to the list or disable this setting temporarily by adding ";" at the beginning of the line. Don't forget to enable this setting again once you are done with the PHP command line operations.

The Apache log contains lots of PHP warnings like this: PHP Warning: date_default_timezone_set() function.

You should set the default time zone as instructed in the warning in your php.ini file. find the entry about timezone and set it to something like this:

 ; Defines the default timezone used by the date functions
 ; http://php.net/date.timezone
 date.timezone = 'Etc/UTC'

nominatim.so version mismatch

  COPY_END for place failed: ERROR: incompatible library "/opt/Nominatim/module/nominatim.so": version mismatch

pg_config seems to use bad includes sometimes when multiple versions of PostgreSQL are available in the system. Make sure you remove the server development libraries (postgresql-server-dev-9.1 on Ubuntu) and recompile (./configure && make).

I accidentally killed the import process after it has been running for many hours. Can it be resumed?

It is possible if the import already got to the indexing stage. Check the last line of output that was logged before the process was killed. If it looks like this:

  Done 844 in 13 @ 64.923080 per second - Rank 26 ETA (seconds): 7.886255

then you can resume with the following command:

  ./utils/setup.php --index --create-search-indices

If the reported rank is 26 or higher, you can also safely add --index-noanalyse.

Reverse is broken. I get an error XML Parsing Error: XML or text declaration not at start of entity Location.

Make sure there are no spaces at the beginning of your settings/local.php file.

I've successfully imported an excerpt of the US but my results differ from OSM's Nominatim instance. All the house numbers seem to be missing!

In the US, the OSM instance of Nominatim uses TIGER address data to complement the still sparse OSM house number data. You can add TIGER data to your own Nominatim instance by following these steps:

  • Get the GDAL library and python bindings (on Ubuntu: apt-get install python-gdal)
  • Get the TIGER 2013 data. You will need the EDGES files.
  • Extract the relevant data: ./utils/imports.php --parse-tiger-2011 <tiger directory>
  • Import the data into your Nominatim database: ./utils/setup.php --import-tiger-data
  • If you have an apache user different than `www-data` then you have to grant it access to the tables,
    e.g. GRANT SELECT ON location_property_tiger_import TO "apache";

Be warned that the import can take a very long time, especially if you are importing all of the US.