Virtual machine image

From OpenStreetMap Wiki
Jump to: navigation, search

In order to download this virtual machine image use the following magnet link to download it via BitTorrent:

magnet:?xt=urn:btih:IYDWGYFZYN5CLYRYZOSPH7A7IOKLKPNY&tr=udp://tracker.openbittorrent.com:80

Or you can download the traditional .torrent file for loading into your client as well:

[1]

Get Started

Login as user "osm", password "osm". Then "startx" to get a desktop environment.

Note 1: Because of URL changes the included script to download coastlines is broken. Before you do anything, please download the latest version from https://trac.openstreetmap.org/browser/subversion/applications/rendering/mapnik/get-coastlines.sh ("Original Format" link at the bottom). chmod +x it and put it into /usr/bin/ , overwriting the existing script. Otherwise the tile rendering will fail hard.

If it fails anyways, run "renderd -f" in a terminal, it might help you debug. You might need to remove a "ne_" prefix from files in /usr/share/mapnik-osm-data/world_boundaries/ .

Note 2: Please note that there is a bug in the nominatim configuration files in version 0.1 of the VM image. The file /home/osm/src/Nomintatim/settings/local.php should not have a leading space on line 1. This extra space gets put at the beginning of the nominatim search results and results in a malformed XML response, giving an XML parsing error. The error is as follows:

This page contains the following errors: error on line 1 at column 6: XML declaration allowed only at the start of the document


Ok, ready? Then doubleclick the "Start Configuration Wizard" on the desktop and choose to run it in a terminal. Follow the instructions and have fun.

"OSM Server" Virtual Machine Image Creation Notes

This section describes the raw operating system setup that was installed to the virtual hard drive. If you just want to run it, this is not important to you.

  • Started By downloading Ubuntu 12.04 Long Term Support, Desktop build, 64 bit.
  • Created a Virtualbox image with a 'dynamically expanding' virtual hard drive* with a max size of 1.5 tb.
  • Installed Ubuntu to the virtual hard drive via a temporary virtual cdrom image mounted in virtualbox.
    • Install language: English
    • Checked 'Download updates while installing', Did not check 'Install Third part Software' (mp3 decoders and such)
    • Chose the default 'Erase disk and install Ubuntu' since the virtual disk image is currenly blank.
    • Left the entire 1.6 tb reported size limit as a single hard disk partition.
    • Chose time zone and location of London, UK to make server local time the same as GMT, and in honor of OSM's UK beginnings.
    • Keyboard Layout is English (US), wasn't sure about UK vs US keyboards but mine is US so I went with that, should be easy to change later on if need be.
    • Entered the following into the fields:
      • Your name: OSM User
      • Your Computer's Name: osm-server
      • Pick a username: osm
      • Choose a password: osm
      • Lastly I chose 'Log in automatically' as this is easiest for users, it disables the 'Encrypt my home folder' option but this is not a problem for a server image such as this.

First Boot from the Virtual Hard Drive

After the installation was finished, I immediately changed a few settings for the base OS and "shut down" the image to make the checkpoint at the end of this section.


  • Got a notice saying new drivers are available, installed the 'x86 virtualization solution - guest module' driver
  • Opened a terminal with Ctrl+Alt+T and used apt to remove as much of the unnecessary initial installed base as possible. (NOTE: I installed synaptic to do this, on a repeat of this process users should just use the apt command directly as there is no need for synaptic to ever be installed).
osm@osm-server:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       1.5T   26G  1.4T   2% /
udev            490M  4.0K  490M   1% /dev
tmpfs           200M  776K  199M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M  124K  498M   1% /run/shm

osm@osm-server:~/Desktop$ sudo apt-get purge ubuntu-artwork aspell-en bluez-alsa libpurple0 ubuntu-desktop telepathy-indicator libreoffice-gnome unity libreoffice-draw libreoffice-help-en-us espeak telepathy-logger branding-ubuntu gnome-media aisleriot libcompizconfig0 compiz-plugins-default libreoffice-style-tango baobab aspell gwibber-service-identica rhythmbox-data libreoffice-impress thunderbird-locale-en-us bluez-gstreamer totem-plugins activity-log-manager-control-center gstreamer0.10-alsa mythes-en-us gnome-disk-utility thunderbird rhythmbox gstreamer0.10-gconf telepathy-haze unity-scope-musicstores gstreamer0.10-plugins-base-apps empathy-common bluez-cups lightdm rhythmbox-plugin-magnatune gnome-sudoku gwibber-service-twitter gwibber-service-facebook telepathy-idle empathy checkbox libtotem0 telepathy-salut bluez zeitgeist-datahub zeitgeist gstreamer0.10-plugins-good toshset libfarstream-0.1-0 compizconfig-backend-gconf libreoffice-math friendly-recovery gir1.2-totem-1.0 deja-dup libreoffice-gtk rhythmbox-ubuntuone libreoffice-common uno-libs3 python-uno speech-dispatcher libgwibber2 espeak-data gnome-bluetooth gnome-user-share libaa1 compiz-gnome remmina-common libtelepathy-farstream2 ubuntu-wallpapers-precise rhythmbox-mozilla gstreamer0.10-pulseaudio libespeak1 gwibber-service libreoffice-style-human libfolks-eds25 gstreamer0.10-x ure dmz-cursor-theme libreoffice-base-core example-content remmina-plugin-rdp zeitgeist-core rhythmbox-plugin-zeitgeist brasero-cdrkit indicator-datetime ubuntu-wallpapers brasero-common libreoffice-calc thunderbird-globalmenu mahjongg thunderbird-gnome-support remmina-plugin-vnc evolution-data-server-common gnome-games-data gstreamer0.10-tools evolution-data-server gnome-online-accounts checkbox-qt brasero libreoffice-emailmerge compiz compiz-plugins-main-default dvd+rw-tools gwibber compiz-core totem libedataserverui-3.0-1 gstreamer0.10-nice totem-mozilla libgwibber-gtk2 rhythmbox-plugin-cdrecorder rhythmbox-plugins gstreamer0.10-plugins-base totem-common light-themes adium-theme-ubuntu activity-log-manager-common thunderbird-locale-en libbrasero-media3-1 nautilus-sendto-empathy libreoffice-core telepathy-mission-control-5 remmina libreoffice-writer gnomine

[sudo] password for osm: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED
  activity-log-manager-common* activity-log-manager-control-center* adium-theme-ubuntu* aisleriot*
  aspell* aspell-en* baobab* bluez* bluez-alsa* bluez-cups* bluez-gstreamer* branding-ubuntu* brasero*
  brasero-cdrkit* brasero-common* checkbox* checkbox-qt* compiz* compiz-core* compiz-gnome*
  compiz-plugins-default* compiz-plugins-main-default* compizconfig-backend-gconf* deja-dup*
  dmz-cursor-theme* dvd+rw-tools* empathy* empathy-common* espeak* espeak-data* evolution-data-server*
  evolution-data-server-common* example-content* friendly-recovery* gir1.2-totem-1.0* gnome-bluetooth*
  gnome-disk-utility* gnome-games-data* gnome-media* gnome-online-accounts* gnome-sudoku*
  gnome-user-share* gnomine* gstreamer0.10-alsa* gstreamer0.10-gconf* gstreamer0.10-nice*
  gstreamer0.10-plugins-base* gstreamer0.10-plugins-base-apps* gstreamer0.10-plugins-good*
  gstreamer0.10-pulseaudio* gstreamer0.10-tools* gstreamer0.10-x* gwibber* gwibber-service*
  gwibber-service-facebook* gwibber-service-identica* gwibber-service-twitter* indicator-datetime*
  libaa1* libbrasero-media3-1* libcompizconfig0* libedataserverui-3.0-1* libespeak1* libfarstream-0.1-0*
  libfolks-eds25* libgwibber-gtk2* libgwibber2* libpurple0* libreoffice-base-core* libreoffice-calc*
  libreoffice-common* libreoffice-core* libreoffice-draw* libreoffice-emailmerge* libreoffice-gnome*
  libreoffice-gtk* libreoffice-help-en-us* libreoffice-impress* libreoffice-math*
  libreoffice-style-human* libreoffice-style-tango* libreoffice-writer* libtelepathy-farstream2*
  libtotem0* light-themes* lightdm* mahjongg* mythes-en-us* nautilus-sendto-empathy* python-uno*
  remmina* remmina-common* remmina-plugin-rdp* remmina-plugin-vnc* rhythmbox* rhythmbox-data*
  rhythmbox-mozilla* rhythmbox-plugin-cdrecorder* rhythmbox-plugin-magnatune*
  rhythmbox-plugin-zeitgeist* rhythmbox-plugins* rhythmbox-ubuntuone* speech-dispatcher* telepathy-haze*
  telepathy-idle* telepathy-indicator* telepathy-logger* telepathy-mission-control-5* telepathy-salut*
  thunderbird* thunderbird-globalmenu* thunderbird-gnome-support* thunderbird-locale-en*
  thunderbird-locale-en-us* toshset* totem* totem-common* totem-mozilla* totem-plugins* ubuntu-artwork*
  ubuntu-desktop* ubuntu-wallpapers* ubuntu-wallpapers-precise* unity* unity-scope-musicstores*
  uno-libs3* ure* zeitgeist* zeitgeist-core* zeitgeist-datahub*
0 upgraded, 0 newly installed, 130 to remove and 20 not upgraded.
After this operation, 460 MB disk space will be freed.
Do you want to continue [Y/n]? 

osm@osm-server:~/Desktop$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       1.5T   25G  1.4T   2% /
udev            490M  4.0K  490M   1% /dev
tmpfs           200M  776K  199M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M  124K  498M   1% /run/shm
  • After the unneeded packages were removed, I did an apt update and upgrade, updating about 20 packages.

At this point the system was shut down and a VM checkpoint was created to avoid having to repeat this process every time.

  • Export Appliance of file:
    • Just after the install and first run to remove the non-essential packages.
  • Take Snapshot: OSM server - 01 - Ubuntu desktop cleaned
    • Just after the install and first run to remove the non-essential packages.

First configuration attempts

Upon booting up the VM I see that the desktop environment is not launching automatically. Logging in at the terminal with osm/osm and typing startx launches the desktop environment. I may keep the system this way as it is actually nice for the user to be able to run the headless servers, etc, but be able to spawn a gui at any time to run configuraton wizards, etc.

Also, the gnome-terminal seems to be missing some fonts or something and you cannot see what you type for commands (commands do exectute properly and their output is shown, just not the commandline you are editing). As a workaround to this problem, I changed the behaviour of the of the Ctrl+Alt+T hotkey to launch the xterm, instead of gnome-terminal. Since the gnome-terminal is no longer needed I removed it as well. In the gconf-editor I navigated to the following key and changed it to run 'xterm'

  • Navigate to key apps/metacity/global_keybindings
    • Find key 'run_command_terminal' and delete its current value of '<Control><Alt>t'
    • Find key 'run_command_1' and give it value '<Control><Alt>t'
  • Navigate to key apps/metacity/keybinding_commands
    • Find key 'command_1' and give it value 'xterm'
sudo apt-get install gconf-editor
gconf-editor
sudo apt-get purge gnome-terminal gnome-terminal-data

fix some missing icon issues:

sudo apt-get install gnome-icon-theme-full

Also removing some cruft on the desktop which was left there from the install process in the first image. Also, right clicking on some of the icons over on the left and undocking them from the launcher to clean up to just the browser, home directory, settings, and trash can.

  • Take Snapshot: OSM server - 02 - Fixing terminal and cleanup desktop environment

Adding a tileserver stack

Now that the base OS has been set up, cleaned of any not needed packages to save space, and the environment has been stabilised, I will begin adding the various OSM stack related components. To begin with I will add the standard renderd based tile server as described at switch2osm on the Building a tile server from packages page.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:kakrueger/openstreetmap
sudo apt-get update
sudo apt-get install libapache2-mod-tile

Also during this time I let the VM go into screensaver while it was downloading a file and upon wakeup it wanted the password to unlock the screen. I disabled the lock By going to System Settings -> Brightness and Lock

  • Turn screen off when inactive for: never
  • Lock: off
  • Require my password when waking from suspend: unchecked

Also for usability I installed the 'human-theme' and logged out and re-ran startx to enable that theme. I also installed the vim runtime environment to make editing files a bit more confortable, this can obviously be skipped (sudo apt-get install vim).

  • Take snapshot: OSM server - 03 - About to install tileserver from apt
    • This was run just before the sudo apt-get install libapache2-mod-tile command was run to allow me to double check the settings on this step. I also added the import script to the desktop and set a few preferences in the firefox browser so they get save at this point.
  • When prompted on whether I want to enable the apache config for 'mod_tile'? I chose the default answer of yes.
  • Do you want to download coastlines 400-500 mb in size? I chose no.
  • Do you want to create a postgis db? I chose the default answer of yes.
  • Name of the database to create: gis
  • Other users that should have access to the db: www-data osm
#!/bin/bash

	if [[ $# -lt 1 ]]
	then
	echo -en "\n\n\n\tUSAGE $0 extract.pbf [extract2.pbf]...\n\n\nThis script imports an extract(s) into the rendering DB, the nominatim DB, and the routing DB.\n\nNOTE:  Currently the script only supports importing a single extract file and it currently only updates the rendering database.\n\n\n"
	exit
	fi

extracts=$@

clear
echo -en "\t--- Importing into the rendering DB ---\n"
echo -en "Running osm2pgsql...\n\n\n"
osm2pgsql --slim -C 512 --cache-strategy sparse $extracts
  • Take snapshot: OSM server - 04 - Rendering db set up and ready for import of extract

Import loading test

At this point I created a "branch" by importing the North Dakota extract and testing out renderd to make sure everything is set up correctly. Since I didn't download and install coastlines renderd was having trouble serving requests, I manually copied the files into place from another computer and renderd then started working successfully. At this point I reverted back to Snapshot 04 so the import is not in the db and no coastline data is present on the system.

Finalizing rendering scripts on the desktop

Having actually tested the import I tweaked the import script a bit. I also added a script to the desktop to run the whole download, import, and tileserver testing process.

Take snapshot: OSM server - 05 - Improvements to user friendliness of import Take snapshot: OSM server - 06 - Fixing typo in import script Take snapshot: OSM server - 07 - Another fix to import script

Begin working on Nominatim

Now that the tileserver is complete I will continue the work on the user scripts and begin working on the Nominatim database import.

To begin with I am tweaking the sudo priveledges on the system to give user OSM the ability to run the sudo command without prompting for a password. I ran the 'sudo visudo' command and edited the sudoers file to have this line for the user osm: osm ALL=NOPASSWD: ALL

Take snapshot: OSM server - 08 - Allow user 'osm' to sudo without typing in a password

Beginning by installing nominatim build pre-requisites and compiling the source code:

sudo apt-get install php5-pgsql postgis postgresql php5 php-pear gcc proj libgeos-c1 postgresql-contrib git postgresql-9.1-postgis postgresql-server-dev-9.1 build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool automake libprotobuf-c0-dev protobuf-c-compiler

sudo pear install DB

mkdir /home/osm/src
cd /home/osm/src
git clone --recursive git://github.com/twain47/Nominatim.git
cd Nominatim
./autogen.sh
./configure --enable-64bit-ids
 make

For the time being I am skipping over the 'Tuning PostgreSQL' Section of the guide. Should prbably revisit this later, or on a second build.

I am also skipping over adding a local.php file since it is not easy to customize that file for the time being. I should add a thing to the install script that lets you edit the file with the zenity file loading/editing system.

I am also skipping over the optional wikipedia article data for now. I will probably do this later in a manner similar to the coastline download.

  • Take snapshot: OSM server - 09 - Nominatim built, ready to start installing it

Creating users:

sudo su postgres
createuser -SDR www-data
createuser -s osm

Then exiting the shell for user postgres and returning to the Nomitatim directory:

chmod +x ~/src
chmod +x ~/src/Nominatim
chmod +x ~/src/Nominatim/module
  • Take snapshot: OSM server - 10 - Nominatim set up ready for import

Next I finalize the script on the desktop with what should be ready to do a Nominatim import. I also create the necessary symlinks in /var/www to enable the nominatim website.

sudo mkdir -m 755 /var/www/nominatim
sudo chown osm /var/www/nominatim
/home/osm/src/Nominatim/utils/setup.php --create-website /var/www/nominatim

I also revisited the local.php file for nominatim that I skipped over above and ended up adding the following file:

 <?php
   // Paths
   @define('CONST_Postgresql_Version', '9.1');
   // Website settings
   @define('CONST_Website_BaseURL', 'http://localhost/nominatim/');

And finally, I added the following section to /etc/apache2/sites-available/default just before the </virtualhost> closing statement at the bottom of the file.

 <Directory "/var/www/nominatim/">
    Options FollowSymLinks MultiViews
    AddType text/html   .php     
 </Directory>


  • Take snapshot: OSM server - 11 - Nominatim website set up ready for import
  • Take snapshot: OSM server - 12 - Fix broken nominatim import commandline

The import of the nominatim db faled because the user osm does not have permission to create a database, the following was done to grant superuser priveleges to fix this.

sudo su postgres
psql
ALTER ROLE osm WITH SUPERUSER;
  • Take snapshot: OSM server - 13 - Give osm user superuser postgresql priveleges

Trying import Nominatim import was successful. Doing a last couple tweaks the the install script and the Readme and shutting down to save a final snapshot before exporting the VM appliance.

  • Take snapshot: OSM Server - 14 - Shut down and ready for export