User:Ika-chan!/Fantasy maps with OSM software

From OpenStreetMap Wiki
Jump to: navigation, search

I am currently developing instructions for creating a standalone OpenStreetMap server because my roleplay world is not compatible with the world of OpenGeoFiction. I am also hopeful that this page will encourage more people to set up OSM-like projects instead of playing around with the live map (OpenPokéMap, anyone?).

I know it is complicated, but someone has to do it.

Part 1: Prepare the server

Install Ubuntu

Install the latest version of Ubuntu Desktop, either 17.10 or 16.04 LTS (Long Term Service), by using the official ISOs.

Change the download server

You should change the download server to reduce the time it takes to download repositories and updates. Open Software & Updates, and change the “Download from” drop-down menu to “Main server” or “Other” (the latter allows you have Ubuntu detect the fastest mirror).

Install Open VM Tools (virtual machines only)

If you are installing the server on a virtual machine, such as VMware Workstation, you should install Open VM Tools Desktop to make it easier to copy and paste between the host and guest operating systems, in both directions. In Terminal, run the following command:

sudo apt update && sudo apt install open-vm-tools-desktop && reboot

Add SSH support

If you are building a server from a fresh install of Ubuntu, you need to install OpenSSH if you plan to access your server from another computer. In Terminal, run the following command:

sudo apt install openssh-client openssh-server && reboot

After restarting, you may also need to set a fixed IP address: to access the IPv4 address configuration, open Settings, then click Network on the sidebar, then click the cogwheel to the right of the active connection, and then click the IPv4 tab.

Change to Long Term Service (LTS) branch (Ubuntu 17.10 only)

Some people who did not like Unity may prefer to install Ubuntu 17.10, because it reintroduces the GNOME interface.

If you install 17.10, you can change to the Long Term Service branch: open Software & Updates, click the Updates tab, and change the notification setting for new Ubuntu versions to For long-term support versions.

When Ubuntu releases 18.04 LTS on 26 April 2018, you will go back to the Long Term Service branch, which by then will also use GNOME instead of Unity.

Disable Wayland (Ubuntu 17.10 only)

In Ubuntu 17.10, there is an issue where graphical applications (such as gedit) would not launch via root in Terminal. You can switch back to Xorg (which will be the default again with Ubuntu 18.04 LTS)[1] by running this command in Terminal:

sudo sed -i -e 's/#WaylandEnable/WaylandEnable/g' /etc/gdm3/custom.conf

Remove unnecessary applications and update Ubuntu

You should now remove the unnecessary 'consumer' applications, which takes up about 485 MB. After removing the consumer applications, install all pending Ubuntu updates, and then restart the computer:

sudo apt remove aisleriot cheese gnome-calendar gnome-mahjongg gnome-mines gnome-sudoku libreoffice-common rhythmbox-data shotwell-common simple-scan thunderbird totem-common transmission-common ubuntu-web-launchers && sudo apt-get autoremove && sudo apt upgrade && reboot

Disable 'hibernation' on lid close (laptops only)

When you close the lid of a laptop-based server, Ubuntu goes into suspend, which prevents users from accessing the server. To disable this, run this command in Terminal:

sudo sed -i -e 's/#HandleLidSwitch=suspend/HandleLidSwitch=ignore/g' /etc/systemd/logind.conf

Make more tweaks

You can also:

  • Disable Notifications (open Settings, then click Notifications on the sidebar, then set everything there to 'Off');
  • Disable searching for software (open Settings, click Search on the sidebar, then set Software to 'Off');
  • Limit Usage & History (open Settings, then click Privacy on the sidebar, then click Usage & History);
  • Disable Problem Reporting (open Settings, then click Privacy on the sidebar, then set Problem Reporting to 'Manual');
  • Disable Autorun (open Settings, then click Devices on the sidebar, then click Removable Media on the sidebar, then enable Never prompt or start programs on media insertion);

Create the base directories

Create the directories that will host a majority of server and planet files, along with the rendered tiles. Terminal will automatically replace $USER with your user name:

sudo mkdir -p /mapserver/planet /mapserver/sources /mapserver/styles /mapserver/website/api /mapserver/website/tiles /mapserver/website/www && sudo chown -R $USER /mapserver

The above command will create this starter folder structure:

mapserver
↳ planet    # Planet dumps (empty by default)
↳ sources   # Software compiled from GitHub source code
↳ styles    # Stylesheets
↳ website  
   ↳ api    # CGImap cache (empty by default)
   ↳ tiles  # Tile cache (empty by default)
   ↳ www    # Rails Port

Configure the Rails Port installation mode

Change the Rails Port installation mode to "production" by adding RAILS_ENV=production to ~/.bashrc.[2]

sed -i '$a\\n# Always run the Rails Port in production mode.\nexport RAILS_ENV=production' ~/.bashrc

Before moving on to Part 2, please restart the computer so that all changed settings come into effect.

Part 2: Build the Website (CGImap, Rails Port, and Phusion Passenger)

Install the Rails Port dependencies

Install the following dependencies for the Rails Port, including Osmosis (without comments):[3][4][5]

sudo apt install apache2 apache2-dev build-essential git-core imagemagick libmagickwand-dev libpq-dev libruby2.3 libsasl2-dev libxml2-dev libxslt1-dev nodejs postgresql postgresql-contrib postgresql-server-dev-all ruby2.3 ruby2.3-dev && sudo gem2.3 install bundler # Rails Port
sudo apt install apt-transport-https ca-certificates dirmngr gnupg # Phusion Passenger
sudo apt install libboost-date-time-dev libboost-dev libboost-filesystem-dev libboost-locale-dev libboost-program-options-dev libboost-regex-dev libboost-system-dev libcrypto++-dev libfcgi-dev libmemcached-dev libpqxx-dev # CGImap (including Apache extension)
sudo apt install osmosis # Osmosis

Once you install the dependencies, you can access the Apache test page by typing localhost on your browser, but the test page is only temporary, as we will point Apache to the Rails Port instead.

Install Phusion Passenger

Install Phusion Passenger by typing the following commands:[4]

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger $(lsb_release -sc) main > /etc/apt/sources.list.d/passenger.list'
sudo apt update && sudo apt install libapache2-mod-passenger

You can check if Phusion Passenger is running by running the following commands, and then follow the on-screen instructions:

sudo /usr/bin/passenger-config validate-install
sudo /usr/sbin/passenger-memory-stats

Clone and configure the Rails Port

Download the OpenStreetMap website from GitHub (you do not need the entire history),[3] create a writeable temporary directory,[6] and then remove the event banners (that are relevant only to the live map):

git clone --depth=1 https://github.com/openstreetmap/openstreetmap-website.git /mapserver/website/www && cd /mapserver/website/www
mkdir /mapserver/website/www/tmp && chmod -R 777 /mapserver/website/www/tmp
rm /mapserver/website/www/app/assets/images/banners/*.png && echo -e 'blank:\n  id: blank' > /mapserver/website/www/config/banners.yml

Next, use Bundler to install the Ruby gems (entering the password when requested), copy the application and database configuration files, and then open application.yml in gedit:

bundle install

cp /mapserver/website/www/config/example.application.yml /mapserver/website/www/config/application.yml
cp /mapserver/website/www/config/example.database.yml    /mapserver/website/www/config/database.yml

gedit /mapserver/website/www/config/application.yml

Change the value of server_url to the domain name that you will use, and then save the file.

Setup the PostgreSQL account and create the Website’s database

Type the following commands to set up your PostgreSQL account and build the databases for the Rails Port: Terminal will replace $USER with your user name, but replace [PASSWORD] with one of your own, and remember it:[3]

sudo -u postgres createuser -s $USER
psql -d osm -c "ALTER ROLE $USER WITH LOGIN PASSWORD '[PASSWORD]' VALID UNTIL 'infinity'"

bundle exec rake db:create
psql -d osm -c "CREATE EXTENSION btree_gist"
cd db/functions && make libpgosm.so && cd ../..
psql -d osm -c "CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 AS '`pwd`/db/functions/libpgosm', 'maptile_for_point' LANGUAGE C STRICT"
psql -d osm -c "CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 AS '`pwd`/db/functions/libpgosm', 'tile_for_point' LANGUAGE C STRICT"
psql -d osm -c "CREATE FUNCTION xid_to_int4(xid) RETURNS int4 AS '`pwd`/db/functions/libpgosm', 'xid_to_int4' LANGUAGE C STRICT"
bundle exec rake db:migrate

Set up the Rails Port session keys

Generate and copy a random secret key to the clipboard or text file (Ctrl+ Shift+C):

rails secret

Open secrets.yml in gedit: under production, set secret_key_base to the newly-generated secret key, and then save the file:

gedit /mapserver/website/www/config/secrets.yml

Download, build and configure CGImap

Next, Install and build CGImap from source:[5]

git clone --depth=1 https://github.com/zerebubuth/openstreetmap-cgimap.git /mapserver/sources/CGImap && cd /mapserver/sources/CGImap
./autogen.sh && ./configure && make

Create a blank file in gedit to configure CGImap:

gedit /mapserver/sources/CGImap/scripts/cgimap-wrapper

Paste the following syntax into the editing window, replacing [USER] and [PASSWORD] with your PostgreSQL credentials, then save the file:

#!/bin/bash

CGIMAP_HOST=localhost; export CGIMAP_HOST
CGIMAP_DBNAME=osm; export CGIMAP_DBNAME
CGIMAP_USERNAME=[USER]; export CGIMAP_USERNAME
CGIMAP_PASSWORD=[PASSWORD]; export CGIMAP_PASSWORD

CGIMAP_PIDFILE=cgimap.pid; export CGIMAP_PIDFILE
CGIMAP_LOGFILE=cgimap.log; export CGIMAP_LOGFILE

CGIMAP_MEMCACHE=localhost; export CGIMAP_MEMCACHE
CGIMAP_RATELIMIT=102400; export CGIMAP_RATELIMIT
CGIMAP_MAXDEBT=250; export CGIMAP_MAXDEBT

exec /mapserver/sources/CGImap/map

Finally, make cgimap-wrapper executable:

chmod +x /mapserver/sources/CGImap/scripts/cgimap-wrapper

Part 3: Install the Tile Server

Install the Tile Server dependencies

In Terminal, install the following dependencies, including osm2pgsql (without comments): [7]

sudo apt install bzip2 clang curl gdal-bin tar unzip wget # Essentials
sudo apt install libboost-all-dev libbz2-dev libcairo2 libcairo2-dev libfreetype6-dev libgdal-dev libharfbuzz-dev libicu-dev libjpeg-dev libpng-dev libproj-dev libtiff-dev mapnik-utils python-cairo python-cairo-dev python-dev # Mapnik
sudo apt install libboost-python-dev python-setuptools python3-setuptools # Python bindings for Mapnik
sudo apt install postgis postgresql-9.6-postgis-2.3 # PostGIS
sudo apt install osm2pgsql # osm2pgsql

Build and install Mapnik and its Python bindings

In Terminal, build and install Mapnik and its Python bindings, from source:

git clone --depth=1 -b v3.0.18 https://github.com/mapnik/mapnik.git /mapserver/sources/Mapnik && cd /mapserver/sources/Mapnik
git submodule update --depth=1 --init && ./configure && make
sudo make install

git clone --depth=1 -b v3.0.16 https://github.com/mapnik/python-mapnik.git /mapserver/sources/Mapnik-Python && cd /mapserver/sources/Mapnik-Python
sudo python setup.py develop
sudo python setup.py install

You can then test Mapnik:

python -c "import mapnik;print mapnik.__file__"

Install Mod_Tile (with Renderd)

In Terminal, build and install Mod_Tile (with Renderd), from source:

git clone --depth=1 https://github.com/openstreetmap/mod_tile.git /mapserver/sources/Mod-Tile && cd /mapserver/sources/Mod-Tile
./autogen.sh && ./configure && make
sudo make install && sudo make install-mod_tile
sudo ldconfig

Create the Tile Server database

Create the tile server database named gis: Terminal will replace $USER with your user name.

sudo -u postgres createdb -E UTF8 -O $USER gis
psql -d gis -c "CREATE EXTENSION postgis;"
psql -d gis -c "CREATE EXTENSION hstore;"
psql -d gis -c "ALTER TABLE geometry_columns OWNER TO $USER;"
psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO $USER;"

Part 4: Install a stylesheet

The following instructions are for the standard style that appears on the OpenStreetMap website.

Other stylesheets may have different dependencies and instructions, but it will be easier for you if they all reside in their own folders at /mapserver/styles.

Install the stylesheet dependencies

First, install the dependencies for the stylesheet in Terminal:[7]

sudo apt install fonts-hanazono fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted nodejs npm python-pip python-yaml ttf-unifont
sudo npm install -g carto

cd /usr/share/fonts/truetype/noto/
sudo wget https://github.com/googlei18n/noto-emoji/raw/master/fonts/NotoColorEmoji.ttf
sudo wget https://github.com/googlei18n/noto-emoji/raw/master/fonts/NotoEmoji-Regular.ttf
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Regular.ttf
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoNaskhArabicUI-Regular.ttf
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Bold.ttf
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoNaskhArabicUI-Bold.ttf

sudo fc-cache -fv
sudo apt install fontconfig
fc-list
fc-list | grep Emoji

Configure and build the stylesheet

Download the latest version of OpenStreetMap-Carto from source:

git clone --depth=1 -b v4.8.0 https://github.com/gravitystorm/openstreetmap-carto.git /mapserver/styles/standard && cd /mapserver/styles/standard
scripts/get-shapefiles.py

At this stage, you can browse to /mapserver/styles/standard and edit the files to your requirements (such as changing the colours of the roads): remember to commit any changes when you are finished, by typing:

carto -a "3.0.0" project.mml > style.xml

Part 5: Connect the Tile Server and the Website to Apache

Configure Renderd

In Terminal, open the Renderd configuration file in gedit:

sudo gedit /usr/local/etc/renderd.conf

Change the configuration file to the following:

You can test if you configured Renderd correctly, ensuring that they return no errors:

ls -l /mapserver/styles/standard/style.xml
grep '^;xxx=\*\*' /usr/local/etc/renderd.conf

Next, open renderd.init:

gedit /mapserver/sources/Mod-Tile/debian/renderd.init

Set the following values (replacing [USER] with your user name) and save the file:

DAEMON=/usr/local/bin/$NAME
DAEMON_ARGS="-c /usr/local/etc/renderd.conf"
RUNASUSER=[USER]

Now, we copy renderd.init, and configure Renderd to start automatically.

sudo cp /mapserver/sources/Mod-Tile/debian/renderd.init /etc/init.d/renderd
sudo chmod a+x /etc/init.d/renderd
sudo systemctl daemon-reload
sudo systemctl start renderd
sudo systemctl enable renderd

Edit the list of available layers

Open the leaflet configuration file (leaflet.osm.js) in gedit:

gedit /mapserver/website/www/vendor/assets/leaflet/leaflet.osm.js

For example (replacing [HOST] with your domain name or IP address):

You can also comment out or change others layers in this file, according to your requirements. The example above is for applying a local version of the standard stylesheet.

Next, modify OpenStreetMap.js:

gedit /mapserver/website/www/vendor/assets/openlayers/OpenStreetMap.js

For example (replacing [HOST] with your domain name or IP address):

Finally, precompile the production assets:[2]

cd /mapserver/website/www && bundle exec rake assets:precompile

Connect the Website and Tile Server to Apache

First, create the system folders for Mod_Tile (Terminal will automatically replace $USER with your user name), then make Mod_Tile available to Apache:

sudo mkdir /var/lib/mod_tile && sudo chown $USER /var/lib/mod_tile
sudo mkdir /var/run/renderd && sudo chown $USER /var/run/renderd
sudo sh -c 'echo LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so > /etc/apache2/mods-available/mod_tile.load'

Now, create a virtual host file for our Rails Port:

sudo gedit /etc/apache2/sites-available/mapserver.conf

Paste the following syntax into the editing window, and then save the file:[5][8]

Next, create shortcuts so that Apache will run our new server and Mod_Tile on startup, and then remove the default configuration file (which has been replaced by mapserver.conf). Finally, restart Apache:

sudo ln -s /etc/apache2/mods-available/mod_tile.load /etc/apache2/mods-enabled/
sudo ln -s /etc/apache2/sites-available/mapserver.conf /etc/apache2/sites-enabled/
sudo rm /etc/apache2/sites-enabled/000-default.conf && sudo service apache2 restart

Reboot the computer, then test the website by visiting your domain name or IP address.

Part 6: Start mapping

Tuning PostgreSQL

The default settings for PostgreSQL are too conservative if you have modern hardware. See PostgreSQL § Tune the database for more details.

To open the configuration file, type:

sudo gedit $(ls /etc/postgresql/*/main/postgresql.conf)

When you finish tuning, type:

sudo /etc/init.d/postgresql reload

Create an account

First, register a new account via the new website that you set up, and save your login details to a password manager.

If you do not prefer to use an email server, you can use the Rails console (on Terminal) to activate your account, and then grant your new account administrator and moderator permissions:

cd /mapserver/website/www && bundle exec rails console

Next, type the following (replacing [USER] with your user name):

user = User.find_by_display_name("[USER]")
user.status = "active"
user.roles.create(:role => "administrator", :granter_id => user.id)
user.roles.create(:role => "moderator", :granter_id => user.id)
user.save!
quit

Create OAuth consumer keys for Potlatch 2, iD, and the Notes functionality: click here for instructions.

Install and configure JOSM

Never install the often-outdated version of JOSM from the Ubuntu Software: instead, install it from the official JOSM repository:

echo deb https://josm.openstreetmap.de/apt $(lsb_release -sc) universe && sudo tee /etc/apt/sources.list.d/josm.list > /dev/null
wget -q https://josm.openstreetmap.de/josm-apt.key -O- && sudo apt-key add -
sudo apt-get update && sudo apt-get install josm

Manually create and update the tiles

If you are the only user of the new server, you do not need to schedule tile updates. Once you make a few edits, run the following commands, after replacing [PASSWORD] with your PostgreSQL password:

osmosis --read-apidb database="osm" user=$USER password="[PASSWORD]" validateSchemaVersion="no" \
        --write-pbf file="/mapserver/planet/planet.osm.pbf"
osm2pgsql --create --database gis --slim --hstore --multi-geometry \
          --tag-transform-script /mapserver/styles/standard/openstreetmap-carto.lua \
          --style /mapserver/styles/standard/openstreetmap-carto.style "/mapserver/planet/planet.osm.pbf"
rm -rf /mapserver/website/tiles/standard/*

Updating your server

Updating Phusion Passenger is the same as updating Ubuntu with Terminal, that is:

sudo apt-get update && sudo apt-get upgrade

Test render

cd ~ && wget http://download.geofabrik.de/europe/great-britain/england/greater-london-latest.osm.pbf
osm2pgsql -s -c -G -k -S /mapserver/styles/standard/openstreetmap-carto.style --tag-transform-script /mapserver/styles/standard/openstreetmap-carto.lua -d gis ~/greater-london-latest.osm.pbf && rm -rf /mapserver/website/tiles/standard/*

And I’m done here.

References

  1. Sneddon, Joey (25 January 2018). “Ubuntu 18.04 LTS is Switching back to Xorg”. OMG! Ubuntu!. Retrieved 26 January 2018. 
  2. 2.0 2.1 Docker file for the OpenStreetMap Rails Port on GitHub, accessed 28 December 2017.
  3. 3.0 3.1 3.2 Mvexel (8 December 2017). “Installation”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  4. 4.0 4.1 “Installing Passenger + Apache, on Ubuntu 17.10 (with APT)”. Phusion. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  5. 5.0 5.1 5.2 Zerebubuth (19 March 2017). “CGImap (Readme)”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018.  See also: Cgimap > Install.
  6. Hughes, Tom (9 January 2018). “Configuration (CONFIGURE.md) § Production Deployment”. GitHub. Retrieved 27 January 2018. 
  7. 7.0 7.1 Ircama (21 October 2017). “Installing an OpenStreetMap Tile Server on Ubuntu”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  8. “Deploying a Ruby application”. Phusion. Archived from the original on 9 January 2018. Retrieved 9 January 2018.