User:Ika-chan!/Fantasy maps with OSM software

From OpenStreetMap Wiki
Jump to: navigation, search

I am currently developing instructions for creating a standalone OpenStreetMap (OSM) server because my roleplay world is not compatible with the world of OpenGeoFiction. I am also hopeful that this page will encourage more people to set up OSM-based projects instead of playing around with the live map (OpenPokéMap, anyone?).

I know it is complicated, but someone has to do it.

Preparation

If you prefer to copy and paste commands, you can get to this page by typing this short link:

minoa.li/geso

Part 1: Prepare the server

Main article: User:Ika-chan!/Server/Part 1

In this part, you will find out how to install and prepare Ubuntu 18.04 LTS — the operating system that OpenStreetMap servers currently use:[1]

Download and install Ubuntu

  1. Visit the Ubuntu website and download the official disk image (ISO).
  2. For physical servers, create a bootable USB stick by following the official tutorials (macOSUbuntuWindows).
  3. Use the newly-created USB stick (or ISO image in the case of a virtual machine) to run a minimal install of Ubuntu on the target server.
  4. When you start Ubuntu for the first time, you will be asked whether to “help improve Ubuntu”: on that screen, you should select “No, don’t send system info”.

Ubuntu 18.04 LTS introduces a minimal install option, which removes all but one “consumer” packages, which will be resolved later.

Change the download server

The download server should be reviewed to reduce the time Ubuntu takes to download packages and updates:

  1. Open Software & Updates.
  2. Change the “Download from” drop-down menu to “Main server” or “Other” (the latter allows you have Ubuntu detect the fastest mirror).

Install Open VM Tools (virtual machines only)

If your server is a virtual machine, install Open VM Tools Desktop, which will make it easier to copy and paste text between the host and guest. Run the following command in Terminal, and restart the computer:

sudo apt update && sudo apt install open-vm-tools-desktop && reboot

Add SSH support

Ubuntu does not come with Secure Shell (SSH) support by default. If you plan to access your server from another computer, run the following command in Terminal to install OpenSSH, and restart the computer:

sudo apt install openssh-client openssh-server && reboot

You may also need to set a fixed IP address. To access the IPv4 address configuration:

  1. Open Settings;
  2. Click Network (for Ethernet connections) or Wi-Fi (for Wireless connections) on the sidebar;
  3. Click the cogwheel to the right of the active connection;
  4. Click the IPv4 tab.

Wi-Fi users may want to uncheck “Make available to other users” for improved security.

Remove telemetry and update Ubuntu

Run the following command in Terminal to remove the Popularity Contest package, and install all pending Ubuntu updates, and restart the computer::

sudo apt remove popularity-contest && sudo apt-get autoremove && sudo apt upgrade && reboot

Disable ‘hibernation’ on lid close (laptops only)

Ubuntu, by default, goes into ‘hibernation’ when you close the lid of a laptop-based server, preventing users from accessing the server. Run the following command in Terminal to correct this:

sudo sed -i -e 's/#HandleLidSwitch=suspend/HandleLidSwitch=ignore/g' /etc/systemd/logind.conf

Make more tweaks

You should also make the following tweaks for better user experience:

  1. Disable Notifications (open Settings, then click Notifications on the sidebar, then set everything there to 'Off');
  2. Disable searching for software (open Settings, click Search on the sidebar, then set Ubuntu Software to 'Off');
  3. Limit Usage & History (open Settings, then click Privacy on the sidebar, then click Usage & History);
  4. Disable Problem Reporting (open Settings, then click Privacy on the sidebar, then set Problem Reporting to 'Manual');
  5. Disable Autorun (open Settings, then click Devices on the sidebar, then click Removable Media on the sidebar, then enable Never prompt or start programs on media insertion);

Create base directories

Run following command in Terminal to create the directories that will host a majority of server and planet files, as well as the tiles (Terminal will automatically replace $USER with your user name):

sudo mkdir -p /mapserver/planet /mapserver/sources /mapserver/styles /mapserver/website/api /mapserver/website/tiles /mapserver/website/www && sudo chown -R $USER /mapserver

The above command will create this starter folder structure:

mapserver
├╸planet  # Planet dumps (empty by default)
├╸sources # Software compiled from GitHub source code
├╸styles  # Stylesheets
└╸website  
  ├╸api   # CGImap cache (empty by default)
  ├╸tiles # Tile cache (empty by default)
  └╸www   # Rails Port

Change the Rails Port installation mode

Finally, run the following command in Terminal to set the Rails Port setup mode to "production", and then restart the computer:[2]

sed -i '$a\\n# Always build the Rails Port in production mode.\nexport RAILS_ENV=production' ~/.bashrc && reboot

Part 2: Build the Website

In this part, you will find out how to install the website part of OpenStreetMap, including CGImap, Rails Port, and Phusion Passenger.

Install the website’s dependencies

Run the following commands in Terminal to install the following dependencies for the Rails Port, including Osmosis (without comments):[3][4][5]

sudo apt install apache2 apache2-dev build-essential git-core imagemagick libffi-dev libmagickwand-dev libpq-dev libruby2.3 libsasl2-dev libxml2-dev libxslt1-dev nodejs postgresql postgresql-contrib postgresql-server-dev-all ruby2.3 ruby2.3-dev && sudo gem2.3 install bundler # Rails Port (18 Feb 2018)
sudo apt install apt-transport-https ca-certificates dirmngr gnupg # Phusion Passenger (26 Apr 2018)
sudo apt install libboost-date-time-dev libboost-dev libboost-filesystem-dev libboost-locale-dev libboost-program-options-dev libboost-regex-dev libboost-system-dev libcrypto++-dev libfcgi-dev libmemcached-dev libpqxx-dev # CGImap (including Apache extension)
sudo apt install osmosis # Osmosis

Install Phusion Passenger

Run the following commands in Terminal to install Phusion Passenger:[4]

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger $(lsb_release -sc) main > /etc/apt/sources.list.d/passenger.list'
sudo apt update && sudo apt install libapache2-mod-passenger

You can run the following command in Terminal, and follow the on-screen instructions, to check if Phusion Passenger is running:

sudo /usr/bin/passenger-config validate-install
sudo /usr/sbin/passenger-memory-stats

Clone and configure the Rails Port

You can run the following command in Terminal to download the OpenStreetMap website from GitHub (you do not need the entire history),[3] create a writeable temporary directory,[6] and then remove the event banners (relevant only to the live map):

git clone --depth=1 https://github.com/openstreetmap/openstreetmap-website.git /mapserver/website/www && cd /mapserver/website/www
mkdir /mapserver/website/www/tmp && chmod -R 777 /mapserver/website/www/tmp
rm /mapserver/website/www/app/assets/images/banners/*.png && echo -e 'blank:\n  id: blank' > /mapserver/website/www/config/banners.yml

Next, use Bundler to install the Ruby gems (entering the password when requested), copy the application and database configuration files, and then open application.yml in gedit:

bundle install

cp /mapserver/website/www/config/example.application.yml /mapserver/website/www/config/application.yml
cp /mapserver/website/www/config/example.database.yml    /mapserver/website/www/config/database.yml

gedit /mapserver/website/www/config/application.yml

Change the value of server_url to the domain name that you will use, and then save the file.

Setup the website’s database

Type the following commands to set up your PostgreSQL account and build the databases for the Rails Port: Terminal will replace $USER with your user name, but replace [PASSWORD] with one of your own, and remember it:[3]

sudo -u postgres createuser -s $USER
psql -d osm -c "ALTER ROLE $USER WITH LOGIN PASSWORD '[PASSWORD]' VALID UNTIL 'infinity'"

bundle exec rake db:create
psql -d osm -c "CREATE EXTENSION btree_gist"
cd db/functions && make libpgosm.so && cd ../..
psql -d osm -c "CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 AS '`pwd`/db/functions/libpgosm', 'maptile_for_point' LANGUAGE C STRICT"
psql -d osm -c "CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 AS '`pwd`/db/functions/libpgosm', 'tile_for_point' LANGUAGE C STRICT"
psql -d osm -c "CREATE FUNCTION xid_to_int4(xid) RETURNS int4 AS '`pwd`/db/functions/libpgosm', 'xid_to_int4' LANGUAGE C STRICT"
bundle exec rake db:migrate

Set up the Rails Port session keys

Generate and copy a random secret key to the clipboard or text file (Ctrl+ Shift+C):

rails secret

Open secrets.yml in gedit: under production, set secret_key_base to the newly-generated secret key, and then save the file:

gedit /mapserver/website/www/config/secrets.yml

Download, build and configure CGImap

Next, Install and build CGImap from source:[5]

git clone --depth=1 https://github.com/zerebubuth/openstreetmap-cgimap.git /mapserver/sources/CGImap && cd /mapserver/sources/CGImap
./autogen.sh && ./configure && make

Create a blank file in gedit to configure CGImap:

gedit /mapserver/sources/CGImap/scripts/cgimap-wrapper

Paste the following syntax into the editing window, replacing [USER] and [PASSWORD] with your PostgreSQL credentials, then save the file:

#!/bin/bash

CGIMAP_HOST=localhost; export CGIMAP_HOST
CGIMAP_DBNAME=osm; export CGIMAP_DBNAME
CGIMAP_USERNAME=[USER]; export CGIMAP_USERNAME
CGIMAP_PASSWORD=[PASSWORD]; export CGIMAP_PASSWORD

CGIMAP_PIDFILE=cgimap.pid; export CGIMAP_PIDFILE
CGIMAP_LOGFILE=cgimap.log; export CGIMAP_LOGFILE

CGIMAP_MEMCACHE=localhost; export CGIMAP_MEMCACHE
CGIMAP_RATELIMIT=102400; export CGIMAP_RATELIMIT
CGIMAP_MAXDEBT=250; export CGIMAP_MAXDEBT

exec /mapserver/sources/CGImap/map

Finally, make cgimap-wrapper executable:

chmod +x /mapserver/sources/CGImap/scripts/cgimap-wrapper

Part 3: Install the Tile Server

Install the Tile Server dependencies

In Terminal, install the following dependencies, including osm2pgsql (without comments): [7]

sudo apt install bzip2 clang curl gdal-bin tar unzip wget # Essentials
sudo apt install libboost-all-dev libbz2-dev libcairo2 libcairo2-dev libfreetype6-dev libgdal-dev libharfbuzz-dev libicu-dev libjpeg-dev libpng-dev libproj-dev libtiff-dev mapnik-utils python-cairo python-cairo-dev python-dev # Mapnik
sudo apt install libboost-python-dev python-setuptools python3-setuptools # Python bindings for Mapnik
sudo apt install postgis postgresql-9.6-postgis-2.3 # PostGIS
sudo apt install osm2pgsql # osm2pgsql

Build and install Mapnik and its Python bindings

In Terminal, build and install Mapnik and its Python bindings, from source:

git clone --depth=1 -b v3.0.18 https://github.com/mapnik/mapnik.git /mapserver/sources/Mapnik && cd /mapserver/sources/Mapnik
git submodule update --depth=1 --init && ./configure && make
sudo make install

git clone --depth=1 -b v3.0.16 https://github.com/mapnik/python-mapnik.git /mapserver/sources/Mapnik-Python && cd /mapserver/sources/Mapnik-Python
sudo python setup.py develop
sudo python setup.py install

You can then test Mapnik:

python -c "import mapnik;print mapnik.__file__"

Install Mod_Tile (with Renderd)

In Terminal, build and install Mod_Tile (with Renderd), from source:

git clone --depth=1 https://github.com/openstreetmap/mod_tile.git /mapserver/sources/Mod-Tile && cd /mapserver/sources/Mod-Tile
./autogen.sh && ./configure && make
sudo make install && sudo make install-mod_tile
sudo ldconfig

Create the Tile Server database

Create the tile server database named gis: Terminal will replace $USER with your user name.

sudo -u postgres createdb -E UTF8 -O $USER gis
psql -d gis -c "CREATE EXTENSION postgis;"
psql -d gis -c "CREATE EXTENSION hstore;"
psql -d gis -c "ALTER TABLE geometry_columns OWNER TO $USER;"
psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO $USER;"

Part 4: Install a stylesheet

The following instructions are for the standard style that appears on the OpenStreetMap website.

Other stylesheets may have different dependencies and instructions, but it will be easier for you if they all reside in their own folders at /mapserver/styles.

Install the stylesheet dependencies

First, install the dependencies for the stylesheet in Terminal:[7]

sudo apt install fonts-hanazono fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted nodejs npm python-pip python-yaml ttf-unifont
sudo npm install -g carto
cd /usr/share/fonts/truetype/noto/
sudo wget https://github.com/googlei18n/noto-emoji/raw/master/fonts/NotoColorEmoji.ttf -O NotoColorEmoji.ttf && \
sudo wget https://github.com/googlei18n/noto-emoji/raw/master/fonts/NotoEmoji-Regular.ttf -O NotoEmoji-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Regular.ttf -O NotoSansArabicUI-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoNaskhArabicUI-Regular.ttf -O NotoNaskhArabicUI-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Bold.ttf -O NotoSansArabicUI-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoNaskhArabicUI-Bold.ttf -O NotoNaskhArabicUI-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansAdlam-Regular.ttf -O NotoSansAdlam-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansAdlamUnjoined-Regular.ttf -O NotoSansAdlamUnjoined-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansChakma-Regular.ttf -O NotoSansChakma-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansOsage-Regular.ttf -O NotoSansOsage-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansSinhalaUI-Regular.ttf -O NotoSansSinhalaUI-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Regular.ttf -O NotoSansArabicUI-Regular.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansCherokee-Bold.ttf -O NotoSansCherokee-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansSinhalaUI-Bold.ttf -O NotoSansSinhalaUI-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansSymbols-Bold.ttf -O NotoSansSymbols-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/hinted/NotoSansArabicUI-Bold.ttf -O NotoSansArabicUI-Bold.ttf && \
sudo wget https://github.com/googlei18n/noto-fonts/raw/master/unhinted/NotoSansSymbols2-Regular.ttf -O NotoSansSymbols2-Regular.ttf
sudo fc-cache -fv
sudo apt install fontconfig
fc-list
fc-list | grep Emoji

Configure and build the stylesheet

Download the latest version of OpenStreetMap-Carto from source:

git clone --depth=1 -b v4.10.0 https://github.com/gravitystorm/openstreetmap-carto.git /mapserver/styles/standard && cd /mapserver/styles/standard
scripts/get-shapefiles.py

At this stage, you can browse to /mapserver/styles/standard and edit the files to your requirements (such as changing the colours of the roads): remember to commit any changes when you are finished, by typing:

carto -a "3.0.0" project.mml > style.xml

Part 5: Connect the Tile Server and the Website to Apache

Configure Renderd

In Terminal, open the Renderd configuration file in gedit:

sudo gedit /usr/local/etc/renderd.conf

Change the configuration file to the following:

You can test if you configured Renderd correctly, ensuring that they return no errors:

ls -l /mapserver/styles/standard/style.xml
grep '^;xxx=\*\*' /usr/local/etc/renderd.conf

Next, open renderd.init:

gedit /mapserver/sources/Mod-Tile/debian/renderd.init

Set the following values (replacing [USER] with your user name) and save the file:

DAEMON=/usr/local/bin/$NAME
DAEMON_ARGS="-c /usr/local/etc/renderd.conf"
RUNASUSER=[USER]

Now, we copy renderd.init, and configure Renderd to start automatically.

sudo cp /mapserver/sources/Mod-Tile/debian/renderd.init /etc/init.d/renderd
sudo chmod a+x /etc/init.d/renderd
sudo systemctl daemon-reload
sudo systemctl start renderd
sudo systemctl enable renderd

Edit the list of available layers

Open the leaflet configuration file (leaflet.osm.js) in gedit:

gedit /mapserver/website/www/vendor/assets/leaflet/leaflet.osm.js

For example (replacing [HOST] with your domain name or IP address):

You can also comment out or change others layers in this file, according to your requirements. The example above is for applying a local version of the standard stylesheet.

Next, modify OpenStreetMap.js:

gedit /mapserver/website/www/vendor/assets/openlayers/OpenStreetMap.js

For example (replacing [HOST] with your domain name or IP address):

Finally, precompile the production assets:[2]

cd /mapserver/website/www && bundle exec rake assets:precompile

Connect the Website and Tile Server to Apache

First, create the system folders for Mod_Tile (Terminal will automatically replace $USER with your user name), then make Mod_Tile available to Apache:

sudo mkdir /var/lib/mod_tile && sudo chown $USER /var/lib/mod_tile
sudo mkdir /var/run/renderd && sudo chown $USER /var/run/renderd
sudo sh -c 'echo LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so > /etc/apache2/mods-available/mod_tile.load'

Now, create a virtual host file for our Rails Port:

sudo gedit /etc/apache2/sites-available/mapserver.conf

Paste the following syntax into the editing window, and then save the file:[5][8]

Next, create shortcuts so that Apache will run our new server and Mod_Tile on startup, and then remove the default configuration file (which has been replaced by mapserver.conf). Finally, restart Apache:

sudo ln -s /etc/apache2/mods-available/mod_tile.load /etc/apache2/mods-enabled/
sudo ln -s /etc/apache2/sites-available/mapserver.conf /etc/apache2/sites-enabled/
sudo rm /etc/apache2/sites-enabled/000-default.conf && sudo service apache2 restart

Reboot the computer, then test the website by visiting your domain name or IP address.

Part 6: Start mapping

Tuning PostgreSQL

The default settings for PostgreSQL are too conservative if you have modern hardware. See PostgreSQL § Tune the database for more details.

To open the configuration file, type:

sudo gedit $(ls /etc/postgresql/*/main/postgresql.conf)

When you finish tuning, type:

sudo /etc/init.d/postgresql reload

Create an account

First, register a new account via the new website that you set up, and save your login details to a password manager.

If you do not prefer to use an email server, you can use the Rails console (on Terminal) to activate your account, and then grant your new account administrator and moderator permissions:

cd /mapserver/website/www && bundle exec rails console

Next, type the following (replacing [USER] with your user name):

user = User.find_by_display_name("[USER]")
user.status = "active"
user.roles.create(:role => "administrator", :granter_id => user.id)
user.roles.create(:role => "moderator", :granter_id => user.id)
user.save!
quit

Create OAuth consumer keys for Potlatch 2, iD, and the Notes functionality: click here for instructions.

Install and configure JOSM

Never install the often-outdated version of JOSM from the Ubuntu Software: instead, install it from the official JOSM repository:

echo deb https://josm.openstreetmap.de/apt $(lsb_release -sc) universe && sudo tee /etc/apt/sources.list.d/josm.list > /dev/null
wget -q https://josm.openstreetmap.de/josm-apt.key -O- && sudo apt-key add -
sudo apt-get update && sudo apt-get install josm

Manually create and update the tiles

If you are the only user of the new server, you do not need to schedule tile updates. Once you make a few edits, run the following commands, after replacing [PASSWORD] with your PostgreSQL password:

osmosis --read-apidb database="osm" user=$USER password="[PASSWORD]" validateSchemaVersion="no" \
        --write-pbf file="/mapserver/planet/planet.osm.pbf"
osm2pgsql --create --database gis --slim --hstore --multi-geometry \
          --tag-transform-script /mapserver/styles/standard/openstreetmap-carto.lua \
          --style /mapserver/styles/standard/openstreetmap-carto.style "/mapserver/planet/planet.osm.pbf"
rm -rf /mapserver/website/tiles/standard/*

Updating your server

Updating Phusion Passenger is the same as updating Ubuntu with Terminal, that is:

sudo apt-get update && sudo apt-get upgrade

Test render

cd ~ && wget http://download.geofabrik.de/europe/great-britain/england/greater-london-latest.osm.pbf
osm2pgsql -s -c -G -k -S /mapserver/styles/standard/openstreetmap-carto.style --tag-transform-script /mapserver/styles/standard/openstreetmap-carto.lua -d gis ~/greater-london-latest.osm.pbf && rm -rf /mapserver/website/tiles/standard/*

And I’m done here.

References

  1. Fairhurst, Richard (27 April 2018). “Manually building a tile server (18.04 LTS)”. OpenStreetMap Foundation. Retrieved 27 April 2018. 
  2. 2.0 2.1 Homme; Avin (11 December 2014). “OpenStreetMap Website – Docker container”. Github. Retrieved 2 May 2018. 
  3. 3.0 3.1 3.2 Mvexel (8 December 2017). “Installation”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  4. 4.0 4.1 “Installing Passenger + Apache, on Ubuntu 17.10 (with APT)”. Phusion. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  5. 5.0 5.1 5.2 Zerebubuth (19 March 2017). “CGImap (Readme)”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018.  See also: Cgimap > Install.
  6. Hughes, Tom (9 January 2018). “Configuration (CONFIGURE.md) § Production Deployment”. GitHub. Retrieved 27 January 2018. 
  7. 7.0 7.1 Ircama (21 October 2017). “Installing an OpenStreetMap Tile Server on Ubuntu”. GitHub. Archived from the original on 1 January 2018. Retrieved 1 January 2018. 
  8. “Deploying a Ruby application”. Phusion. Archived from the original on 9 January 2018. Retrieved 9 January 2018.