OpenFireMap/Installation

From OpenStreetMap Wiki
Jump to navigation Jump to search

This page describes how to install the software you need for creating a map like openfiremap.org (see also OpenFireMap Wiki page).

Prerequisites

Hardware

A weak CPU or a virtual Internet server, will suffice if you limit the geographical region (e.g. France, Germany, Australia). For rendering a larger area, at least 8 GB RAM are recommended.

Operating System

We assume that you use Ubuntu >=20.04 as operating system. All of the following steps have been tested with version 20.04.

Prepare your System

This chapter describes the steps which are to be done once. After having run through this work, you will only need to update the OSM data for your map from time to time.

If you are not using a virtual machine solely for this purpose it is recommended to create a separate user. In this example, we decide to go on with the user root since the map is going to be the only task running on this virtual server.

Software Packages

At first, install all packages necessary to run your own web server. Additionally, you will need some other packages. Ensure the installation of all necessary packages by executing these commands:

apt update
apt upgrade
apt install nano unzip gcc zlib1g-dev
apt install postgresql-12-postgis-3 postgresql-contrib
apt install osm2pgsql
apt install python3-mapnik mapnik-utils
apt install python2.7-minimal
apt install apache2 php php-pgsql libapache2-mod-php libjs-openlayers

At this point, it does not hurt to reboot the system to ensure all services have been started in proper order.

PostgreSQL Database

The GIS database must be initialized. Please follow these steps:

sudo -u postgres -i -H
createuser -SdR hyuser
createdb -E UTF8 -O hyuser hygis
psql -d hygis -f /usr/share/postgresql/12/contrib/postgis-3.0/postgis.sql
psql -d hygis -f /usr/share/postgresql/12/contrib/postgis-3.0/spatial_ref_sys.sql
psql hygis -c "ALTER TABLE geography_columns OWNER TO hyuser"
psql hygis -c "ALTER TABLE geometry_columns OWNER TO hyuser"
psql hygis -c "ALTER TABLE spatial_ref_sys OWNER TO hyuser"
exit

To enable easy database login by user hyuser we need to edit one of the database configuration files. In case you are running Ubuntu with a graphical interface, you could use a more comfortable editor, e.g. gedit instead of nano.

nano /etc/postgresql/12/main/pg_hba.conf

Near to the bottom of the file you will find these lines:

local   all         all                               peer
host    all         all         127.0.0.1/32          md5
host    all         all         ::1/128               md5

In these lines change the word "peer" and the two words "md5" each to "trust" and close the editor (for nano: Ctrl-O, Enter, Ctrl-X). Now reload the database configuration:

service postgresql reload

For a short test, login to the database by using the previously created database user hyuser:

psql hygis hyuser

Type \d to see a list with the two tables which we have created some steps above (geography_columns, geometry_columns, and spatial_ref_sys). Then logout with \q

If you encounter any problems, you may find a solution here: PostGIS.

Choose a Project Directory

Our example directory for downloading OSM data and generating the data base contents will be the subdirectory hy of the user root, "/root/hy", which can be abbreviated as "~/hy" if being logged in with this user. Let's create this directory and a directory for the rendering tool Mapnik:

mkdir /root/hy
mkdir /root/hy/mapnik_hy

If you want to use a different directory, please create it and grant all access rights to your user – in case it is not root. You also will have to replace all cd /root/hy commands in the following examples with cd and the full path to your alternate directory.

Tools osmconvert and osmfilter

These tools are mainly used to accelerate the PostgreSQL import process. If we filter out all information we need and drop everything else, osm2pgsql will run faster. On top of this, database queries will be faster, so the rendering process will be accelerated as well. You can install these programs from Ubuntu repository. They are supplied by package osmctools. However, it is recommended to download and compile the source code because the binary may be out of date. To do this – downloading and compiling – from the command line, enter these commands:

cd /root/hy
wget -O - http://m.m.i24.cc/osmconvert.c |cc -x c - -lz -O3 -o osmconvert
wget -O - http://m.m.i24.cc/osmfilter.c |cc -x c - -O3 -o osmfilter

For further details on both tools refer to osmconvert and osmfilter.

Get Icons

Mapnik renderer will need certain icons to represent the objects we want to display on the map. You can create these icons by yourself or download sets of icons from various Internet sources. In this example we take the icon set from openfiremap.org:

cd /root/hy/mapnik_hy
wget -r -l 1 -nd -A \*.png -P symbols http://openfiremap.de/f/symbols_hy/

Mapnik Renderer Initialization

For the rendering tool Mapnik, some initializations are to be done. Afterwards, a Mapnik style file needs to be created in folder /root/hy/mapnik_hy. You can create the file mapnik_hy.xml on your own or use one of the examples from the Internet. You also may download the OpenFireMap style file from here: openfiremap.de/f/mapnik_hy.xml

cd /root/hy/mapnik_hy/inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/datasource-settings.xml.inc.template
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/entities.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/fontset-settings.xml.inc.template
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-addressing.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-admin.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-aerialways.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-points.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-stations.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-symbols.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-buildings.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-citywall.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-ferry-routes.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-landcover.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-placenames.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-power.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-shapefiles.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-water.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-water_features.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layers.xml.inc
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/settings.xml.inc.template
cd /root/hy/mapnik_hy
mkdir /root/hy/mapnik_hy/world_boundaries
wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/generate_xml.py
/usr/bin/python2.7 ./generate_xml.py --host localhost --user hyuser --dbname hygis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --port '' --password ''
wget -N http://openfiremap.de/f/mapnik_hy.xml

Create the Map

This chapter will show you how to create or update the tiles for your map. Tiles are small bitmaps which will be assembled to a whole map by the map browser. We assume that all tasks of the previous chapter have been completed successfully.

The process of map creation involves two steps – filling the database and rendering the map tiles.

Fill the Database

All information we need to create the map tiles must be written into our database. To do this, we need to develop a script which performs several tasks, step by step. We will do this step by step, entering every single command at the command line terminal. That makes it much easier to find errors.

Get the OSM Data File

It is not recommended to use the file of the entire planet. Please choose the file of an area you are interested in, in this example a part of Germany. The first task will be to download this file and to store it using .o5m file format.

cd /root/hy
wget -N http://download.geofabrik.de/europe/germany/bayern/mittelfranken-latest.osm.pbf -O - |./osmconvert - -o=a_hy.o5m

Filter the OSM Data File

We need to do a hierarchical filtering because there will be nodes in ways, ways in relations and relations in other relations. For performance reasons a pre-filtered file will be used for interrelational filtering. In this example we decided to filter public-transport specific information because we want to create a public transport map. The filtering criteria need to be specified in a file. This can be done via commandline like this:

cd /root/hy
cat <<eof >toolchain_hy.filter
--keep=
emergency=fire_hydrant =water_tank =suction_point =fire_water_pond
amenity=fire_hydrant =fire_station

--keep-tags=all
emergency=
amenity=
name=
ref=
fire_hydrant:type=
fire_hydrant:diameter=
fire_hydrant:pressure=
fire_hydrant:count=
fire_hydrant:position=
water_tank:volume=
type=
eof

Then the OSM raw data can be filtered:

./osmfilter a_hy.o5m --parameter-file=toolchain_hy.filter -o=f_hy.o5m

Some seconds or minutes later – depending on the size of the chosen area – we will get the file gis_hy.o5m, containing only that information we need.

Note: We are using .o5m file format, because this will accelerate the filter process. For further information see osmconvert and .o5m.

Transfer the Data to Postgres Database

Now we can transfer the OSM data from the file f_hy.o5m into the Postgres database. The program osm2pgsql will do this job. To fit the needs of our specialised map, we need to create our own osm2pgsql style file first. This can be done with an editor or by executing these commands:

cd /root/hy
cat <<eof >osm2pgsql_hy.style
node,way emergency text polygon
node,way amenity text polygon
node,way name text linear
node,way ref text linear
node,way fire_hydrant:type text linear
node,way fire_hydrant:diameter text linear
node,way fire_hydrant:pressure text linear
node,way fire_hydrant:count text linear
node,way fire_hydrant:position text linear
node,way water_tank:volume text linear
eof

Now you can start the data import:

osm2pgsql -s -C 1500 -d hygis -U hyuser -S osm2pgsql_hy.style -c f_hy.o5m -e 18-18 -o dirty_tiles_hy

Because the .o5m file had been filtered in the previous step, this transfer should take only a few minutes. If you have more main memory to spend, you can increase the number of available MB increasing the parameter -C 1500.

If you encounter any problems, you may find a solution here: osm2pgsql.

Getting a List of all Dirty Tiles

Map tiles which are not up-to-date are referred to as dirty tiles. During map data import the program osm2pgsql generates a list of all dirty tiles. Unfortunately the program does not provide us with the names of all affected tiles because this list is stripped of all redundant information. If there is, for example, the tile 3/0/1 (zoom, x, y) in the list, the also affected tiles 2/0/0 and 1/0/0 will be omitted. Furthermore, a list entry 3/0/1 means that all four tiles of the next zoom level have also been omitted although affected: 4/0/2, 4/0/3, 4/1/2, and 4/1/3.

This is an effective way to reduce the dirty-tiles list length but we still need a list of all effected tiles. For this reason, the dirty tiles list must be expanded accordingly. This can be done with a small C program which needs to be added to our toolchain:

cd /root/hy
cat <<eof |cc -x c - -O3 -o dtexpand
// dtexpand 2017-03-10 18:00
//
// expands the dirty-tiles list as provided by osm2pgsql
// example: dtexpand 4 17 <dirty_tiles >dirty_tiles_ex
// (c) 2017 Markus Weber, Nuernberg, License LGPLv3
#include <ctype.h>
#include <inttypes.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc,char** argv) {
  int fieldedge[20];
  uint8_t* field[20],*fieldend[20];
  int zoom_min,zoom_max,zoom_sub;
  int z,x,y,zz,xx,yy,xd,yd,q; uint64_t u;
  uint8_t b; uint8_t* bp,*bpa,*bpe;
  char line[40]; char* sp;
  typedef struct {int zs,tx_min,tx_max,ty_min,ty_max;} sub_t;
  sub_t* sub,*subp; int subn;

  if(argc<3 || (argc-3)%5!=0) { fprintf(stderr,
    "Usage: dtexpand ZOOM_MIN ZOOM_MAX <DT_IN >DT_OUT\n"
    "Optional value sets each: ZOOM_SUB TX_MIN TX_MAX TY_MIN TY_MAX\n"
    "         ZOOM_SUB: max zoom level for blindly adding subtiles\n"
    "         TX...TY: concerning tile range at ZOOM_MAX level\n"
      ); return 1; }
  zoom_min= atoi(argv[1]); zoom_max= atoi(argv[2]);
  subn= (argc-3)/5;
  sub= (sub_t*)malloc(sizeof(sub_t)*(subn+1));
  argv+= 2; subp= sub;
  for(z= 0;z<subn;z++) {  // for each subtile value set
    subp->zs= atoi(*++argv);
    subp->tx_min= atoi(*++argv); subp->tx_max= atoi(*++argv);
    subp->ty_min= atoi(*++argv); subp->ty_max= atoi(*++argv);
    if(zoom_min<0 || zoom_min>19 || zoom_max<zoom_min || zoom_max>19 ||
        subp->zs<=zoom_max || subp->zs>19) { fprintf(stderr,
        "Unsupported zoom value(s).\n"); return 2; }
    subp++; }  // for each subtile value set
  subp->zs= 0;
  for(z= zoom_max-1;z>=zoom_min;z--) {  // for each zoom level
    fieldedge[z]= u= UINT32_C(1)<<z; u= u*u+7>>3;
    if((field[z]= (uint8_t*) malloc(u))==NULL) { fprintf(stderr,
      "Not enough main memory.\n"); return 3; }
    memset(field[z],0,u);
    fieldend[z]= field[z]+u;
    }  // for each zoom level
  while(fgets(line,sizeof(line),stdin)!=NULL) {  // for each input line
    sp= line;
    z= 0; while(isdigit(*sp)) z= z*10+*sp++-'0'; if(*sp=='/') sp++;
    x= 0; while(isdigit(*sp)) x= x*10+*sp++-'0'; if(*sp=='/') sp++;
    y= 0; while(isdigit(*sp)) y= y*10+*sp++-'0';
    if(z<zoom_min || z>zoom_max)
  continue;
    zz= z-1; xx= x>>1; yy= y>>1;
    while(zz>=zoom_min) {  // for all lower zoom levels
      u= xx; u*= fieldedge[zz]; u+= yy; bp= &field[zz][u/8];
      if(bp<fieldend[zz]) *bp|= 1<<(u&7);  // (preventing overflow)
      zz--; xx>>= 1; yy>>= 1;
      }  // for all lower zoom levels
    q= 1;
    do {  // for this and all higher zoom levels
      for(xd= 0;xd<q;xd++) for(yd= 0;yd<q;yd++) {  // all these tiles
        xx= x+xd; yy= y+yd;
        if(z<zoom_max) {
          u= xx; u*= fieldedge[z]; u+= yy; bp= &field[z][u/8];
          if(bp<fieldend[z]) *bp|= 1<<(u&7);  // (preventing overflow)
          }
        else {  // at zoom_max level
          printf("%i/%i/%i\n",z,xx,yy);
          // determine relevant subtile zoom level 
          zoom_sub= zoom_max;
          subp= sub;
          while(subp->zs!=0) {  // for each subtile value set
            if(subp->zs>zoom_sub &&
                xx>=subp->tx_min && xx<=subp->tx_max &&
                yy>=subp->ty_min && yy<=subp->ty_max) zoom_sub= subp->zs;
            subp++; }  // for each subtile value set
          // write subtiles
          int xxd,yyd,qq;
          zz= z; qq= 1;
          while(zz<zoom_sub) {  // for each subtile level
            zz++; xx<<= 1; yy<<= 1; qq<<= 1;
            for(xxd= 0;xxd<qq;xxd++) for(yyd= 0;yyd<qq;yyd++)
              printf("%i/%i/%i\n",zz,xx+xxd,yy+yyd);
            }  // for each subtile level
          }  // at zoom_max level
        }  // all these tiles
      z++; x<<= 1; y<<= 1; q<<= 1;
      } while(z<=zoom_max);  // for this and all higher zoom levels
    }  // for each input line
  for(z= zoom_max-1;z>=zoom_min;z--) {  // for each zoom level
    bp= bpa= field[z]; bpe= fieldend[z];
    do {  // for each byte in bitfield
      b= *bp;
      if(b) {  // at least one bit in byte is set
        int be= fieldedge[z];
        for(int bi=0;bi<8;bi++) {  // for each bit in byte
          if(b&1) {  // bit is set
            u= (bp-bpa)*8+bi; x= u/be; y= u&be-1;
            printf("%i/%i/%i\n",z,x,y);
            }  // bit is set
          b>>= 1;
          }  // for each bit in byte
        }  // at least one bit in byte is set
      } while(++bp<bpe);  // for each byte in bitfield
    free(field[z]);
    }  // for each zoom level
  free(sub); return 0;
  }  // main()
eof

Besides its main function this program can also be used to add subtile names for certain regions to the dirty-tiles list. If you want to create a global map up to zoom level 17, for example, and having a small region rendered up to zoom level 18, you may later add this subtile zoom level as well as the region's level-17 tile range to the command line:

./dtexpand 4 17 18 69000 70000 44000 45000

If there are two or more of such regions, just enter all of their 5-value sets. For example:

./dtexpand 4 17 19 80000 81000 48000 49000 18 69000 70000 44000 45000

At this point we do not need to care about expanding the list of dirty tiles since that will be dealt with by the toolchain script which is going to be introduced some sections below.

Rendering Test

At first, we should test if the renderer works as expected. Thereby, the nik4 program will help us. Let's pick an area for which we have already imported GIS data – in this example: Nürnberg, Germany – generate a test image and examine it with the viewer (here Eye of Gnome).

cd /root/hy/mapnik_hy
wget -N https://raw.githubusercontent.com/Zverik/Nik4/master/nik4.py
/usr/bin/python3 nik4.py -c 11 49.5 -z 12 -d 800 400 -f png256 mapnik_hy.xml image.png
eog image.png &

Render the Tiles

If the test-run has been successful, the next step should be to generate map tiles. For this purpose, we use the scripts render_hy.py and toolchain_hy.sh which must be created first:

cd /root/hy
nano render_hy.py

Enter the following lines:

#!/usr/bin/python
# render.py 2020-12-25 15:50
# reads dirty-tiles file and renders each tile of this list;
# afterwards, deletes the rendered dirty-tiles file;
# call: LAY=hy DTF=dirty_tiles_hy render.py
# parallel call: LAY=hy DTF=dirty_tiles_hy PID=123 render.py
# adaptations: Markus Weber, Nuernberg
from math import pi,cos,sin,log,exp,atan
from subprocess import call
import sys, os
import mapnik
import threading
import shutil

DEG_TO_RAD = pi/180
RAD_TO_DEG = 180/pi

def minmax (a,b,c):
    a = max(a,b)
    a = min(a,c)
    return a

class GoogleProjection:
    def __init__(self,levels=18):
        self.Bc = []
        self.Cc = []
        self.zc = []
        self.Ac = []
        c = 256
        for d in range(0,levels):
            e = c/2;
            self.Bc.append(c/360.0)
            self.Cc.append(c/(2 * pi))
            self.zc.append((e,e))
            self.Ac.append(c)
            c *= 2

    def fromLLtoPixel(self,ll,zoom):
         d = self.zc[zoom]
         e = round(d[0] + ll[0] * self.Bc[zoom])
         f = minmax(sin(DEG_TO_RAD * ll[1]),-0.9999,0.9999)
         g = round(d[1] + 0.5*log((1+f)/(1-f))*-self.Cc[zoom])
         return (e,g)

    def fromPixelToLL(self,px,zoom):
         e = self.zc[zoom]
         f = (px[0] - e[0])/self.Bc[zoom]
         g = (px[1] - e[1])/-self.Cc[zoom]
         h = RAD_TO_DEG * ( 2 * atan(exp(g)) - 0.5 * pi)
         return (f,h)

if __name__ == "__main__":

  # read environment variables
  lay = os.environ['LAY']
  dt_file_name = os.environ['DTF']
  try:
    temp_file = "/dev/shm/tile" + os.environ['PID']
  except:
    temp_file = "/dev/shm/tile"

  # do some global initialization
  os.chdir("mapnik_"+lay+"/")
  #sys.path.append("mapnik")
  print("render.py " + dt_file_name + ": started.")
  mapfile = "mapnik_" + lay + ".xml"
  ###############################################################
  tile_dir = "/var/www/html/" + lay + "tiles_new/"
  ###############################################################
  max_zoom = 18
  tile_count = 0
  empty_tile_count = 0

  # create tile directory and all possible zoom directories
  if not os.path.isdir(tile_dir):
    os.mkdir(tile_dir)
  for z in range(0, max_zoom+1):
    d = tile_dir + "%s" % z + "/"
    if not os.path.isdir(d):
      os.mkdir(d)

  # open the dirty-tiles file
  try:
    dt_file = open("../" + dt_file_name, "r")
  except:
    dt_file = None
    print("render.py: dt file not found: "+dt_file_name)
  if (dt_file != None):

    # do some Mapnik initialization
    mm = mapnik.Map(256, 256)
    mapnik.load_map(mm, mapfile, True)
    # Obtain <Map> projection
    prj = mapnik.Projection(mm.srs)
    # Projects between tile pixel co-ordinates and LatLong (EPSG:4326)
    tileproj = GoogleProjection(max_zoom + 1)

    # process the dirty-tiles file
    while True:

      # read a line of the dirty-tiles file
      line = dt_file.readline()
      if (line == ""):
        break
      line_part = line.strip().split('/')

      # create x coordinate's directory - if necessary
      d = tile_dir + line_part[0] + "/" + line_part[1] + "/"
      if not os.path.isdir(d):
        os.mkdir(d)

      # get parameters of the line
      z = int(line_part[0])
      x = int(line_part[1])
      y = int(line_part[2])
      tile_name= tile_dir + line[:-1] + ".png"

      # render this tile - start

      # Calculate pixel positions of bottom-left & top-right
      p0 = (x * 256, (y + 1) * 256)
      p1 = ((x + 1) * 256, y * 256)

      # Convert to LatLong (EPSG:4326)
      l0 = tileproj.fromPixelToLL(p0, z);
      l1 = tileproj.fromPixelToLL(p1, z);

      # Convert to map projection (e.g. mercator co-ords EPSG:900913)
      c0 = prj.forward(mapnik.Coord(l0[0], l0[1]))
      c1 = prj.forward(mapnik.Coord(l1[0], l1[1]))

      # Bounding box for the tile
      if hasattr(mapnik,'mapnik_version') and mapnik.mapnik_version() >= 800:
        bbox = mapnik.Box2d(c0.x,c0.y, c1.x,c1.y)
      else:
      	bbox = mapnik.Envelope(c0.x,c0.y, c1.x,c1.y)
      render_size = 256
      mm.resize(render_size, render_size)
      mm.zoom_to_box(bbox)
      mm.buffer_size = 256
      # (buffer size was 128)

      # render image with default Agg renderer
      im = mapnik.Image(render_size, render_size)
      mapnik.render(mm, im)
      im.save(temp_file, 'png')  # changed from 'png256' to 'png', 2012-03-21
      tile_count = tile_count + 1

      # render this tile - end
      # copy this tile to tile tree
      l= os.stat(temp_file)[6]
      if l>116:
        shutil.copyfile(temp_file,tile_name)
      else:
        try:
          os.unlink(tile_name)
        except:
          l= l  # (file did not exist)
        empty_tile_count = empty_tile_count + 1

    # close and delete the dirty-tiles file
    dt_file.close()
    try:
      os.unlink("../" + dt_file_name)
    except:
      l= l  # (file did not exist)

  # print some statistics
  print("render.py " + dt_file_name + ":", tile_count, "tiles (" + "%s" % empty_tile_count + " empty).")

Now open the toolchain file:

nano toolchain_hy.sh

Enter the following lines:

#!/bin/bash
# toolchain.sh
# This script cares about creating a thematical map.
# (c) Markus Weber, 2020-12-26 17:00
# License: AGPL V3

LAY="hy"
PLANETURL=http://download.geofabrik.de/europe/germany/bayern/mittelfranken-latest.osm.pbf
PLANETMINSIZE=60000000  # minimum size of OSM data file in .o5m format
TILEMINSIZE=130000 # minimum size of tile directory in blocks
#BORDERS="-B=bayern.poly"
MAXPROCESS=2  # maximum number of concurrent processes for rendering
OSM2PGSQLPARAM="-s -C 1500 -d "$LAY"gis -U "$LAY"user -S osm2pgsql_"$LAY".style"
  # main parameters to be passed to osm2pgsql

# do some initializations
PLANETFILE="a_"$LAY".o5m"
LOG="tc_"$LAY".log"
DTLOG="dt_"$LAY".log"
RUNNING="toolchain_running_"$LAY".txt"
ENDED="toolchain_ended_"$LAY".txt"
FILTER="f_"$LAY".o5m"
DTDIR="dt_"$LAY"/"
DTFILE="dirty_tiles_"$LAY
PROCN=1000  # rendering-process number (range 1000..1999)
mkdir $DTDIR 2>/dev/null

# enter working directory and start logging
if [ "0"$LAY = "0" ]; then exit 1; fi
cd /root/hy
echo >>$LOG
echo $(date)"  toolchain started." >>$LOG
mkdir "dt_"$LAY"/" 2>/dev/null

# publish that this script is now running
rm -f $ENDED
echo -e "The toolchain script has been started at "$(date)"\n"\
"and is currently running. To terminate it, delete this file and\n"\
"wait until the file \""$ENDED"\" has been created.\n"\
"This may take some minutes." \
>$RUNNING

# clean up previous Mapnik processes
killall "mapnik_"$LAY".py" 2>/dev/null
while [ $(ls $DTDIR""at* 2>/dev/null |wc -l) -gt 0 ]; do
    # there is at least one incompleted rendering process
  AT=$(ls -1 $DTDIR""at* 2>/dev/null|head -1)  # tile list
  echo $(date)"  Cleaning up incomplete rendering: "$AT >>$LOG
  DT=${AT:0:6}d${AT:7:99}  # new name of the tile list
  mv $AT $DT  # rename this tile list file;
    # now the tile is marked as 'to be rendered'
  done

# delete old tile files
echo $(date)"  Deleting old tile files." >>$LOG
find /var/www/html/"$LAY"tiles_old/ -type f -delete 2>/dev/null
rm -rf /var/www/html/"$LAY"tiles_old 2>>$LOG

# download and process planet file - if necessary
if [ "0"$(stat --print %s $PLANETFILE 2>/dev/null) -lt $PLANETMINSIZE ]; then
  echo $(date)"  Missing file $PLANETFILE, downloading it." >>$LOG
  rm -f $PLANETFILE
  wget -nv $PLANETURL -O - 2>>$LOG |./osmconvert - \
    $BORDERS --drop-author -o=$PLANETFILE 2>>$LOG
  if [ "0"$(stat --print %s $PLANETFILE 2>/dev/null) -lt $PLANETMINSIZE ]; then
    echo $(date)"  toolchain Error: could not download"\
      $PLANETURL >>$LOG
    exit 1
    fi
  rm -f $DTFILE $DTDIR""*
  echo $(date)"  Filtering the downloaded planet file." >>$LOG
  ./osmfilter $PLANETFILE --parameter-file=toolchain_"$LAY".filter \
    -o=$FILTER 2>>$LOG  # filter the planet file
  echo $(date)"  Writing filtered data into the database." >>$LOG
  osm2pgsql $OSM2PGSQLPARAM -c $FILTER -e 18-18 -o $DTFILE >/dev/null 2>&1
    # enter filtered planet data into the database
  echo $(date)"  All tiles need to be rerendered." >>$LOG
  # prepare tiles directory
  D=/var/www/html/"$LAY"tiles_new
  mkdir $D $D/0 $D/1 $D/2 $D/3 $D/4 $D/5 $D/6 $D/7 $D/8 $D/9 $D/10 \
    $D/11 $D/12 $D/13 $D/14 $D/15 $D/16 $D/17 $D/18 2>/dev/null
  fi

# main loop
while [ -e $RUNNING ]; do
  echo $(date)"  Processing main loop." >>$LOG

  # limit log file size
  if [ "0"$(stat --print %s $LOG 2>/dev/null) -gt 1500000 ]; then
    echo $(date)"  Reducing logfile size." >>$LOG
    mv $LOG $LOG"_temp"
    tail -c +1000000 $LOG"_temp" |tail -n +2 >$LOG
    rm -f $LOG"_temp"
    fi

  #care about entries in dirty-tile list
  MAINTCOUNT=0
  while [ $(ls $DTFILE $DTDIR""?t* 2>/dev/null |wc -l) -gt 0 -a \
      -e $RUNNING ]; do
      # while still tiles to render

    # clean-up failed rendering tasks (about every 20 minutes)
    MAINTCOUNT=$(($MAINTCOUNT + 1))
    if [ MAINTCOUNT -gt 200 ]; then
      T=$(find d_t_"$LAY" -name "at*" -mmin +100|head -1|tail -c +7)
      if [ "1"$T -gt "1" ] ; then
        mv -n d_t_"$LAY"/at$T d_t_"$LAY"/dt$T 2>/dev/null
        echo $(date)"  Restarted rendering: "$T >>$LOG
        fi
      MAINTCOUNT=0
      fi

    # start as much render processes as allowed
    while [ $(ls $DTDIR""dt* 2>/dev/null |wc -l) -gt 0 -a \
        $(ls -1 $DTDIR""at* 2>/dev/null |wc -l) -lt $MAXPROCESS ]; do
        # while dirty tiles in list AND process slot(s) available
      DT=$(ls -1 $DTDIR""dt* |head -1)  # tile list
      AT=${DT:0:6}a${DT:7:99}  # new name of the tile list
      touch -c $DT  # remember rendering start time
      mv $DT $AT  # rename this tile list file;
        # this is our way to mark a tile list as 'being rendered now'
      #echo $(date)"  Rendering "$DT >>$LOG
      PROCN=$(($PROCN + 1))
      if [ $PROCN -gt 1999 ]; then
        PROCN=1000;  # (force range to 1000..1999)
        fi
      N=${PROCN:1:99}  # get last 3 digits
      LAY=$LAY DTF=$AT PID=$N nohup python3 render_"$LAY".py >/dev/null 2>&1 &
        # render every tile in list
      echo $(date)"  Now rendering:"\
        $(ls -m $DTDIR""at* 2>/dev/null|tr -d $DTDIR"a ") \
        >>$LOG
      done

    # determine if we have rendered all tiles of all lists
    if (ls $DTDIR""?t* >/dev/null 2>&1); then  # still tiles to render
      sleep 6  # wait some seconds
  continue  # care about rendering again
      fi
    # here: we have rendered all tiles of all lists

    # care about dirty-tiles master list and split it into parts
    if (ls $DTFILE >/dev/null 2>&1); then
        # there is a dirty-tiles master list
      echo "=== "$(date) >>$DTLOG
      cat $DTFILE >>$DTLOG  # add list to dirty-tiles log
      echo $(date)"  Expanding \"dirty_tiles\" file." >>$LOG
      ./dtexpand 11 18 <$DTFILE >$DTFILE"_ex" 2>/dev/null
      mv -f $DTFILE"_ex" $DTFILE 2>/dev/null
      echo $(date)"  Splitting dirty-tiles list" \
        "("$(cat $DTFILE |wc -l)" tiles)" >>$LOG
      split -l 1000 -d -a 6 $DTFILE $DTDIR""dt
      echo "+++ "$(date) >>$DTLOG
      cat $DTFILE >>$DTLOG  # add list to dirty-tiles log
      rm -f $DTFILE
      # limit dirty-tiles log file size
      if [ "0"$(stat --print %s $DTLOG 2>/dev/null) -gt 750000000 ]; then
        echo $(date)"  Reducing dirty-tiles logfile size." >>$LOG
        mv $DTLOG $DTLOG"_temp"
        tail -c +500000000 $DTLOG"_temp" |tail -n +2 >$DTLOG
        rm -f $DTLOG"_temp"
        fi
      fi

    done  # while still tiles to render
  if [ ! -e $RUNNING ]; then  # script shall be terminated
continue;  # exit the main loop via while statement
    fi
  # here: all tiles have been rendered

  # check if there is a plausible amount of tiles
  if [ "0"$(du -s /var/www/html/"$LAY"tiles_new 2>/dev/null|cut -f 1 2>/dev/null) \
      -lt $TILEMINSIZE ]; then
    echo $(date)"  toolchain Error: implausible amount of tiles" >>$LOG
    exit 1
    fi

  # move tile directory
  echo $(date)"  Moving tile directory." >>$LOG
  ls -lL $PLANETFILE $FILTER >>$LOG
  rm -f $PLANETFILE $FILTER
  mv -f /var/www/html/"$LAY"tiles /var/www/html/"$LAY"tiles_old 2>/dev/null
  mv -f /var/www/html/"$LAY"tiles_new /var/www/html/"$LAY"tiles 2>/dev/null

  break
  done  # main loop

# wait until all rendering processes have terminated
while (ls $DTDIR""at* 2>/dev/null) ; do sleep 30; done

# publish that this script has ended
rm -f $RUNNING
echo "The toolchain script ended at "$(date)"." \
  >$ENDED
echo -e $(date)"  Toolchain ended.\n" >>$LOG

Make the shell script executable:

chmod +x toolchain_hy.sh

Now start the toolchain and wait until it is completed. Before this you may want to change some parameters in the toolchain script: the number of rendering processes, for example.

cd /root/hy
./toolchain_hy.sh

Depending on the size of the region you have downloaded this step may take a few minutes up to several hours. You can watch the rendering process(es) by invoking the process monitor top (from a separate terminal window):

top

Inspect the logfile tc_hy.log for any possible errors which might have occurred. If everything went fine, it is time to automatise the rebuild process. This can be done via crontab:

crontab -e

Enter this line:

22 5 3 * * /root/hy/toolchain_hy.sh

Now the map will be rebuilt automatically every month (at 05:22 on day 3). Please be aware that all OSM data will be downloaded from the location you have chosen above and therefore lead to some effort on the side of that company. Please refer to their terms for using the service. If you are building this map for commercial purposes, please consider to make a donation – no matter if it is legally required or not.

Build the Website

The web server Apache will expect our web content at "/var/www/html/". To display, move and zoom the map on the screen a specialized framework will be needed: openlayers. For this, some additional files must be copied resp. downloaded:

cd /var/www/html
cp /usr/share/javascript/openlayers/OpenLayers.js .
cp /usr/share/openlayers/theme/default/style.css .
mkdir img
cp /usr/share/openlayers/img/* img/
wget -N https://www.openstreetmap.org/openlayers/OpenStreetMap.js

Now open the existing example index file and replace its contents by the following text. You can use gedit or nano editor for this task: nano /var/www/html/index.html

<!DOCTYPE HTML>
<html> <head>
<title>OpenFireMap</title>
<title>Fire Hydrants</title>
<link rel="stylesheet" href="style.css" type="text/css">
<style type="text/css">
html, body, #map {
  width: 100%;
  height: 100%;
  margin: 0;
  }
.olImageLoadError { 
  display: none !important;
  }
</style>
<script src="OpenStreetMap.js"></script>
<script src="OpenLayers.js"></script>
<script>

function init() {
  OpenLayers.Util.onImageLoadError= function() {
    this.src= "emptytile.png"; };
  map= new OpenLayers.Map("map", { controls:[
      new OpenLayers.Control.Navigation(),
      new OpenLayers.Control.PanZoomBar(),
      new OpenLayers.Control.LayerSwitcher(),
      new OpenLayers.Control.Permalink(),
      new OpenLayers.Control.ScaleLine(),
      new OpenLayers.Control.Permalink('permalink'),
      new OpenLayers.Control.MousePosition(),                    
      new OpenLayers.Control.Attribution()],
    maxResolution: 156543.0399,numZoomLevels: 18, units: 'm',
    projection: new OpenLayers.Projection("EPSG:900913"),
    displayProjection: new OpenLayers.Projection("EPSG:4326"),
    restrictedExtent: new OpenLayers.Bounds(550000,5700000,2000000,7450000)
    } );
  map.addLayer(new OpenLayers.Layer.OSM("OSM"));
  map.addLayer(new OpenLayers.Layer.OSM("No Background","img/blank.png"));
  map.addLayer(new OpenLayers.Layer.OSM("Fire hydrants","hytiles/${z}/${x}/${y}.png",
    { maxZoomLevel: 18, numZoomLevels: 19, alpha: true, isBaseLayer: false }));
  if(!map.getCenter()){
    map.setCenter(new OpenLayers.LonLat(11.075,49.455).transform(
      new OpenLayers.Projection("EPSG:4326"),map.getProjectionObject()),11);
    }
  }

</script>
</head><body onload="init();">
<div style="width:100%; height:100%;" id="map"></div><br>
<div style="border:1px solid black; padding:10px;">
This map shows those fire hydrants which have been entered into the OpenStreetMap database. <a href="http://openstreetmap.org/copyright" target="_blank">(c) OpenStreetMap contributors</a>
<br><br>
Further information can be found at OpenStreetMap Wiki in
<a href="http://wiki.openstreetmap.org/wiki/OpenFireMap" target="_blank">English</a>,
<a href="http://wiki.openstreetmap.org/wiki/DE:OpenFireMap" target="_blank">German</a> and
<a href="http://wiki.openstreetmap.org/wiki/Pl:OpenFireMap" target="_blank">Polish</a><br>
</div>
</body></html>

Further information about OpenLayers can be found here.

Test your new Map

Open a web browser and try to access your new website. If your Internet browser is on the same computer as the Apache server, type localhost or 127.0.0.1 as URL. If you did not render the complete map yet but have rendered that single tile from the rendering test example above, you will find it here: 127.0.0.1/?zoom=10&lat=49.5&lon=11&layers=B0000T

Please be aware that this map is an overlay map. That means in this case, the background tiles are fetched directly from openstreetmap.org map server, and just the foreground tiles – the OpenFireMap layer – comes from your server.

Benchmarks

Example setup: weak virtual server, two cores, OSM data from DACH region (Germany, Austria, Switzerland)

  • 10 minutes downloading and converting the OSM data file
  • 2 minutes filtering the file
  • 20 seconds writing the filtered data into the database
  • 2 hours rendering all tiles of the OpenFireMap overlay (1.2 Mio. tiles)

Troubleshooting

Too many links

If this message occurs during rendering, you most likely are using a file system which supports only a limited number of subdirectories (e.g. ext2 or ext3). Please upgrade to ext4. The migration from ext3 to ext4 should be possible without any loss of data. Please refer to the users guide of your operating system.