Tiles@home/Server install guide
 |
Deprecated - Tiles@home is no more |
The Tiles@home server is the central hub of the Tiles@home distributed rendering system. Clients running all over the world talk to this server to get new rendering jobs, and upload their rendered results.
To be clear, the tile@home server is not the bit that runs "at home". It's not map rendering software. If you're interested in rendering maps, there are many rendering alternatives, including osmarender which forms part of the tiles@home client. This guide is to help developers who might be interested in running the server software (mostly to help with developing it)
Contents |
Prerequisites
- Connectivity note: (June 2008) Network traffic has increased to abouth 3 TB/month (2 TB/mo inbound, 1 TB/mo outbound), with 8-9M Apache hits/day while the CPU usage is mostly IOwait, not acutal CPU time.
- Disk Storage: Whole world requires lots of gigabytes of storage space. It was earlier running on ReiserFS (mounted notails noatime) to not waste to much unused space per file. This is not a big issue now when using tilesetfiles.
Programs and libraries
In a Debian/Ubuntu system, you can install most of them with the following command
sudo apt-get install apache2 libapache2-mod-python python-imaging python-pysqlite2 wget unzip
Django needs a database module to store its data in. If you don't have a preference, sqlite is the simplest to use. You will need a python module for your database, such as python-mysql.
In Debian now 2008-08-03 we also need a not yet released development subversion version of the python-django package. Make your own package or download here python-django_0.97~svn8198-1_all.deb md5sum 695f6db34d1d4dd9ebab7045978b5cfe and install with sudo dpkg -i python-django_0.97~svn8198-1_all.deb
Installation
Download the source and main directory from the OpenStreetMap svn server.
svn co http://svn.openstreetmap.org/sites/other/tilesAtHome_tahngo
- Put it in a location of your choice, eg. /usr/local/tah. The base directory name must be "tah". The svn download will give you a directory named "tileAtHome_tahngo" by default; change it to "tah".
mv tilesAtHome_tahngo tah
- The "Tiles" directory should be moved to the place where you want to store all the tileset files. The setting "base_tile_path" should point to this directory as it will save tileset files here and expect the stock images (blank sea/land/error) here. This is set later in the Django configuration. It also needs to be writable from the webbserver user. In case of Debian it is www-data.
- The "media" directory contains files that should be accessible through the web server and should be moved to a place where the files can be served. The setting MEDIA_URL in settings needs to point to the URL that contains the "media" path. http://tah.openstreetmap.org/ if the media is in http://tah.openstreetmap.org/media/
Adapt the settings to your configuration
- Move settings.py.example to settings.py
- Database choice
- Where it is located
- Change LOGIN_REDIRECT_URL to just /user
- URL-handler change to ROOT_URLCONF = tah.urls
- Update TEMPLATE_DIRS to the tah/templates dir.
Check the django installation
- Go to the media directory and add links for the administration webinterface if missing
In Debian:
ln -s /usr/share/python-support/python-django/django/contrib/admin/media/img . ln -s /usr/share/python-support/python-django/django/contrib/admin/media/js . ln -s /usr/share/python-support/python-django/django/contrib/admin/media/css .
- Go to the the python site-packages directory and check that django is active there.
ls -l `python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"`/django
If the link to django is missing, create it.
ln -s /usr/share/python-support/python-django/django `python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"`
- Run "python manage.py syncdb" to create the database
- Configure your web server for the new paths.
- Read docs/INSTALL in the tah directory for apache vhost code
- Also, for django n00bs : tah.settings is not a real file. Set the PythonPath (vhost) to the parent dir of tah. Django will then load tah.settings as "tah/settings.py" automatically.
- In a web browser, go to the /admin interface and add the settings "base_tile_path", "unzipPath","logFile" to the table Settings.
- In the case of tah.osm.org these settings are "/mnt/agami/openstreetmap/tah/Tiles", "/tmp", and "tah.log" respectively.
- In tah_intern/Tile.py the base_tile_path is hardcoded for efficiency reasons. Open that file and adapt the path there too.
- tah_intern/views.py currently contains a hardcoded path to the log file location. Adapt the line: f = open('/home/spaetz/tah/tah_intern/tah.log', 'rU')
- >= r17297 does not need this.
- tah_intern/LegacyTileset.py contains hardcoded paths to your oceantiles.dat file and your fallback legacy tile directories.
- Configure the html templates for the URL to your tile server. In the templates directory, the files base_browse.html, base_browse_slippy.html, and tile_details.html refer to tah.openstreetmap.org/Tile. For a test installation, change this to the URL for your "/Tile" location in your Apache configuration below.
In the web admin interface, you will need to add some "layers" for your maps. In t@h these are currently: tile (the default layer from z0-17), captionless (from z0-17; part of the default layers), maplint(transparent; from z12-17; part of the default layers), and caption (transparent; from z0-17).
Oceantiles
Get or better link to the oceantiles.dat file from the t@h client. The oceantiles file is used to roughly categorize Z12 tiles into land or sea tiles. As this file gets regularly refined, you should probably keep this file up to date with svn.
Running
- Processing uploaded tilesets will not happen automatically by the django server this is done in a separate thread. In the directory tah_intern, you need to call "./process_uploads" to unzip the uploads and put the resulting tilesetfiles to their final location. To start and stop this process more conveniently there is Tools/startstop_process_uploads which is a variant of a Debian like /etc/init.d/ script. You need to adapt some paths in here (and create the directory where the .pid file will be saved) and you can then use ./startstop_process_ploads start|stop|restart to start the tile processing. If it already runs, it will not be started again, so I run the startstop script with option "start" every half hour in a cron script to restart it after a reboot (or an unlikely crash).
Other
Munin
- Install munin if desired
In Debian/Ubuntu, install with
apt-get install munin munin-node
- Point munin to /Log/Requests/History/munin.php and /Log/Requests/History/munin_conf.php to get the monitoring of queue length
As root:
cd /etc/munin/plugins
cat > tah << 'EOF'
#!/bin/sh
case $1 in
config)
cat ~ojw/public_html/Log/Requests/History/munin_conf.txt
exit 0;;
esac
cd ~ojw/public_html/Log/Requests/History; php munin.php
EOF
chmod +x tah
- Restart munin
In a Debian/Ubuntu server, as root, execute the command:
/etc/init.d/munin-node restart
Crontab entries
- Edit the crontab of the user ojw
crontab -e
- Add the following entries to the file, replacing dev.openstreetmap.org with your HTTP server name
# m h dom mon dow command 0 0 * * * wget -O /dev/null http://dev.openstreetmap.org/~ojw/Upload/rmtemp.php 2>&1 &>/dev/null 40 9 * * * wget -O /dev/null --timeout=3600 --tries=1 http://dev.openstreetmap.org/~ojw/Stats/update.php 2>&1 &>/dev/null 42 13 * * * perl /home/ojw/parselog/parselog.pl < /var/log/apache2/access.log.1 > /home/ojw/public_html/Stats/Data/access.htm 15 */2 * * * wget -O /dev/null http://dev.openstreetmap.org/~ojw/Log/Requests/History/update.php 2>&1 &>/dev/null 25 */2 * * * wget -O /dev/null http://dev.openstreetmap.org/~ojw/Log/Requests/History/timeout.php 2>&1 &>/dev/null 0 * * * * /home/ojw/queue_handler/check_runner.sh > /dev/null
Meaning of the entries:
- Remove old temporary directories
- Daily export of tile details
- Update daily access logs
- Record history of queue size (optional, replaced by munin)
- Remove old items from queue, and re-request items which weren't uploaded as promised
- Run the script which checks that the queue handler is running
