User:Geekasylum/Gentoo Linux Virtual Tile Server
Gentoo Linux Virtual Tile Server
I'm building a virtual tile server to become part of my local network. I figure that a guide to setting up an up-to-date tile server on Gentoo will be useful, so here it is. This HowTo will also work for a real server - just skip over the section on building a virtual guest operating system, and use the Gentoo Handbook directly to install Gentoo on a real machine.
Note: While this page is still in my user space, its incomplete and being worked on. Please bear with me.
The virtual machine that I'm building is set up specifically for my network. Ie: It's registered in my DNS, and uses IP addresses that have been allocated to me, but its fairly simple to customise your own system to your needs while you install the base operating system.
The Virtual Tile Server described here will make use of the following applications:
- Apache (latest)
- Postgresql 9.1 and Postgis 2.0
- Mapnik 2.1
- Mod_tile and renderd (latest)
- Osm2pgsql (latest)
You will need VirtualBox, running on your physical server to create and run virtual machines. This is easily installed on either Linux or Windows (and yes, you can run your Gentoo Linux Virtual Tile Server as a virtual machine under Windows if you like).
To install VirtualBox on a physical Gentoo Linux server, run the following command:
emerge -av virtualbox
It comes with command-line utilities to manage your virtual machines, but also has a GUI which makes common tasks easy.
Gentoo Virtual Guest
Follow the instructions here to create a Gentoo Virtual Guest. It's entirely up to you if you want to install X and a desktop such as KDE. I chose not to install a desktop on my virtual machine.
For my system, which will be part of a larger network, I added the following to the base installation. You probably wont need most of these - the Gentoo Handbook gives simpler alternatives to syslog-ng, and joe is our preferred editor. You can use the default editor (nano) or install app-editors/vim to edit the config files.
Set the following global use flags either by editing /etc/portage/make.conf directly, or installing and running app-portage/ufed
Note: On older distributions make.conf is in /etc. On my system, ufed expected it to be in /etc, but that may just be because I didn't reinstall ufed after moving make.conf.
+postgres +postgis +threads +icu +bidi +geos +gdal
The only reason you would need to install X and a desktop is to run a web browser on the virtual machine. Since I already have a web browser on another machine I decided not to install X and unset the appropriate flag. Since Im not building X, I don't need gtk, and I also disabled perl to prevent packages from pulling in support for various perl libraries that we don't need.
-X -gtk -perl
Some packages (like cairo) require flags that we just disabled (in this case X). Portage will warn you when this happens. Simply add those packages to /etc/portage/package.use and enable the required flags there. This way you dont have to set them globally - just on the package that needs them.
Enabling php support in geos causes threading conflicts with apache and php since its not thread-safe and forces -threads on these. You will have a much easier time if you simply disable php support in geos.
Add the following to /etc/portage/package.use, substituting the versions that your own system will be installing.
# Cairo needs X =x11-libs/cairo-1.10.2-r2 X # geos causes use flag conflicts with apache and php. The current version also seems locked to php-5.3 =sci-libs/geos-3.3.3 -php
Whenever you're installing a package with emerge, you should examine the list of dependent packages to be installed and set any use flags which will allow them to build support for other packages that have been installed previously. Do a pretend emerge, before you launch the real thing, and look them over: (note the 'p' and 'v' options below)
emerge -pv <package>
Add any use flags that will be beneficial to your build, and remove any that will install extra packages that you don't need. You can quickly set/unset use flags globally with ufed, or add the individual packages to /etc/portage/package.use.
Many of the packages we are about to install are not part of the official portage tree and are therefore not marked as 'stable' for any given architecture. Some of the official packages are not yet stable on some architectures, so portage will prompt you along the way to add them to /etc/portage/package.keywords. Go ahead and add them as required.
For this project I used my own portage overlay which you can also use to obtain the ebuilds which are not yet in the official portage tree. Make sure that you don't have the git use flag set. For this overlay we only require subversion support, and enabling git will pull in a lot of packages that we don't need. Just set the subversion use flag, as shown in the example (Of course if you really need git for your own build, you can go ahead and add it - we just don't need it for the virtual server described here).
Install Postgres Database
Installing postgis will also install postgresql, since it is a dependency, but to ensure that postgresql is added to the world file (and thus updated by future portage updates) we will install them separately.
Note: This will pull in several other packages, including php
emerge -av postgresql emerge -av postgis
Portage will suggest the following. Ignore it. We do this by hand later
* To finish installing or updating PostGIS edit: * /etc/postgis_dbs * * Then, run: * emerge --config =dev-db/postgis-2.0.0
Edit /etc/conf.d/postgres-9.1 and ensure the following is set to a valid locale that is installed on your system.
You can get a list of installed locales with the following command
Once the locale has been set, use portage to configure postgresql
emerge --config =dev-db/postgresql-server-9.1.5
By default, Gentoo is set up to be secure, however postgres allows full access to anyone with a shell account. Lets change that to make it require passwords (Check below to see how we avoid having to type the password every time)
Edit /etc/postgresql-9.1/pg_hba.conf and change 'trust' to 'password' for the localhost connections only
# IPv4 local connections: host all all 127.0.0.1/32 password # IPv6 local connections: host all all ::1/128 password
Start the postgresql server
# /etc/init.d/postgresql-9.1 start
Open a connection to the server and set the password for the postgres super user.
# psql -U postgres
Set a password
postgres=# \password Enter new password: Enter it again: postgres=# \q
Edit /etc/postgresql-9.1/pg_hba.conf again, and this time, change 'trust' to 'password' for the local socket connection.
# "local" is for Unix domain socket connections only local all all password
Tell the server to reload it's configuration.
Typing passwords all the time is no fun, so lets fix that. In your home directory (cd ~) create a file called .pgpass and add the following to it. Put the password you just created in the password field. The first line is a comment so you can remember where everything goes. Note that the file name begins with a dot (.)
If you're using nano as your editor, you would enter:
Add the following:
# hostname:port:database:username:password localhost:*:*:postgres:<your password>
Postgres will ignore this file if anyone else can read it, so set the permissions like so:
chmod 600 ~/.pgpass
Create the user that mapnik and other software will use to access the database. You can make it a super user for now (just say 'yes' when asked). We will refer to this user as <username> but you type in the username that you chose.
createuser -U postgres <username>
Create the database (most applications assume its called 'gis' by default, so lets go with that)
createdb -U postgres -E UTF8 -O <username> gis
Set a password for your database user
# psql -U postgres
Set the password
postgres=# \password <username> Enter new password: Enter it again: postgres=# \q
Add the database user to your .pgpass file
# hostname:port:database:username:password localhost:*:*:postgres:<postgres user password> localhost:*:*:<username>:<database user password>
Set up postgis on the database
psql -U <username> -f /usr/share/postgresql-9.1/contrib/postgis-2.0/postgis.sql -d gis psql -U <username> -d gis -c "ALTER TABLE geometry_columns OWNER TO <username>;" psql -U <username> -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO <username>;"
emerge -av osm2pgsql psql -U <username> -f /usr/share/osm2pgsql/900913.sql -d gis
emerge -av mapnik
Mapnik requires at least 3Gb of memory to compile and will take a while. If the build fails with cc1plus (internal error) you probably need to give the virtual machine more memory. I had 2Gb of real memory and 2Gb of swap.
Verify the mapnik installation
python >>> import mapnik >>>
If python replies with the second chevron prompt (>>>) mapnik is installed properly. Type quit() or press CTRL-D to exit.
This will also install Apache
emerge -av mod_tile
Install Mapnik Utilities
We need these to configure renderd later.
emerge -av mapnik-utils
We will also need unifont since Mapnik-2.1 uses it by default.
emerge -av media-fonts/unifont
Edit /etc/conf.d/apache2 and add -D MOD_TILE to the options
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D MOD_TILE"
Configure mod_tile and renderd
Edit the mod_tile configuration file installed as /etc/apache/modules.d/14_mod_tile.conf. I have updated the file installed by my portage overlay and taken care of most of it, but you should check that the server name matches your local machine (localhost should be fine for now) and notice the <Directory> section that I have added towards the end. The Apache default is to not allow access to anything. This section allows access our web site's DOCROOT directory where the slippymap is installed. (/var/www/osm).
Edit /etc/renderd.conf. There are a few changes needed here - its simpler if I just show you mine
[renderd] socketname=/var/run/renderd/renderd.sock num_threads=4 tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET stats_file=/var/run/renderd/renderd.stats [mapnik] plugins_dir=/usr/lib/mapnik/input font_dir=/usr/share/fonts font_dir_recurse=1 [default] URI=/osm/ XML=/usr/lib/mapnik/osm-local.xml HOST=localhost ;HTCPHOST=proxy.openstreetmap.org
Lines beginning with a semicolon are comments
Make sure the socketname is uncommented. Check your paths for font_dir, plugins_dir, the host name, and the URI and XML settings.
We will create the XML stylesheet in a moment. The URI is the virtual path used by mod_tile (and also found in the slippymap html).
The tile_dir path cannot be changed here, but the example shows where it should be.
Next, create the directories needed by renderd
mkdir /var/lib/mod_tile mkdir /var/run/renderd
Load some data
You can download the complete planet.osm file https://planet.openstreetmap.org/
To save space we chose not to install the entire planet file and used an extract from http://download.geofabrik.de/openstreetmap/
You can use bzip2 or pbf files directly with the latest osm2pgsql:
wget http://download.geofabrik.de/openstreetmap/australia-oceania/australia.osm.bz2 osm2pgsql -U <username> -d gis -m -s australia.osm.bz2
Generate the stylesheet
Generate the default stylesheet for renderd
cd /usr/lib/mapnik ./generate_xml.py osm.xml osm-local.xml --dbname gis --user <username> --accept-none
Edit the slippymap
Edit the slippymap html located at /var/www/osm/index.html
- Update the start lat/lon and zoom (if you dont know the values, you can do this later)
- Update URI for local tiles (localhost/osm)
Add startup scripts
rc-update add postgresql-9.1 rc-update add apache2 rc-update add renderd
Have a coffee - we're done
Point your browser at the slippymap on http://localhost/ and play
Now that our tile sever is built, we'll be exploring the following options, and more as we develop applications on top of it.
- configure remote access to the postgres database if needed (/etc/postgres-9.1/pg_hba.conf)
- emerge phppgadmin and install the webhost to visualise/manage the postgres database
- Fine tuning - Eg: add use flags to enhance php features
- Disk space
At this point, having browsed around the slippy map a bit the system is using 8Gb of disk space. The virtual disk image is using 13Gb which suggests I should use the Virtualbox tools to compact the image.
- Expiry of tiles and updating the database
There are some scripts in the mod_tile distribution but they are hard coded and out of date. They could be used as examples to create your own expiry/update process. These were not installed by default. Look for them in the SVN sources under /usr/portage/distfiles/svn-src