The Fugro aerial imagery layer is based on a bunch of data and software. At the moment the layer is maintained by Peter Brodersen.
This page describes our general server setup. The setup is Linux based; Debian Stable (Lenny).
The data source is 662 .ecw files from Fugro Aerial Mapping A/S. The files span from few MB to 235 MB. The total size is 77.4 GB.
Every file covers a quadrant of 10km x 10km in EPSG25832 projection (Euref89/ETRS89). The files are named after their position, e.g. 10km_636_59.ecw - where 636 og 59 is the Northing and Easting (* 10 km) values.
All files are indexed in a Shapefile using
gdaltindex -tileindex LOCATION -write_absolute_path fugroindex.shp 10km_ecw_e89/*.ecw
This updates (and creates if necessary) the file
fugroindex.shp which we point Mapserver at.
The primary tool is Mapserver which is a CGI application. At default the GDAL library (that mapserver is linked to) does not support ECW, which we then have to compile.
For Debian Stable (Lenny) we have fetched the following resources with git:
git clone http://rivendell.lovergine.com/pubgit/ecw.git/ git clone http://rivendell.lovergine.com/pubgit/libgdal-ecw.git/
First .deb files are built from ecw just by running
debuild. After they are installed a .deb is built from libgdal-ecw in the same way, and is installed.
You may have trouble building the libgdal-ecw package. If so, you can build and install it manually (it is only a single file) using the following:
sh configure --with-ecw=/usr --with-autoload=/usr/lib/gdal16plugins --with-include=/usr/include/ecw make sudo mkdir -p /usr/lib/gdal16plugins sudo cp gdal_ECW_JP2ECW.so /usr/lib/gdal16plugins
Now there is support for ECW files in libgdal. This can be confirmed with:
gdalinfo --formats|grep -i ecw
.. which would output something like:
ECW (rw): ERMapper Compressed Wavelets JP2ECW (rw+): ERMapper JPEG2000
Furthermore, mapserver from Debian Stable (Lenny) is an old version that does not support tilemode. If you run a newer version than Lenny (or perhaps another distribution) with a newer mapserver, it should work now. Otherwise we would have to compile a newer version af mapserver ourselves, just by downloading the mapserver source, and configure and make. The produced mapserv file can be moved to /usr/lib/cgi-bin/ or another adequate place for script execution.
In our case mapserver was configured with these options:
./configure --with-proj --with-gdal --with-agg --with-jpeg
Mapfiles are configuration files for Mapserver, which describes the options of what is to be outputted. These refer to the fugroindex.shp shapefile.
Ideally one could do with just one mapfile with different layers configured.
fugro.map with the layer "fugro" is used to output tiles:
MAP IMAGETYPE jpeg EXTENT 470000 6070000 730000 6250000 SIZE 400 300 IMAGECOLOR 255 255 255 PROJECTION "init=epsg:25832" END LAYER NAME "fugro" STATUS ON TILEINDEX "/home/username/web/fugro/mapserv/fugroindex.shp" TILEITEM "LOCATION" TYPE RASTER END END
overview.map is an overlay that shows which parts the different .ecw files covers. It can also be used if an area contains error, but the layer itself might not be that useful inside an editor. The overview also helped prioritizing the queue of files for upload, before they were all online.
The output can be seen at http://tile.openstreetmap.dk/?layers=BFT
MAP IMAGETYPE transpng EXTENT 440000.000000 6040000.000000 900000.000000 6410000.000000 SIZE 3000 1800 IMAGECOLOR 255 255 255 SHAPEPATH "/home/username/web/fugro/mapserv/10km_ecw_e89" FONTSET "/home/username/web/fugro/mapserv/fonts.list" PROJECTION "init=epsg:25832" END OUTPUTFORMAT NAME "transpng" DRIVER AGG/PNG MIMETYPE "image/png" IMAGEMODE RGBA TRANSPARENT ON EXTENSION "png" END LAYER NAME "denmark" STATUS ON DATA ortooversigt TYPE POLYGON LABELITEM "NAVN" CLASS NAME "Denmark" STYLE COLOR 232 232 232 OUTLINECOLOR 32 32 32 END LABEL COLOR 255 0 0 SHADOWCOLOR 255 255 255 SHADOWSIZE 0 0 TYPE TRUETYPE FONT DejaVuSans SIZE 9 ANTIALIAS TRUE POSITION CC PARTIALS FALSE MINDISTANCE 300 BUFFER 4 WRAP "_" END END END LAYER NAME "fugro" STATUS ON TILEINDEX "/home/username/web/fugro/mapserv/fugroindex.shp" TILEITEM "LOCATION" TYPE RASTER END END
The overview uses ortooversigt.shp, which was shipped along the imagery files and is found in the same folder as the imagery files (10km_ecw_e89).
fonts.list is just a reference to a TrueType file and simply contains:
The server already runs Apache 2.2. A virtual host for the tileserver purpose has been created with the appropriate configuration.
The direct URL to a tile is e.g.:
This includes a reference to the mapfile
fugro.map, the name of the layer, that we want a tile as output, that the coordinates are in gmap format and finally the coordinate for our tile (x,y,z).
As most of these parameters aren't relevant for the end user, and that we would like the option of changing the configuration and renaming files without changing the URLs we choose to rewrite the URLs. This is also for the sake of more simple URLs in the editors' configuration.
Mapserver requires a lot of resources, both with regard to memory and CPU. An obvious optimization is caching of generated tiles. This can be done in several ways (Varnish, Nginx, TileCache and so on). We have chosen Apache's own mod_cache to avoid introducing more layers of software, and to be able to control the caching at the virtual host levels, regardless of other existing virtual hosts at the server.
Now and then mapserver fails at the generation of a tile, usually if the server runs out of memory in the process. As requests are very intensive the memory usage oscillates a lot with several simultaneous users. If maserver crashes under the way a tile will not be generated (and therefore fortunately won't end up as a "dead" tile in the cache). The end user can usually just reload or zoom in and out to regenerate the tile.
We have the following rules in a
.htaccess file, which is located in DocumentRoot for the virtual host:
RewriteEngine On # Aerial Imagery and File Reference RewriteRule ^fugro2005//?([0-9]+)/([0-9]+)/([0-9]+)\.(png|jpe?g)$ /cgi-bin/mapserv?map=/home/username/web/fugro/mapserv/fugro.map&layers=fugro&mode=tile&tilemode=gmap&tile=$2+$3+$1 [PT,L] RewriteRule ^fugro2005/overview/([0-9]+)/([0-9]+)/([0-9]+)\.(png|jpe?g)$ /cgi-bin/mapserv?map=/home/username/web/fugro/mapserv/overview.map&layer=denmark&mode=tile&tilemode=gmap&tile=$2+$3+$1 [PT,L]
The PT (passthrough) flag is set to enable us to refer to /cgi-bin/ which is not located under DocumentRoot, but is ScriptAlias'ed in place.
If one want to move the rules to a virtual host configuration instead of using a .htaccess files the different path in the different RewriteRule expressions must be prepended with en slash, e.g.:
Apache's mod_cache and mod_disk_cache (part of Apache Core in version 2.2, not enabled per default) is configured to cache requests to mapserv. From the virtual host block:
CacheEnable disk /cgi-bin/mapserv CacheIgnoreNoLastMod On CacheMaxExpire 2592000
MaxExpire is set to 2592000 seconds (30 days). Whatever the value of a resource's Expire time in the HTTP header the cache does not make use of older files. Default is only one day so this is a relevant value to raise.
The cache is maintained and vacuumed by
htcacheclean. In Debian this is configured in
/etc/default/apache2. Here it is possible to adjust the cache max size, e.g.:
mod_cache is only caching files with query string (where "?" is part of the internal URL, including all tile requests), if an Expire header is present. Mapserver does not output this header, meaning that we have to add it ourselves.
mod_expires (not enabled per default) gives os this opportunity, based on file type.
In the virtual host block we add the following directives:
ExpiresActive On ExpiresByType image/jpeg A2592000
Here we ask Apache to add an Expires header on image/jpeg output (including the files generated by mapserver - given that we are generating JPEG files), set to 2592000 seconds (= 30 days) after the time of the client request.
This way we ensure that the tiles have an Expires header needed for mod_cache to cache the tile.