Mapsplit

From OpenStreetMap Wiki
Jump to: navigation, search

Mapsplit is a command line Java application for splitting OSM data in pbf format into single tiles. The tool is written with a custom hashmap and bitmanipulation for an efficient and fast way to split the data in main memory. The code is open source and under public domain.

Current status

Mapsplit is activly used to split OSM data for 3D rendering using OSM2World. Therefore bayern.pbf (225MB) is split in 2-3 minutes on a average PC with all data kept easily in main memory.

Downloading

The source code and a prebuilt jar can be obtained from github.

Compiling

Initiate an

ant

in the directory in which you'll find "build.xml".

Usage

Mapsplit has a simple command line interface. The quick usage description is given with the help option

mapsplit -h

The most basic way to run mapslipt will take an input.pbf file and write any tile that has changed in the last two days to output_dir

mapsplit infile.pbf output_dir

Options

There are also a number of options to influence the behaviour of mapsplit:

  • -v, --verbose will print additional information during the splitting process
  • -t, --timing will output timing information
  • -m, --metadata will include extra metadata into the written tilefiles. You usually won't need these extra data, however some tools (e.g. JOSM) depend on them for further processing
  • -c, --completeMPs will store the complete data needed for multi polygons in each tile
  • -f, --fd-max=val denotes the maximum number of open file handles mapsplit may use at any time
  • -b, --border=val enlarge tiles by val ([0-1[) of the tile's size to get a border around the tile.
  • -p, --polygon=file only save those tiles that intersect or lie within the given polygon file
  • -d=file, --date=file will use the Data (written in UTF-8 by Java's DateFormater) in file as the base time. Every tile that has changed after this time will be written by the split process.
  • -s=n,w,r, --size=n,w,r tells the code how big the hashmaps size should be chosen. The defaults are quite large, however, if they don't fit for your input file you have to enlarge the hashmaps. With the verbose-flag above you can see the load of the hashmaps for your input.

Development

Mapsplit is reliable, however, there are still some missing features and there are still possible improvements that will be incorporated in the future.

A list of things that will probably be added in the near future can be found at github:

Notes

As the data gets written as it's read, mapsplit needs to keep many open file descriptors. In a default linux installation a user process may typically only open 1024 files at a time. For larger input data with many changes this limit will be reached quite soon. To extend this limit edit /etc/security/limits.conf and add the following line

heinz            -       nofile          4096

This will allow a process run by user heinz to open 4096 at once.


In case you run of of memory use

-Xmx2G

or appropriate.

See also