Osmconvert

From OpenStreetMap Wiki
Jump to navigation Jump to search

osmconvert can be used to convert and process OpenStreetMap files. It masters fewer functions than the commonly-used Osmosis: for example, there is no way to access a database with osmconvert. However, the program runs faster and offers a few special functions (--all-to-nodes, --complete-multipolygons and --out-statistics).

Download

As usual: There is no warranty, to the extent permitted by law.

Binaries

Linux

Windows

Limitations: The Windows version (the downloaded binary) does not work with large files such as planet.osm (greater than 2Gb) when using the --complete-ways, --complete-multipolygons or --complete-boundaries option. For example, the Windows 64-bit version fails with the command "osmconvert planet.osm.pbf -b=17.7,59.15,18.4,59.5 -o=stockholm.osm --complete-ways --drop-version --verbose". The same command works perfectly using the Linux 64-bit version. This seems to be caused by a problem with seeking to file positions outside the signed 32-bit range, which is a limitation of the zlib library currently linked to the Windows binaries.

DIY: If you want to build your own 64bit binary without limitations, you'll have to build latest zlib with the flags _LARGEFILE64_SOURCE=1 and _LFS64_LARGEFILE=1. And make sure, that type z_off64_t is defined as __int64 in zconf.h

Source

  • source code (regular version) (need to link zlib (i.e. cc osmconvert.c -lz -o osmconvert)
    • (previous source code)
    • (next source code (if available))
    • Download and build in one run:   wget -O - http://m.m.i24.cc/osmconvert.c | cc -x c - -lz -O3 -o osmconvert
    • Install in Ubuntu and upgrate to latest version:   sudo apt install osmctools && wget -O - http://m.m.i24.cc/osmconvert.c | sudo cc -x c - -lz -O3 -o osmconvert

In case of error with "zlib.h" not found : install zlib1g-dev

Start Program

Windows users can start osmconvert by double-clicking on the executable.

Users of all platforms can start osmconvert from the command-line. Simply executing osmconvert starts a friendly, interactive text-based interface which guides users through the process of using the program. The full capabilities of osmconvert are not available in interactive mode. Power users can bypass the interactive interface by passing flags, like this:

osmconvert --help

Program Description

To get a detailed description, please use the built-in help function of the program:

osmconvert --help

This call will display a brief parameter overview:

osmconvert -h

The following chapters demonstrate the most important program functions.

Converting Files

Conversion may take place between these data formats:

If you want to route the program's output to standard output you will have to tell the program which data format shall be used:

--out-osm (default), --out-osc, --out-osh, --out-o5m, --out-o5c, or --out-pbf.

If you supply the output file's name by applying the option -o=, osmconvert will determine the data format by evaluating the file name extension. Examples for both ways:

osmconvert norway.pbf >norway.osm
osmconvert region.pbf -o=region.o5m
osmconvert region.o5m -o=region.pbf
osmconvert 20110510_20110511.osc --out-o5c >20110510_20110511.o5c

You also can use compressed input files if you supply the data via standard input. Examples:

bzcat europe.osm.bz2 | osmconvert - -o=europe.o5m
osmconvert norway.pbf | gzip -1 >norway.osm.gz

The option "-" informs the program to expect input data via standard input.

osmconvert offers limited decompression functionality: you can decompress .gz files. The program will recognize gzip compression on its own, hence you do not need to care about which input file is gzip compressed and which is not. The built-in decompression algorithm is less powerful than specialized decompression programs, however this feature is really useful if you want to update an OSM file by using a number of newly downloaded compressed .osc files. Examples:

osmconvert old.o5m daily_updates/2011*.osc.gz -o=new.o5m
osmconvert daily_updates/2011*.osc.gz --merge-versions --out-osc | gzip > cumulative.osc.gz

PBF input files with non-standard lon/lat granularity

osmconvert expects PBF files to use the standard OpenStreetMap granularity, which is 100 nanodegrees. It is possible to produce PBF files with different granularity using osmosis. If asked to convert a file with a different granularity then osmconvert will give an error: "node nanodegrees must be 100: 10000". The solution for reading files with a non-standard granularity is to use the --pbf-granularity=<val> option.

Applying Geographical Borders

Clipping based on Longitude and Latitude

Supplying geocoordinates is the easiest way to define a geographical region which shall be extracted. Some online maps continuously display the geocoordinates of the mouse cursor. For example: Osmarenderer map, public transport map.

To define this limiting quasi rectangle you need to supply the coordinates of its southwestern and northeastern corners (WSEN). (Bottom Left / Top Right) For example:

osmconvert germany.pbf -b=10.5,49,11.5,50 -o=nuernberg.o5m
osmconvert ontario.osm -b=-75.8,45.19,-75.7,45.23 -o=ott.osm

Clipping based on a Polygon

Instead of a simple bounding rectangle you can use a border polygon file. This will allow a more accurate limitation to a political border, for example:

osmconvert germany.pbf -B=hamburg.poly -o=hamburg.pbf

The format of a border polygon file can be found in the OSM Wiki: here. You do not need to adhere strictly the format description, but you need to ensure that every line of coordinates starts with blanks.

osmconvert is able to deal with separate polygons in one file, it even considers "islands" in polygons if they have been defined properly.

Example using a polygon file from polygons.openstreetmap.fr
  1. obtain the ID of the relation that describes the area you want to extract by following this guide or by pressing ctrl+i on the relation in JOSM.
  2. insert the ID on polygons.openstreetmap.fr and download the poly file
  3. Run osmconvert:
osmconvert country-latest.osm.pbf -B="file.poly" --complete-ways --complete-multipolygons -o=extract.pbf

The complete-* parameters ensures that no elements are clipped at the boundaries, but included in whole.

Clipping OSM Change Files?

It is not recommended to apply geographical borders to change files. Since only nodes carry geographical locations, the program does not know what to do with ways and relations whose related nodes are not in the same file. As a result these ways and relations will be excluded from the file. This is usually not what you would like to accomplish.

Keeping Cross-Border Ways Complete

Some applications require lines (so-called ways) to stay intact even if they lie partially outside the defined geographical region. This can be accomplished by applying the option --complete-ways. Examples:

osmconvert germany.o5m -b=10.5,49,11.5,50 --complete-ways -o=nuernberg.o5m
osmconvert germany.o5m -B=hamburg.poly --complete-ways -o=hamburg.pbf

This option, and that described in the next two sections, will limit the size of the input file to 2 GiB if your operating system is 32 bit Windows. As the input file is needed to be read two or three times, the program must "jump" within this file. Unfortunately the presently used link library does not support long jumps with 32 bit Windows. There is no such limitation for Linux (neither for the 32 bit nor for the 64 bit version).

Likewise for this and the following two sections, it is recommended to use .o5m as data format for the input file. The reason is that .pbf files are usually compressed internally and therefore will be read much slower than .o5m files.

None of the --complete-... option is available through the Osmupdate command.

Keeping Cross-Border Multipolygons Complete

More and more areas of rivers, forests and lakes are surrounded not only by one closed line but by several concatenated lines. So-called multipolygons are used to logically connect these physically connected lines. Each line holds a role depending on the position: "outer" for the surrounding border or "inner" for excluded areas (e.g. an island within a lake).

When applying geographical borders, osmconvert can consider these multipolygons and keep them intact even if there is only a small part of the multipolygon's area within the borders. To instruct the program to do so, choose option --complete-multipolygons. Examples:

osmconvert germany.o5m -b=10.5,49,11.5,50 --complete-multipolygons -o=nuernberg.o5m
osmconvert germany.o5m -B=hamburg.poly --complete-multipolygons -o=hamburg.pbf

Please note the three remarks at the end of the previous section.

Until 2016 this option had been named --complex-ways.

Keeping Cross-Border Boundaries Complete

To ensure boundaries being kept intact, even if some of their way objects lie outside the applied geographical borders, use the --complete-boundaries option. Examples:

osmconvert germany.o5m -b=10.5,49,11.5,50 --complete-boundaries -o=nuernberg.o5m
osmconvert germany.o5m -B=hamburg.poly --complete-boundaries -o=hamburg.pbf

Please note the three remarks two sections above.

Excluding References to Objects outside the Borders

If you need to delete references to nodes which have been excluded because lying outside geographical borders, use option --drop-broken-refs (might be helpful for data imports into OSM Map Composer or JOSM).

Merging two or more Geographical Areas

Under certain conditions, OSM data files can be merged. If they hold objects (nodes, ways, relations) with the same id, they must have the same content. For example: If a way crosses the geographical border between two regional files, the way dataset must contain every node reference, even the references to nodes which do not lie within the borders of the regarding file. I.e., such a region must not have been cut out using the option --drop-brokenrefs. Examples for geographical merging:

osmconvert austria.o5m germany.o5m switzerland.o5m -o=dach.o5m
osmconvert north_america.osm south_america.osm -o=americas.osm

Special case: clipping contour data

See here: Howto render Garmin countour layers with no artefacts

Exclude Information or Contents from the Output File

Dispose of Author Information

For most applications the author tags are not needed. If you decide to exclude user name, user id, changeset and object timestamp information, add the command line option --drop-author. For example:

osmconvert --drop-author a.pbf -o=a.osm

Usually you will not encounter any problems when deleting the author information from .osm or .o5m files, however it is not encouraged to do this with .pbf files because most programs will not cope with this change of format.

If you need to reappend author information at a later time, let's say because a subsequent program depends on this format, you can generate them anew with the option --fake-author. Naturally, the new author information will be just replacement values which adhere the format description, nothing more.

Excluding certain OSM Object Types

If necessary, you can get rid of whole sections of a file:

--drop-nodes
--drop-ways
--drop-relations

The program osmfilter offers more granular filter functionality.

Dispose of Ways and Relations and Convert them to Nodes

Sometimes it will be easier for subsequent processing if the file contains only objects of the most primitive object type: nodes. osmconvert offers a function which deletes every way and every relation and creates a node as replacement for each. Each node's longitude and latitude are set to the geographical center of the deleted object. If the deleted object was a non-closed way, one of its nodes' position will be taken instead of the center. Each tag of the deleted object is copied to the node. As an id for the new node the way's (resp. relation's) id is taken and incremented by 1015 (resp. 2*1015). For example:

osmconvert hamburg.pbf --all-to-nodes -o=hamburg_nodes.osm

The --object-type-offset= option allows you to change the id offset, from 1015 to a different value.

The option --add-bbox-tags will provide a bounding box for each way and relation which has been converted to a node. These bounding boxes will appear as tags. For Example (area of London):

<tag k="bBox" v="-0.5000,51.0000,0.5000,52.0000"/>

Modify Tags

Sometimes it is useful to change certain tags to make further data processing easier.

Note that "--modify-tags" is used for lines, while "--modify-node-tags" is used for nodes. In any case, both keys and values can be changed.

Modify Values

You can specify values to be modified. For example:

./osmconvert a.o5m --modify-tags="highway=primary to =tertiary highway=secondary to =tertiary" -o=all_streets_are_small.o5m

This will make all primary and secondary roads to tertiary ones.

Modify Keys

Keys can also be modified:

./osmconvert a.o5m --modify-node-tags="amenity=fire_hydrant to emergency=fire_hydrant" -o=new_hydrant_syntax.o5m

Add new Tags

Similar to osmfilter's filtering, tag modification by osmconvert allows comparisons. Thus you can add redundant tags if this helps to simplify subsequent processing of your data:

./osmconvert a.o5m --modify-tags="maxspeed<=20 add speed_category=slow" -o=speed_categories.o5m

There is no check if there are already tags with the same key name. If necessary, use tags filter function of osmfilter to prevent possible collisions.

Updating OSM Files

If you have an OSM data file (.osm, .o5m or .pbf), you can merge it with one or more OSM change files (.osc or .o5c) to update it. For example, you have a planet.osm file or a regional germany.o5m file from yesterday, you can apply the daily change file from this morning to get an up-to-date planet.osm, resp. germany.o5m file. The syntax is like this:

osmconvert planet_old.osm changefile.osc -o=planet_new.osm
osmconvert planet_old.o5m changefile.osc.gz -o=planet_new.o5m
osmconvert germany_old.o5m changefile.osc -B=germany.poly -o=germany_new.o5m

In case your data file is older, you can apply two or more change files simultaneously:

osmconvert veryold.osm c1.osc c2.osc c3.osc -o=new.osm
osmconvert day24.o5m c24_25.osc c25_26.osc -o=day26.o5m
osmconvert day01.o5m november/*.osc -o=day30.o5m

The OSM objects in the change file must be unique. That means, there has to be only one occurrence of every node, way or relation. Minutely and hourly change files may contain more than one version of OSM objects, therefore you will get warning messages. You can combine all versions of each object if you specify the option --merge-versions. Then, only the newest version of each object will remain in the file.

To automatically update an OSM file or to create cumulative .osc files, please see osmupdate.

Retrieving the Differences between two OSM Files

You can create an .osc or an .o5c change file by comparing two .osm or .o5m files. For example:

osmconvert old.osm new.osm --diff -o=changefile.osc
osmconvert old.o5m new.o5m --diff -o=changefile.o5c

Other operations, like applying regional borders, are not allowed in the same run. Both files must be sorted by object type and id. Created objects will appear in the output file as "modified", unless having version number 1.

When calculating file differences osmconvert relies on the version numbers of the objects which are to be compared. If the version numbers are not available or if the objects shall be compared by contents, you can order the program to do so by applying the option --diff-contents (works for .o5m files only).

If an object is to be deleted, only its id (and author data) is stored. It has proven useful not to store the object's contents because it is going to be deleted anyway. However, a few programs expect the nodes' longitude and latitude values for formal reason even if the only action they do is to delete these values. The option --fake-lonlat helps you to create such formally required replacement values.

Set the File Timestamp

Usually, OSM files have a file timestamp which allows you to determine the actuality of the file. During file conversions, osmconvert will keep this timestamp. Nevertheless, it can be adjusted by you. For example:

osmconvert hamburg.o5m --timestamp=2011-08-01T23:50:00Z -o=hamburg2.o5m

Retrieving Statistical Data

There are different ways to get meta data or statistical data of an OSM file. First, you can read a file's timestamp with the --out-timestamp option (the Z at the end stands for Zulu):

osmconvert file_with_timestamp.o5m --out-timestamp
2011-08-01T23:50:00Z
osmconvert file_without_timestamp.o5m --out-timestamp
(invalid timestamp)

Second, you can analyze the whole file and create a set of statistical data:

$ osmconvert germany.osm.pbf --out-statistics
timestamp min: 2005-07-05T02:14:17Z
timestamp max: 2011-07-31T19:59:46Z
lon min: -20.0712330
lon max: 21.1441799
lat min: 47.0830289
lat max: 59.9982830
nodes: 78138447
ways: 11342322
relations: 176024
node id min: 1
node id max: 1380816490
way id min: 92
way id max: 123952798
relation id min: 159
relation id max: 1693098

Writing CSV Files

To get character-separated lists you may define ".csv" as output format. This can be done by using the -o= option, e.g. -o=my_table.csv or by defining one of the csv related options: --out-csv , --csv= , --csv-headline , --csv-separator= .

The table will have three tab-separated columns: object type name, id, name. To change the column separator or to select a different set of columns please use the --csv-separator= resp. the --csv= option. Use the --help option to display further information on this topic. For example (in combination with option --all-to-nodes ):

osmconvert shops.osm --all-to-nodes --csv="@id @lon @lat amenity shop name" --csv-headline
@id               @lon        @lat        amenity  shop       name
21548298          11.6122123  48.6884848  shop     bakery     Miller
21552613          9.0651970   49.9979332  shop     butcher    Jaeger
1000000168276611  6.6058085   51.4556093  shop     drugstore  AllForYou

Columns will be separated by Tab characters of your system's default size. It is recommended to use -o=somefilename.csv if the data are to be written into a file.

Combining the functions

Most of the previously introduced functions can be combined. Thus, for example, you can update an .osm file and limit its region in one pass:

osmconvert day24.osm -B=p.poly c24_25.osc -o=day25.osm

Parallel Processing

osmconvert does not support parallel processing on its own. However you can use your operating system's capabilities and enter such instructions at the command line. This might also be useful in cases you want to process more than one .pbf file, since osmconvert is presently not able to read more than one .pbf file at the same time. For example:

osmconvert region1.pbf --out-o5m | osmconvert - region2.pbf -o=all.pbf

In this example the first process will read the .pbf file "region1.pbf" and output it o5m-formatted to standard output. The second process will read this data from standard input, merge it with the other regional file "region2.pbf", and write it to the file "all.pbf". You will have recognized the minus sign in the second osmconvert command: it advices the program to read data from standard input. The pipe operator "|" connects standard output of the first command to standard input of the second command.

You also can use more than one pipe by creating so-called named pipes. Unfortunately this might not work with Windows. On Linux, there is the mkfifo command to create pipes. These pipes can be addressed the same way files are. In this example, three processes will be created to merge three .pbf files:

mkfifo p1 p2
osmconvert a.pbf --out-o5m -o=p1 & osmconvert osmconvert b.pbf --out-o5m -o=p2 & osmconvert p1 p2 c.pbf -o=all.pbf

The ampersand operators will each create a background process for the command which stands left to it. The last command in the line remains in foreground and will collect the data the two background processes produce.

Alternately, many Unix shells provide a technique called Process Substitution, which essentially creates the named pipes for you automatically, no need to use mkfifo. An equivalent to the above command would be:

osmconvert <(osmconvert a.pbf --out-o5m) <(osmconvert b.pbf --out-o5m) c.pbf -o=all.pbf

Parameter File

The command line argument can get long if you specify complex operations. Please use a parameter file instead and refer to this file with --parameter-file=. For example:

osmconvert --parameter-file=my_parameters

File "my_parameters":

// verbose
-v
// input file
planet.o5m
// bounding box
-b=8.123,10.123,9.456,11.456
--complete-ways
// output file
-o=region.o5m

Empty lines are used to separate the parameters. Linefeeds within parameters will be converted to spaces. Lines which start with "// " are treated as comments and therefore ignored by the program.

Technical Details

Temporary Files

To perform certain operations osmconvert needs to create temporary files. These files are small – in comparison to the OSM files which are going to be processed. Their names each start with "osmconvert_tempfile" and end with numbers. You may change the left part of the file name, including the path. For example:

osmconvert germany.pbf -B=n.poly -t=/media/hd70/temp -o=nuernberg.o5m

Memory Management

There are three options to affect the program's memory management: --hash-memory=, --max-refs= and --max-objects=. Please refer to the detailed description shown by the help option:

osmconvert --help

Test Output

Sometimes it is nice to get some information about what the program is doing at the moment. You can activate the verbose mode by applying this option: -v. With -v=2 you will get even more detailed output, however it might be a bit confusing.

Benchmarks

Planet .pbf -> .o5m

A quick one, as Osmconvert was not alone accessing the disk and I did not think of running 'time':

16GB ram, 7200rpmdisk.

./osmconvert planet-latest.osm.pbf -o=planet-latest.o5m

Approx. 12 minutes.

Planet .osm -> .o5m

$ date
Sat Dec 29 14:36:19 PST 2012
$ ./osmconvert planet.osm -o=planet-121207.o5m
$ date
Sat Dec 29 18:34:16 PST 2012

$ ls -lah
...
-rw-------   1 user  staff    32G Dec 29 18:34 planet-121207.o5m
-rw-r--r--   1 user  staff   319G Dec  7 02:41 planet.osm
...

So.... about 4 hrs on a 16GB i7 with planet.osm on a USB 3 external hd. Turns out this was a defective USB3 drive...

Planet .osm -> clipped .osm

$time ./osmconvert planet-130123.osm -b=-144,20,-50,90 -o=na_subset.osm

real    54m59.172s
user    39m18.091s
sys     5m3.111s

On an 16 core server with 48GB ram, 12x2TB raid5. Performance limited by use of single cpu thread. Input planet.osm was ~360GB, output subset was ~112GB.

Planet .o5m -> Germany .o5m

Extracting Germany from the Planet file, using Germany polygon from Geofabrik:

./osmconvert planet.o5m -B=germany.poly -o=germany.o5m

Approx. 5 minutes on a computer with i7 cpu, using only 1 core.