Osmfilter version 0

From OpenStreetMap Wiki
Jump to: navigation, search
Available languages — Osmfilter version 0
Afrikaans Alemannisch aragonés asturianu azərbaycanca Bahasa Indonesia Bahasa Melayu Bân-lâm-gú Basa Jawa Baso Minangkabau bosanski brezhoneg català čeština dansk Deutsch eesti English español Esperanto estremeñu euskara français Frysk Gaeilge Gàidhlig galego Hausa hrvatski Igbo interlingua Interlingue isiXhosa isiZulu íslenska italiano Kiswahili Kreyòl ayisyen kréyòl gwadloupéyen Kurdî latviešu Lëtzebuergesch lietuvių magyar Malagasy Malti Nederlands Nedersaksies norsk bokmål norsk nynorsk occitan Oromoo oʻzbekcha/ўзбекча Plattdüütsch polski português português do Brasil română shqip slovenčina slovenščina Soomaaliga suomi svenska Tiếng Việt Türkçe Vahcuengh vèneto Wolof Yorùbá Zazaki српски / srpski беларуская български қазақша македонски монгол русский тоҷикӣ українська Ελληνικά Հայերեն ქართული नेपाली मराठी हिन्दी অসমীয়া বাংলা ਪੰਜਾਬੀ ગુજરાતી ଓଡ଼ିଆ தமிழ் తెలుగు ಕನ್ನಡ മലയാളം සිංහල ไทย မြန်မာဘာသာ ລາວ ភាសាខ្មែរ ⵜⴰⵎⴰⵣⵉⵖⵜ አማርኛ 한국어 日本語 中文(简体)‎ 吴语 粵語 中文(繁體)‎ ייִדיש עברית اردو العربية پښتو سنڌي فارسی ދިވެހިބަސް

osmfilter version 0 is a tool to provide an easy way of shrinking a .osm XML file in situations where you need only a few tags. It has proven useful to filter the OSM XML file before you import it into a PostgreSQL database with osm2pgsql. Although the result will be the same, you can speed-up the import process significantly, in particular because osmfilter is written in C and quite fast.

Note that there is a newer filter program (osmfilter version 1) which offers some more options and is easier to use if object dependencies are to be cared about.


These Downloads are available:

(As usual: There is no warranty, to the extent permitted by law.)

Program Description

This program operates as filter for OSM XML data. Only sections containing certain tags will be copied from standard input to standard output. Use the calling line parameter -k to determine which sections you want to have in standard output.
For example:

-k"key1=val1 key2=val2 key3=val3"
-k"amenity=restaurant =bar =pub =cafe =fast_food =food_court =nightclub"
-K/ -k"description=something with blanks/name=New York"

Limitations: the maximum number of key/value pairs is 1000, the maximum length of keys and/or values is 100. The -t option invokes a test mode which prints a list of accepted search strings to standard output.

To suppress certain records, please use the -d option. For example:

 -d"highway=path =footway =cycleway railway=rail"

All objects containing at least one of the mentioned values will be dropped, regardless of their being part of a relation which is not dropped. I.e., key/val pairs in the -d parameter overrule the pairs which have been defined in the -k parameter.

Considering Dependencies

To get dependent elements, e.g. nodes of a selected way or ways of a selected relation, you need to feed the input OSM XML file more than once. You need to do this at least 3 times to get the nodes of a way which is referred to by a relation.
If you want to ensure that relations which are referred by other relations are also processed correctly, you must input the file a 4th time. If there are more than one inter-relational hierarchies to be considered, you will need to do this a 5th or 6th time.

If you feed the input file into an osmfilter more than once, you must tell the program the exact beginning and ending of the pre-processing sequence. For example:

cat lim a.osm a.osm a.osm a.osm lim a.osm | ./osmfilter -k"lit=yes" >new.osm

where 'lim' is a file containing this sequence as a delimiter:


If you have a compressed input file, you can use bzcat instead of. cat. If this is the case, be sure to have compressed the 'lim' file as well.

To speed-up the filter process, the program uses some main memory for a hash table. By default, it uses 320 MiB for storing a flag for every possible node, 60 for the way flags, and 20 relation flags.
Every byte holds the flag for 8 ID numbers, i.e., in 320 MiB the program can store 2684 million flags. As there are less than 1000 million IDs for nodes at present (Oct 2010), 120 MiB would suffice. So, for example, you can decrease the hash sizes to e.g. 130, 12 and 2 MiB using this option:


But keep in mind that the OSM database is continuously expanding. For this reason the program-own default value is higher than shown in the example, and it may be appropriate to increase it in the future. If you do not want to bother with the details, you can enter the amount of memory as a sum, and the program will divide it by itself. For example:


These 1000 MiB will be split in three parts: 800 for nodes, 150 for ways, and 50 for relations.

Because we are taking hashes, it is not necessary to provide all the suggested memory; the program will operate with less hash memory too. But, in this case, the filter will be less effective, i.e., some nodes and some ways will be left in the output file although they should have been excluded.
The maximum value the program accepts for the hash size is 4000 MiB; If you exceed the maximum amount of memory available on your system, the program will try to reduce this amount and display a warning message.

Optimizing the Performance

As there are no nodes which refer to other objects, preprocessing does not need the node section of the OSM XML file. Nearly the same applies to ways, so the ways are needed only once in preprocessing - in the last run.
If you want to enhance performance, you should take pre-filtering the OSM XML file into consideration. Pre-filtering can be done using the drop option. For example:

cat a.osm | ./osmfilter --drop-nodes >wr.osm
cat wr.osm | ./osmfilter --drop-ways >r.osm
cat lim r.osm r.osm wr.osm lim a.osm | ./osmfilter -k"lit=yes" >new.osm

If you are using pre-filtering, there will be no other filtering, i.e., the parameter -k will be ignored.

Example of Use

The project OpenGastroMap.de uses osmfilter to speed-up the database import. That makes it possible to run the application on a small virtual Internet server. Here are the details: OpenGastroMap/install#Tool_osmfilter.