Talk:Osmfilter
Error filtering europe dump
I tried o5mfilter to filter the geofabric europe dump of Jun 10th but I got an error.
I called
osmconvert.exe europe.osm.pbf --out-o5m >europe.o5m o5mfilter.exe europe.o5m --keep-nodes= --keep-ways-relations="boundary=administrative =postal_code" >europe-boundaries.osm
and got the error
o5mfilter Error: could not rewind input file to position 1.
The europe.o5m file is 10.833.622.274 bytes. I am using Win 7 and the o5mfilter and osmconvert download of today (Jun 10th).
Is there a file size limit or is this a known problem? --WanMil 21:18, 10 June 2011 (BST)
- Hi WanMil, thanks for your report! Which Windows version are you using? There were problems reported with Windows 64 bit, but when I checked this with Win7 (64 bit) everything worked fine. You are right, there seems something to go wrong with large files ( >4 GB). The rewind problem occurs when lseek() does not work. Maybe the lseek64() wasn't used. I will check o5mfilter tomorrow. --Marqqs 21:45, 10 June 2011 (BST)
- I possibly found the reason: Win7 could have used lseek() instead of lseek64(). Please try version 0.1T. --Marqqs 22:46, 10 June 2011 (BST)
- Thanks, it's working now. The speed is really great for such filter stuff.
- I possibly found the reason: Win7 could have used lseek() instead of lseek64(). Please try version 0.1T. --Marqqs 22:46, 10 June 2011 (BST)
Bounding Box
Is it possible to preserve the bounding box information?
<bound box="49.10868,6.35017,49.64072,7.40979" origin="http://www.openstreetmap.org/api/0.6"/>
--WanMil 11:18, 11 June 2011 (BST)
- Do you need the "origin" tag? I think about recreating the bound box information from scratch: the filtered data may have a smaller bound box than the source file. --Marqqs 13:00, 11 June 2011 (BST)
- On the other hand, I'm not sure if this can be accomplished because o5mfilter writes the header before it has inspected every node. What about the syntax? Usually the bounding box is described as "<bounds minlat=...". --Marqqs 13:09, 11 June 2011 (BST)
- I copied the XML node from an osmosis generated file. For me both variants (bound and bounds) are ok so feel free to decide yourself. I would prefer to recreate the bounds information only if a special parameter is set (e.g. --recreate-bounds). Otherwise the info should just be copied. --WanMil 13:38, 11 June 2011 (BST)
- Please try version 0.1V. I added a bounding box syntax to o5m format description and extended the programs o5mfilter and osmconvert accordingly. Presently there is no "--recreate-bounds" option; the coordinates are just copied. --Marqqs 23:18, 11 June 2011 (BST)
- I copied the XML node from an osmosis generated file. For me both variants (bound and bounds) are ok so feel free to decide yourself. I would prefer to recreate the bounds information only if a special parameter is set (e.g. --recreate-bounds). Otherwise the info should just be copied. --WanMil 13:38, 11 June 2011 (BST)
Drop specific dependencies
Is it possible to drop some specific dependencies only? I think about a relation which has subrelations. I want to get all ways of this relation but I am not interested in the referenced relations.
Example:
- Spain: http://www.openstreetmap.org/browse/relation/1311341
- has some linked relations with role "subarea" (http://www.openstreetmap.org/browse/relation/349041, http://www.openstreetmap.org/browse/relation/349049 etc).
I want to include all dependent ways but not the "subarea" relations. --WanMil 11:16, 18 June 2011 (BST)
- I'm not sure, but I think you could accomplish this with
--keep="admin_level=2" --drop="admin_level=3 =4 =5 =6"
- --Marqqs 20:33, 18 June 2011 (BST)
don't work in debian 4.0
osmfilter.c cc say a lot of errors trying to compile.
./osmfilter32 (binary)
./osmfilter32: /lib/libc.so.6: version `GLIBC_2.4' not found (required by ./osmfilter32)
regards.
Add --fake-timestamp
It would be nice to have an option --fake-timestamp which works analogue to --fake-version. -- Simon04 12:30, 29 August 2011 (BST)
- Hi Simon, interesting idea. But please forgive me the question: For what purpose do you need this? And what date/time should the faked timestamp have? --Marqqs 20:31, 30 August 2011 (BST)
Write error
When I execute Osmfilter (Windows 7) I get a "write error". What is wrong?--Kozuch 15:49, 2 October 2011 (BST)
Numeric Comparison
Is it also possible to implement a numeric comparison (e.g. population>1000) --WalterSchloegl 15:31, 12 November 2011 (UTC)
- Hi! Good suggestion. Meanwhile I'm working on implementing alphabetic and numeric comparison. Furthermore, I'd like the program to be able to deal with Boolean terms and brackets. I'm not sure when this will be finished, but I will keep you up-to-date. --Marqqs 18:59, 3 December 2011 (UTC)
Role filters
Is it possible to filter relation members based on their roles? As far as I understand, current filters only work on tags. I would like to filter relations (e.g. you have a multiploygon and want to filter all members with the role "inner"). --Eiger 22:00, 16 December 2011 (UTC)
- Hi, unfortunately this cannot be done with osmfilter at present. However, this seems to be a very special use case. Maybe you could use grep for this purpose? --Marqqs 11:17, 17 December 2011 (UTC)
- Well, I'm not sure grep could help (my example did not really reflect what I want to do). I have an .osm file covering a whole region, and I want to keep only "boundary+admin_level=n" relations which have no member with the role "admin_centre". I could imagine doing this in two steps: 1. keep only "boundary+admin_level=n" relations (this can be done with osmfilter) 2. drop all relations which have a member with the role "admin_centre" (which doesn't work with osmfilter). I will write my own filter then. Anyway, I don't think that roles are a special use case: they belong to the definition of a relation, just as tags. It would be nice to support them in osmfilter.--Eiger 12:49, 17 December 2011 (UTC)
- Refroles are a bit different from tags because the can exist multiple times within the same object. This would require a different algorithm. But – you're right – it would be really nice to have this kind of filter function in osmfilter. However, there might be an interim solution for you: Take the .osm XML file and add a temporal tag for the relations in question. This could be done with sed. Example (not tested):
- sed "s/role=\"admin_centre\"\/>/role=\"admin_centre\"\/>\n<tag k=\"ac_role\" v=\"yes\"\/>/"
- After this, you can filter: --drop=ac_role=yes --Marqqs 13:19, 17 December 2011 (UTC)
- Good idea, I'll try that. Thanks. --Eiger 13:27, 17 December 2011 (UTC)
NOT-EQUAL doesn't seem to work
I tried these 2 variants and the resulting file is empty (176 Bytes, no osm-objects)
osmfilter $FILENAME.o5m --keep="( railway=station or railway=halt ) and disused!=yes" >${FILENAME}_stations.osm
osmfilter $FILENAME.o5m --keep="railway=station and disused!=yes or railway=halt and disused!=yes" >${FILENAME}_stations.osm
am I using a wrong syntax? --Dieterdreist 14:35, 30 January 2012 (UTC)
- Hi! It's neither a program error nor did you do anything wrong. The program just expects the key "disused" to appear in the OSM object if this key is used within an equation. Your "disused!=yes" is such an equation. It will result in the value false whenever no "disused" key is present. Usually this is not what a user would expect. For this reason I've changed osmfilter that way that "!=" comparisons will result true if there is no such key in the inspected OSM object. Please download and try version 1.2M. --Marqqs 20:22, 3 February 2012 (UTC)
Nodes not belonging to ways or relations
Hi, is it possible to filter all the nodes that don't belong to ways or relations? Groppo 20:16, 14 June 2012 (BST)
- Hi! As far as I know, this is not possible with osmfilter. However you can do the opposite and get all the nodes that do belong to other objects. The resulting file, if stored in XML, can be changed into a .osc file by adding "<delete>" and "</delete>". Then this .osc file can be used to delete all the andesired nodes inthe original file (use osmconvert for this). --Marqqs 11:45, 20 June 2012 (BST)
- Thank you for the answer. --Groppo 21:56, 20 June 2012 (BST)
Install / compile warning on OSX 10.8.2
Following the instructions at Osmfilter#Download:
$ wget -O - http://m.m.i24.cc/osmfilter.c |cc -x c - -O3 -o osmfilter
--2012-12-29 15:11:53-- http://m.m.i24.cc/osmfilter.c
Resolving m.m.i24.cc... 80.67.17.148, 2a00:1158:0:300:432f::1
Connecting to m.m.i24.cc|80.67.17.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 213497 (208K) [text/plain]
Saving to: ‘STDOUT’
100%[===========================================================================================>] 213,497 154KB/s in 1.4s
2012-12-29 15:11:55 (154 KB/s) - written to stdout [213497/213497]
<stdin>:5166:27: warning: too many arguments in call to 'read_close'
read_close(oo__ifp->ri);
~~~~~~~~~~ ^
1 warning generated.
$
Is there a way to avoid this? Is it a problem? Jeffmeyer 23:18, 29 December 2012 (UTC)
- Hello! Don't worry, this bug does not do any harm. However it's wrong programming, of course. It will be fixed by the next version. Thank you for reporting this issue! --Marqqs 13:28, 30 December 2012 (UTC)
--all-to-nodes behaviour
Hi Marqqs. I want to use this option (--all-to-nodes) but with little differences in its behaviour:
First: at now for not closed ways osmfiletr tooks first point, is it possible to take n/2'th point? Or first, last and n/2'th point? I writing geocode engine, and import streets with this option. After tahat I search for streets inside city polygon, and offtenly last or first point in way lay outside city border.
Second: same behaviour for closed ways: at now, to find center of closed way, osmfilter needs to find all way points and calculate mean value, but sometimes it's not necessary. If closed way is small (like building for ex.) - any point from way lay rather closer to center and calculating of real figure center is not neaded.
I try to do it my self, but I'am not so skilled in C. Dkiselev (talk) 13:49, 12 March 2013 (UTC)
Force dependencies
Is there a possibility to force every dependency to be in the output? I often got relations where members (mostly ways and other relations) are skipped.
Nick3331 (talk) 15:26, 24 May 2013 (UTC)
- Hi Nick, this should not happen. If a relation meets filter criteria, all hierarchical subordinated elements should be included as well. Can you give an example? --Marqqs (talk) 08:48, 16 June 2013 (UTC)
- I'm seeing a similar issue as well. This file (12 MB) is filtered from a daily updated planet file using the commands documented here. There's something going on with way 180,296,516 that prevents the data for it from being extracted correctly. To make things more manageable, I further extracted the data using a bounding box:
./osmconvert pipelines.o5m -b=-65.75,-5.2,-59.8,-2.75 -o=urucupipes2.osm
- What I see in the urucupipes2.osm file (when browsing the raw XML) is that not all the nodes associated with the way are actually present in the data, and this causes further issues with other programs that then read in the data. When using JOSM, some of the ways are rendered, but some are just represented as a set of nodes, while QGIS doesn't render anything. I've also tried getting all data from the planet file within the same bounding box without filtering by tags, and I still see the same issue with the ways not being included:
./osmconvert planet.o5m -b=-65.75,-5.2,-59.8,-2.75 --complete-ways -o=allData_bbox.osm
- When I use JOSM to download the latest map data directly from the OSM server, then everything shows up correctly. --ChrisDavis (talk) 09:54, 8 June 2014 (UTC)
- Hi Marqqs,
- Have you had any insight yet as to what is going on here? Im trying to download various slices from the OSM database from the daily updated planet file and commands linked to above and this bug is causing them all to have significant chunks missing when I process them in QGIS.
- Thanks, --Martin2014 (talk) 08:06, 17 September 2014 (UTC)
- This seems to be related to the discussion here. In the API documentation, there is a way/full call, but I'm not sure if osmfilter currently supports such a request. --ChrisDavis (talk) 10:54, 17 September 2014 (UTC)
- It would be really useful to me if something equivalent to the API's way/full call were included in the next version of Osmfilter.
- Thanks, --Martin2014 (talk) 11:47, 18 September 2014 (UTC)
- I've looked into this further, and I don't think this is a problem with osmfilter. I was updating a planet.o5m file daily using osmupdate, and it seems that some of the dependencies got lost. Downloading a fresh planet.o5m file fixed the issues. --ChrisDavis (talk) 07:34, 21 October 2014 (UTC)
@version
It would be really nice if you could filter by version as well, like you can by @user, @id, @uid. Martijn van Exel (talk) 17:27, 11 May 2014 (UTC)
Update: added this functionality to my version of osmfilter, not sure if this is solid, my C skills are very limited:
--- osmfilter.c
+++ osmfilter.c
@@ -5701,7 +5701,7 @@ return 18;
// simply add them as key/val tags;
keyf= keye; valf= vale;
if(fil_filterheader) {
- static char ids[30],uids[30];
+ static char ids[30],uids[30],vers[30];
*keyf++= "@id";
*valf++= int64toa(id,ids);
@@ -5709,6 +5709,8 @@ return 18;
*valf++= int32toa(hisuid,uids);
*keyf++= "@user";
*valf++= hisuser;
+ *keyf++= "@ver";
+ *valf++= int32toa(hisver,vers);
}
/* care about filtering, dependent on the filterstage */ {
- Hi Martijn, excellent! Please tell me if it works; I would like to include it into osmfilter master. For what purpose did you need to filter for specific versions? --Marqqs (talk) 09:31, 14 May 2014 (UTC)
Search/replace?
I realise it's a bit outside the scope of osmfilter as it stands, but have you considered adding a simple key/value search and replace function - so, for example, all "highway=trunk" could be replaced with "highway=main"?
A lot of OSM data is country-specific - for example, highway=trunk allows bikes in the UK, but not France. Some applications can cope with this fine - for example, with a PostGIS database, you can obviously do a spatial query to set bicycle=no for all highway=trunk within a certain polygon - but others can't.
My particular use case is for OSRM. I create a .pbf for Western Europe by taking Geofabrik country extracts, filtering and merging them with osmconvert and osmfilter. But since OSRM's Lua way_function doesn't know the geometry of a way, there's no way of finding out whether a trunk road allows bikes or not. If I could run osmfilter over the France extract first to change "highway=trunk" to "highway=trunk_no_bikes", which OSRM's way_function could then read, that would fix it nicely.
Thanks!
--Richard (talk) 09:38, 3 July 2014 (UTC)
FIlter and keep just the ways that dont have any tags
It is possible to load and see just the way that are added in OSM, but they do not have any tags ?
Unclear use of the keyword "all"
The role of the "all" keyword in the various "--keep..." parameters is not clear to me.
On one hand, the examples for Keep specific object type, do not use the "all" keyword for "--keep-nodes=".
On the other hand, the examples for Keep only specific Tags and for "--keep-relations" in a Parameter File use it.
--Zstadler (talk) 11:06, 17 August 2017 (UTC)
Is there a way to drop all tags?
osmfilter input.osm --keep-tags="all specific_tag=a" --drop-tags="specific_tag=a" -o=output.osm hack works, but I would prefer something less convoluted. Mateusz Konieczny (talk) 12:53, 15 October 2019 (UTC)