Overpass API/versions

From OpenStreetMap Wiki
Jump to: navigation, search

Contents

Overpass API v0.7.51

Released: 2014-10-30

This version marks a very important milestone: for the first time, a complete feature has been contributed by a second developer, mmd. A big thank you to mmd.

Furthermore, this version fixes the bugs that were in the preivous version and makes attic data reliable. It is completed with some new and experimental features and some performance optimizations.

Database changes

The database remains binary compatible to its predecessor.

Interface changes

The probably biggest change is that now queries for regular expressions on keys are possible. The syntax currently allows only for the essential forms of combining a regular expression for a key with a regular expression for a value.

Another change is a new output format "csv". This allows to receive OSM data as a table. The format still needs fine tuning. For example, there is no canonical place for error messages. Escaping is not yet decided and so on. Please experiment with your favourite software such that we can in the end agreee on a format variant as interoperable as possible. Please feel free to leave commment as github issues.

Some day-to-day changes have improved the performance for queries on large areas in combination for rare tags. They are now almost as fast as a similar bounding box.

Other changes are bugfixing: the JSON output format now can also contain a global bounding box. Jsonp calls may contain dots in their function names. On queries with a global bounding box, empty filtering criteria on queries is possible and means there to take only the global bounding box as criterion. In attic printing, elements are now always in the correct order.

The statement dump has been completed. Also a big thank you to mmd for this work.

Other changes

A big thank you to lonvia for updating the Automake configuration. Newer versions has limited backwards compatibility. The adapted configuration should now work again.

Overpass API v0.7.50

Released: 2014-07-13


Database changes

In this version, the core of the database has again changed its format. This is obviously necessary to provide attic data. But also an extension to allow searches for keys only has required a change of the database format.

Hence, please rebuild your database if you update from version 0.7.4.


Summary for the impatient

For current data, this version is completely reliable. For attic data, I suggest to use rather the successor. This version has known flaws, and these issues have been solved in that version.

For Augmented Diffs, this version brings a new mode of operations, mostly because the former approach has intrinsic stability problems. The last bigger incident happened on 18th April this year. Currently, the new Augmented Diff mode can be tested, and the old one should be continued until end of July.

The old mode has been retired since then.


Current data

Queries on current data are such well-adopted and in widespread use that I will not break backward compatibility. Essentially only two features have been added:

Sparse queries are now faster. An example. That query would have taken hours with such a large bounding box in the past. Now it should run in less than 15 seconds. There is still room for improvement: It doesn't work yet that fast for regular expressions, but this is postponed to the next version.

And features like ways and relations can expose their geometry directly on the feature. This is triggered by adding the word "geom" to the "out" statement. This works in this version only for XML output. The next version includes support for JSON output of the geometry format.


Augmented diffs

The Augmented Diffs are redesigned to be always generated on the fly. This is because the Augmented Diffs have been piling up on the server to almost a terabyte of data and we are running out of space. A second advantage is that their generating does no longer block applying the diffs to the current database.

You can access them via e.g. (by changing the id to the number you actually need).

There are also advantages to the users:

Augmented diffs now carry a number that can be straightforward computed from the desired date. Number 1 starts at 2012-09-12T06:56:00Z, the effective license change date. And the date interval of Augmented Diff n is expressed in seconds since the epoch always

 (n - 1347432900 ) / 60
 

until one minute later. You can get the current Augmented Diff with the call

 http://overpass-api.de/api_0750/augmented_diff_status

which is deduced from

 http://overpass-api.de/api_0750/timestamp
 

giving the date of the current database state.

A second advantage is that filtering now makes sense. The full Overpass QL language can be used for filtering on Augmented Diffs, and it almost always makes the request faster. For that purpose take this request from and adapt it to your needs.

A third advantage is that you can requests arbitrary timeframes, not only minutes.

However, the price to pay is that these changes are not completely backwards compatible; I'll take the reservation that the format was experimental. The "info" elements are no longer available, in favour of the geometry features of ways and relations. This is the format more widely adopted, and offering the "info" variant would have taken too much time to implement again.


Attic data

This is a completely new kind of feature, and complements the "Historic dumps" of OSM data. I'll prefer the notion "attic data" like in version control conventions to avoid confusion with mapping of historical features.

You can run a query against the database state as it were at an arbitrary date in the past. Just put

 [date:"2014-06-02T20:00:00Z"];
 

in the front of your query.

In a similar way, you can get what has changed in the results by adding

 [diff:"2014-06-02T20:00:00Z"];
 

or

 [diff:"2014-06-02T20:00:00Z","2014-06-02T20:00:00Z"];
 

in front. The first form takes as second time implicitly the current time.

If you need in addition deletion dates, you can use

 [adiff:"2014-06-02T20:00:00Z"];
 

or

 [adiff:"2014-06-02T20:00:00Z","2014-06-02T20:00:00Z"];
 

In this case, you will get meta information on otherwise gone objects, either that they have been deleted, bearing "visible=false", or still exist but have gone out of scope of the request ("visible=true").


Overpass API v0.7.4

Released: 2013-07-30

The main change of this version is the acceleration of geographically bounded queries for ways (and indirectly also relations). This makes the beta popup feature of the main site sufficiently performant.

Database changes

In this version, the core of the database has changed its format to improve spatial queries for ways. So you need to discard and regenerate the complete database if you upgrade from an earlier version. It is expected that the next version again will change the database format, but that may take another three months.

Interface changes

A new statement called difference has been added. This allows to reduce the amount of data found by a query also in more complex cases. For simpler cases, like all elements with a tag but not a certain other tag I suggest to rather use negated conditionals because they are faster.

Queries can now be equipped with a global bounding box. Thus an arbitrary query can act as a filter for a full map query, e.g. in the JOSM plugin or for slippy map overlays. The global box is returned as bounds element in the answer, which fulfills an often expressed wish in a consistent way.

A dedicated map call now mimics exactly the behaviour of the main API map call. With the timestamp API call, the timestamp of the database can be queried. Finally, the interpreter API call now properly handles HTTP HEAD.

Additionally, a lot of bugs have been fixed. Most prominently, relations and ways are now really only returned as inside areas when a part of them is contained in the interior of the area. Also, the locale of the regular expressions has been corrected to UTF-8. Thus, a dot now matches a character regardless whether it is multibyte.

Overpass API v0.7.3

Released: 2013-03-11

This release mostly fixes the semantics of the area clause for relations. Please see details below. Along this, some straightforward extensions have been made to the query language.

Database changes

In this version, the core of the database is binary compatible to its predecessor. You need to discard and regenerate all area related derived data.

The area data structure has been reorganized to improve performance. A typical query time for a coord query is reduced from 1 second to around 0.2 seconds.

Interface changes

The most important change is an adjustment of the area statement for relations: A relation is now found by the area statement if and only if one of its members is contained in the interior of the area. This means that for an area all contained relations are found but not the boundary relation itself. In the previous version, also all attached relations are found and thus it was not possible to find exactly the relations that are subrelations of a given area.

The missing boundary relation to an area is compensated by using the new pivot statement: it finds the element that was the base for the area. This also allows to find the building described by a closed way in which a coordinate is situated.

In QL, any Unicode character can now be escaped by a sequence \u plus four digits.

Furthermore, in the recurse statement the link resolution can be limited to a chosen role.

Overpass API v0.7.2

Released: 2013-02-11

After the last version has been dedicated to the switch to 64-bit node ids, this version again add several features, makes existing ones faster and has less bugs than the previous version.

Database changes

This version is binary compatible to its predecessor.

Interface changes

This version again brings a couple of extensions of the query language: Both the area-query statement and the polygon statement now support ways and relations as target types. This is not only a convenient shortcut, but it is designed to allow printing a complete and proper street list of a region.

The has-kv statement in its regular expression variant now also allows to search case insensitively.

And the around statement now also accepts coordinates as starting point.

Furthermore, the CORS treatment is now more elaborate: the script return the CORS related headers now according to the requested headers by the server.

Other changes

The most impressive speed-up has been made for the area creation process. The turnaround time for areas has been reduced from about 24h to 4h while at the same time seven times more areas are created. This is one of the cornerstones to empower in the future full data popups on the osm.org site.

Also the around statement has been accelerated. This makes an intersection test possible: just call around with a radius of 0.

The source code now uses the macro "__APPLE__" to build also on Mac OS X.

And of course, a couple of bugs have been fixed. On multiple places, the escaping of special characters has been fixed. The popup output format now also prints areas. Also some bugs resulting in wrong output have been fixed.

Overpass API v0.7.1

Released: 2012-12-10

The version 0.7.1 is the first with 64-bit node ids.

Database changes

Due to the change to 64-bit node ids, the database of this version is not backwards compatible.

Interface changes

No major interface changes.

However, the fair use policy has been changed to reflect changed usage patterns. Please do only one query after another from the same host, not multiple queries in parallel. This ensures that other users still have a chance to use the server. If you send multiple queries at the same time, Overpass API will execute them one after another, and if the queue gets too long (if a query is for more than 15 seconds in the queue), the query is rejected with a HTTP code 429.

Overpass API v0.7

Released: 2012-11-04

The version 0.7 finally is complete. A couple of changes have made it in this last version before the change to 64-bit node ids.

Database changes

This version is binary compatible to its predecessor.

Interface changes

Overpass API now allows to take an almost arbitrary polygon instead of just bounding boxes as query boundary. This type of boundary is currently only supported for queries of type node.

Also the area features have been extended at large: Areas can be searched for with tag based criteria, like all other data types. Moreover, areas can be found from nodes and vice versa. Or less technical: you could almost do Geocoding with a single Overpass QL query. And all these features have got a Overpass QL syntax; area features no longer require the XML syntax.

The Augmented Diffs have got a new format. In particular, for deleted items are both the old meta data and the deletion date included. Other changes are geared to simplify evaluation by JavaScript. As a useful extra, a tool to clip Augmented Diffs to bounding boxes has been added. This has been developed to large extends during the last Karlsruhe hack weekend.

The handling of HTTP headers has changed: While the old approach was to avoid the subject completely, now we care for all the issues around HTTP CORS to allow JavaScript access from the browser.

The XAPI compatbility layer now allows arbitrary "or" constructions for tag values, like what XAPI always should have been able to.

The hard coded data license for output has been changed from CC-BY-SA to ODbL.

Overpass API v0.6.99

Released: 2012-08-31

This version is the first licensed with the GNU Affero GPL. This version introduces Augmented Diffs, but it is likely that their format will get some fine-tuning.

Database changes

This version is binary compatible to its predecessor.

Augmented Diffs

The Augmented Diffs fill the gap that the minute diffs leave: They contain also the necessary data to roll back in time, and to reconstruct the geometry of a way even the nodes linked in the way. This feature works stable, but the format is still experimental. If the consumers or potential consumers of the augmented diffs need slightly different data, the format will be changed at a later version.

Interface changes

The most important extension of the interface is the new output format popup. This allows to pre-format data in a HTML style such that in can be directly pasted into a popup showing additional information about a location. Example pages are this rather low-level and this more human-readable. Just click on the map.

If a query is rejected due to too much resorce consumption, this is now answered with HTTP status code 504 instead of 200. This helps certain client to identify this as a failed request.

Queries for keys only or for regular expressions in small bounding boxes have been accelerated. The enhanced tests also made it possible to track down and remove several bugs around the query statement.

Other changes

The IPC in the dispatcher again received a lot of attention. This includes a couple of bugfixes in corner cases and a throughput improved by the order of a magnitude.

Furthermore, running the softwar on MacOS revealed a couple of both weaknesses in the software and surprising behaviour of MacOS. After all, Overpass API now also runs on MacOS. A big thank you to Malcolm Herring for figuring out all the dirty details.

Some non-standard C++ has been rectified. If you find any additional non-standard C++, please report it.

License

The project changed its license to the GNU Affero GPL, because it is a typical server application.

Overpass API v0.6.98

Released: 2012-05-01

Otherwise than expected, quite a lot of changes were made since version 0.6.97, and there is still a lot of polishing necessary for version 0.7. Thus, this additional intermediate version has been released.


Database changes

The database format has been changed; the ways and relations have got more precise spatial coverage data. This improves the database response speed, but makes the database scheme incompatible to all earlier versions.


Interface changes

Now you can set up your own instance of Overpass API by just downloading an existing instance, of compressed size 15 GB (or 25 GB with meta data): This only takes 4 to 8 hours, and then you have already ready-made database when others still download the planet file. See the reworked installation instructions for details.

For memberships in ways and relations, some extra recurse operators are available:

  • ">" and ">>" collect the ways and nodes that are members of the given relations or ways.
  • "<" and "<<" collect the ways and relations that have the given nodes, ways or relatios as members.

The exact semantics are documented in the language guide.

The negation clauses have changed their meaning, as the meaning in version 0.6.97 caused more confusion than benefit. The new semantics are also documented in the language guide.


Other changes

As mentioned in the database section, the query internals have been changed to improve the performance of membership link resolution and bboxed queries for ways and relations. Also, the implementation of the dispatcher has changed from shared memory to Unix Domain Sockets. This should fix the occasional timeouts of queries. The changes are documented in the updated technical overview.

Overpass API v0.6.97

Released: 2012-03-12

This version combines the Permanent ID feature with some bugfixes. On the schedule for version 0.7 now are mostly some optimizations left.


Database changes

This version is binary compatible to its predecessor. Version 0.7 is likely to contain some extra tables for data lookup acceleration.


Interface changes

New in this version is the Permanent ID facility. This makes it possible to set permanent links from e.g. the wiki to specific elements in the OSM database by their properties (e.g. the motorway with ref "A 555"). A detailed description is in the Permanent ID documentation.


Other changes

The timestamp is now printed without the escaping backslashes from the .state.txt.


Overpass API v0.6.96

Released: 2012-02-10

This version gives a first preview to the new, more concise query language for Overpass API, OverpassQL. This is an additional query language, the XML syntax will also be kept.


Database changes

This version is binary compatible to its predecessor. It is likely to be compatible to the expected version 0.7.


Interface changes

The most important change is the introduction of OverpassQL.

The query statement can now be used almost orthogonal: The statements around, recurse and id-query are now all possible substatements of query.

The around statement can now be used also for ways and relations. In detail: If the reference set for which around is called contains ways, elements are also searched in the neighborhood of these ways. The same holds for relations, where the location of a relation is defined to be the union of all locations of its way and node members. The type of elements to be found by around is the type of the embracing query statement.

Also, the has-kv statement has got several extensions: First, the value to search for can now be described by a regular expression. Second, a value can be negated; this finds all elems with this key but with a value different from the provided. For example "oneway"!="no" finds any element that has a tag with key "oneway" but not the value "no".

As an alternative to XML, now also JSON and its cousin JSONP can be used as output formats.


Other changes

  • The script /api/ping returns as a simple self test of the server several small queries in HTML.
  • The download script fetch_osc has been changed to check the integrity of the downloaded files.
  • A major bug in make-area has been fixed.


Overpass API v0.6.95

Released: 2011-12-23

This is once again an intermediate release of the features on the way to version 0.7 that are implemented now.


Database changes

The version is binary compatible to its predecessor. It is likely to be compatible to the expected version 0.7.


Interface changes

The query statement has got two further extensions towards its pursuited orthogonality: The bbox-query statement can now be used also for ways and relations. This needs some subtleties to be clarified: A way is contained in a bounding box if one segment intersects the bounding box even if no node of the way is contained in the bounding box. By contrast, a relation is contained in a bounding box if one of its members of type way or node is contained in the bounding box.

The around statement searches, as before, only for nodes around other nodes. Ways and relations are neither considered in the input nor included in the output.

The timeout has been extended from the XML syntax to the XAPI layer. Add a [@timeout=seconds] to the request to change the default timeout of 180 seconds.


Other changes

Several bugs have been fixed in the rather long time period of two and a half months. Some important are:

  • The Xapi compability layer did not properly delete its temporary files and caused an overflow of file entries in the /tmp directory.
  • The bbox-query had been very slow and memory consuming for large bounding boxes.
  • To prevent deadlocks from reading processes that have been killed by the web server, the dispatcher now offers more maintenance functionality.
  • A basic auto restart mechanism has been implemented to enable an automatic recovery and restart in case of system failure. By its very nature, it has not been tested much.

Furthermore, several components have been refactored to be faster at least in the use cases common on the overpass-api.de server:

  • The Xapi compability layer produces more concise queries.
  • The statements query, bbox-query, and recurse
  • The statement framework in general.


Overpass API v0.6.94

Released: 2011-10-05

This version primarily fixes a larger number of bugs. However, the enhancements scheduled for version 0.7 that are already implemented have been released too.


Database changes

The version is binary compatible to its predecessor. It is likely to be compatible to the expected version 0.7.

However, this version fixes a bug in the database engine. It occured only under rather obscure conditions: inserted items at the end might have been lost when the data ends with two blocks of more than 100'000 items with the same index. It did not occur on the public database. But the bug might have affected other installations (with a small probability), so I suggest all users to re-import at the earliest opportunity and to use this version. I'm sorry for the inconvenience. More excessive testing showed no further bugs of that kind.

Interface changes

Importing data into JOSM has still been painful with version 0.6.93. The current version changes the header to comply with the expectations of JOSM, but you can only load files with meta data into JOSM. I'll try that part of the problem in JOSM. Please tell me if other tools don't like the data format, I'll work hard to find a solution.

User names are now properly escaped. Before that, tools trying to load data from users with characters >, <, &, and " failed with more or less understandable alerts by their XML parser.

The query statement has undergone a lot of refurbishment, to prepare its more central role in version 0.7. This means that you can now freely combine one or multiple has-kv, item, and user statements. You can combine them also freely with area, around, and bbox statements if you query for nodes. These statements don't work yet for ways and relations. The newer statement can be combined with all other statements but cannot be used alone.


Other changes

A bug in query has been fixed: has-kv statements to nonexistant tags had no effect at all. They now clear the result instead, as expected.

The portability has been further improved. If no /proc filesystem is present, the old version may have not properly insulated transactions. This is now fixed as good as possible with an ample timeout.

Further bugfixes cover memory holes or execssive memory consumption (taking more than 4 GB RAM for no apparent reason).


Overpass API v0.6.93

Released: 2011-08-30

Updated version (interacts better with JOSM): 2011-09-01


Database changes

This version is not binary compatible to its predecessor. It is likely to be compatible to the expected version 0.7.

The database now can optionally contain meta data, see below. It roughly doubles the size of the database. If you use the database without meta data, it is still compatible to v0.6.92.


Interface changes

Overpass API now also provides the OSM meta data (timestamp, version, changeset id, user name, and user id). This allows to use the data directly in for example JOSM, including re-upload.

The print statement now allows an attribute limit to limit the size of the response.

The meta data give rise to the following special keys inside the XAPI compability layer: @meta, @user, @uid, and @newer.

The corresponding tags in the scripting language are: print mode="meta", user name="...", user uid="...", and newer than="..."


Other changes

Other features are a hardening of the software against file errors. The reworked planet import works substantially faster.


Overpass API v0.6.92

Released: 2011-07-25


Database changes

This version is neither binary compatible to its predecessor nor to its successor. The difference in the database format comes from an improved indexing scheme for ways and relations.

Depeding on the query, this new indexing scheme makes queries up to three times faster.


Interface changes

A completely new statement has been added, around. It can be used on itself or as part of query. Also, it is now possible to use item as a substatement of query.

A new compability layer for queries in the XAPI syntax has been added.


Other changes

Some remaining bugs in the transactionality management have been fixed. In particular, it is no longer possible to leave file blocks blocked forever by killing a running query.


OSM3S v0.6.91

Released: 2011-07-05


Database changes

This version is binary compatible to OSM3S v0.6.90.


Interface changes

From this version on, OSM3S is re-enabled to handle areas, i.e. allow the statements area-query and coord-query. On each call to osm3s_query or web_query, this extension is only invoked if one of those statements is contained in the input query.

The areas are created by the ruleset specified as an osm script in the rules subdirectory. As a rule editing mechanism is currently not implemented, this rule is not edible over the web. To perform the area recomputation from updates, a new shell script rules_loop.sh has been created to run as a permanent background process.

The front-end web_query has moved to cgi-bin and renamed interpreter. This goes in-hand with an essential hardening of interpreter to keep it safe when open to any query over the internet: long running queries will be aborted at their time-out, large queries when exceeding their memory quota.

A bug in query had been spotted: only one has-kv clause with a wildcard value has been considered and all subsequent clauses have been ignored. This is now fixed.


Other changes

  • A new script pull_and_publish.sh simplifies the installation and startup.
  • The programs give more often error information when called with invalid parameter values.
  • The dispatcher lock files are handled atomically.
  • The programs ignore the user's umask to reduce file permission conflicts when accessing later with the user www-data.


OSM3S v0.6.90

Released: 2011-06-07


Database changes

This version is not binary compatible to any previous and not expected to be binary compatible to its successor (yet it is binary compatible to its successor). The database format will be currently reworked to fit the needs of the envisioned version 0.7.


Interface changes

From this version on, OSM3S behaves transactional: With a running dispatcher, multiple reading instances and one writing instances can work in parallel. Thus, only one database instance needs to be updated which doubles the update speed.

The script apply_osc_to_db and the daemon-like program dispatcher now suffice to keep a minutely updated database.


Other changes

The introduction of transactionality made some other changes necessary:

  • The Random_File file format is nor also organized in blocks. This saves some space in case of sparse indices.
  • The logging got a new format but materially tells the same information.
  • The Scripts for pt_diagrams have been reworked and relocated.


OSM3S v0.6.1

Released: 2011-04-06


Database changes

This version is binary compatible to the previous version 0.6.


Interface changes

The log levels of osm3s_query have been reworked. I hope the default log level now contains sufficient comprehensible and only comprehensible messages. The progress information of update_database shall now be rather self-explanatory than diagnostic.


Other changes

With the advent of a comprehensive automated testbed, a couple of bugs have been revealed and fixed:

  • The print statement doesn't throw anymore an error on absent optional parts of the database.
  • update_database can now properly apply diff files. It didn't update the indexes of ways and relations when only the underlying nodes have moved before.
  • bbox_query doesn't anymore crash on very large bounding boxes.
  • Relations that have only other relations as members have been accidently unchangeable in the previous version.
  • A possible buffer overflow in the database backend has been fixed.
  • Two other, rather arcane bugs in the database fixed.


OSM3S v0.6

Released: 2011-03-11

This was the first version mature enough to be released to the public.