Overpass API/Augmented Diffs

From OpenStreetMap Wiki
Jump to: navigation, search
Available languages
English русский

The augmented diffs extend the ordinary minute diffs. They make it possible to keep even specialized databases up to date minutely. They are based on an idea by Matt Amos, refreshed by Frederik in a talk in German.

This feature is still experimental. This means on the one hand that the service may change or be suspended at any time, on the other hand that suggestions are welcome as the format and content of the augmented diffs will be improved on good suggestions.

The augmented diffs can be downloaded from http://overpass-api.de/augmented_diffs/


Overview

The augmented diffs contain three kinds of data

  • The new data that comprises the usual minute diffs
  • The old data that is either replaced by new data or explicitly deleted
  • Context data: ways and relations that have modified objects as members and for all contained ways and relations also all their nodes and members

This allows in particular the following two applications

  • Proper minute updates on datasets restricted by a bounding box
  • A wayback mechanism: You can start with a planet file or other database and walk back instead of forward through time

While the use of the former is immediately clear, the second needs some explaining words: A couple of examples from the talks above like the counting application need to know what is deleted to decrease the counter.

Contained data

first version

Each diff is structured in three sections, one for each type of element. Within a section, the elements are ordered by id. If an element appears in multiple versions, these appearances are ordered by version. Each element is wrapped in erase, keep, or insert, depending on whether the element is added (insert only), deleted (erase only), changed (delete with old version and insert with new version), or not changed (keep only).

In the preamble, the file contains the end date of the diff.

<?xml version="1.0" encoding="UTF-8"?>
<osmAugmentedDiff version="0.6" generator="Overpass API">
<note>... </note>
<meta osm_base="2012-08-26T20\:24\:02Z"/>
 
  <!-- Elements are ordered as: nodes first, then ways, then relations.
       Within each class of elements they are ordered by id -->
 
<erase>
  <node ... />
  <!-- contains the nodes that are either explicitly deleted or the old version of nodes that are replaced by a new version. -->
</erase>
<keep>
  <node ... />
  <!-- contains the nodes that belong to changed ways or relations including ways that contain a changed node or to ways that are members of a changed relation. -->
</keep>
<insert>
  <node ... />
  <!-- contains the nodes that are updated in their newest version. -->
</insert>
 
<erase>
  <way ... />
  <!-- contains the ways that are either explicitly deleted or the old version of ways that are replaced by a new version. -->
</erase>
<keep>
  <way ... />
  <!-- contains the ways that contain a changed node or ways that are members of a changed relation. -->
</keep>
<insert>
  <way ... />
  <!-- contains the ways that are updated in their newest version. -->
</insert>
 
<erase>
  <relation ... />
  <!-- contains the relations that are either explicitly deleted or the old version of relations that are replaced by a new version. -->
</erase>
<keep>
  <relation ... />
  <!-- contains the relations that contain a changed node or way (including ways changed only by changing their underlying nodes). -->
</keep>
<insert>
  <relation ... />
  <!-- contains the relations that are updated in their newest version. -->
</insert>
 
</osmAugmentedDiff>

id sorted

Like all other diffs, these diff enumerate OSM elements by type and by id. Every element is included in an action block, and depending on its status, it either appears in one or in two versions. The action section carries additional attributes to explain the status of the object in detail.

<action type="delete" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

An object is noted as "deleted" if it has been deleted in the Augmented Diff. Because this produces a new version, there exists also a "new" section. This contains an empty OSM element with marker "visible=false" and the meta data of the deletion.

<action type="modify" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

Same as "delete". Here, the new version is the proper new version of the element. Please note that for ways and relations old and new version could have the same version number. This happens when the element itself is unchanged but a member element has changed and thus also the geometry of the element itself.

<action type="create" ...>
  <!-- OSM element -->
</action>

The "create" action only contains one version, the new version.

<action type="info" ...>
  <!-- OSM element -->
</action>

The "info" action only contains one version, the unchanged current version.

For all the of actions, the attributes "waymember=yes", "relationmember=yes", and "indirectmember=yes" flag elements that are members of changed ways or relations. In the case of "indirectmember=yes", the node is member of a way that is member of a changed relation. Relations are not flagged as members of other relations. This is because relation memberships are assumed not to contribute to the relation's geometry.

If the tagging of an element has changed, the flag "tags=changed" is added as an attribute to action.

If the geometry of an element has changed, the flag "geometry=changed" is added as an attribute to action. This only applies to "modify", because for "delete" and "create" the geometry is always and for "info" the geometry is never changed. For nodes, the geometry is the coordinates. For ways, the geometry is the coordinates of the referred nodes. For relations, the geometry is the collection of the geometries of the referred nodes and ways.

If the collection of members of an elements has changed, the flag "members=changed" is added as attribute to action. This only applies to "modify", because for "delete" and "create" the members are always and for "info" the members are never changed.

If one of the members of an element has changed its tagging, the flag "memberproperties=changed" is added. This only applies to "modify" or "info". In particular, if only the tagging of a member has changed, the action is still "info" and not "changed".

The header section:

<osmAugmentedDiff version="XXX" generator="Overpass API">
<note>... </note>
<meta osm_base="2012-08-26T20\:24\:02Z"/>
 
All ''way'' and ''relation'' elements contain a additional ''bounds'' element:
<source lang=xml>
<way ... >
  <bounds minlat="..." minlon="..." maxlat="..." maxlon="..."/>
  ...
</way>

All nd elements inside of way elements contain additional lat and lon attributes:

 <way id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   ...
</way>

Time slices and numbering

Basically, an augmented diff is produced once per minute, like the minute diffs. As with the minute diffs, a single diff may contain more than one minute to allow the server to catch up. Furthermore, there is no one-to-one match of minute diffs and augmented diffs. An augmented diff may contain more than one minute diff, but a minute diff is never broken up and distributed over more than one augmented diff.

The augmented diffs have a numbering scheme similar to the numbering scheme of the minute diffs but have their own counter.

The augmented_diff API call

On overpass-api.de exists an API that allows to filter the Augmented Diffs already on the server. The base URL is

 http://overpass-api.de/api/augmented_diff?

It accepts the following three arguments:

  • id= is mandatory and is the id of the Augmented Diff to process
  • bbox= is optional and is a min_lon,min_lat,max_lon,max_lat limited bounding box of the area of interest
  • info= defaults to yes. With info=no, all blocks of action type info are omitted.

The info=no allows to drastically reduce the file size but contains still the necessary information to show way geometry: The coordinates are replicated in the nd tags of the ways. On the other hand, minor changes to country boundary relations happen quite often, and info=no then sends only the geomtry of the affected way, not the geometry of the entire relation.

The id of the last produced Augmented Diff can be obtained from

 http://overpass-api.de/augmented_diffs/state.txt

The id of the Augmented Diff belonging to a certain date can be obtained via

 http://overpass-api.de/api/augmented_state_by_date?

It takes as only parameter osm_base. This must be an OSM formatted time stamp. The call returns the id of the newest Augmented Diff that is older than the given date.

Applications

Software and Services using Augmented Diffs:

  • osmconvert (since 0.7E): The program can now read Augmented Diffs and convert them to .osc. It also can process them and update an existing .osm or .pbf. All this is highly experimental, you should expect one or two bugs... :-)
  • Osm-watch: osm-watch is an OSM contributions almost real-time monitoring tool with filters based alerts that you can created and receive by RSS or email.
  • achavi: Augmented Change Viewer - visualizes updates to OpenStreetMap
  • Show Me The Way on OSMLab