Overpass API/Augmented Diffs

From OpenStreetMap Wiki
Jump to: navigation, search
Available languages
English русский

The augmented diffs extend the ordinary minute diffs. They make it possible to keep even specialized databases up to date minutely. They are based on an idea by Matt Amos, refreshed by Frederik in a talk in German.

The experimental version of the feature is going out of service in the end of July. Augmented diffs are available from the API endpoint

http://overpass-api.de/api/augmented_diff

That endpoint is a stable feature that gets long time support.

The experimental augmented diffs can be still downloaded from http://overpass-api.de/augmented_diffs/


Overview

The augmented diffs contain three kinds of data

  • The new data that comprises the usual minute diffs
  • The old data that is either replaced by new data or explicitly deleted
  • Meta data for deleted objects

The ways and relations are in addition enhanced by their geometry.

This allows in particular the following two applications

  • Proper minute updates on datasets restricted by a bounding box
  • A wayback mechanism: You can start with a planet file or other database and walk back instead of forward through time

While the use of the former is immediately clear, the second needs some explaining words: A couple of examples from the talks above like the counting application need to know what is deleted to decrease the counter.

Contained data

Each diff is structured in three sections, one for each type of element. Within a section, the elements are ordered by id. Each element only appears its state at the beginning and the state at the end of the minute. Shorter lived versions are omitted.

Every element is considered as having changed when its meta date has been updated or its tags have changed. In addition, nodes are considered to have changed when their coordinates have changed. Ways are considered as changed when their members have changed or one of its members has changed its coordinate. Relations are considered as changed when their members have changed or one of its way members has changed by its members or coordinates or one of the node members has changed its coordinate. By contrast, changes on members of type relation don't propagate to their parent relations.

In the preamble, the file contains the end date of the diff.

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>... </note>
<meta osm_base="2012-09-20T20\:24\:02Z"/>
 
  <!-- Elements are ordered as: nodes first, then ways, then relations.
       Within each class of elements they are ordered by id -->
 
</osm>

Actions

Each element is wrapped in an action of type delete, modify, or create, depending on whether the element is deleted, changed, or newly created.

<action type="delete" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

An object is noted as "deleted" if it has been deleted in the Augmented Diff. Because this produces a new version, there exists also a "new" section. This contains an empty OSM element with marker "visible=false" and the meta data of the deletion.

<action type="modify" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

Same as "delete". Here, the new version is the proper new version of the element. Please note that for ways and relations old and new version could have the same version number. This happens when the element itself is unchanged but a member element has changed and thus also the geometry of the element itself.

<action type="create" ...>
  <!-- OSM element -->
</action>

The "create" action only contains one version, the new version.

Elements

All way and relation elements contain a "bounds" element. This elements tells you the enclosing bounding box.

All nd elements inside of way elements contain additional lat and lon attributes:

 <way id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   ...
</way>

All members of type node of relations also contain additional "lat" and "lon" attributes:

 <rel id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <member type="node" role="" ref="..." lat="..." lon="..." />
   <member type="node" role="" ref="..." lat="..." lon="..." />
   <member type="node" role="" ref="..." lat="..." lon="..." />
   ...
</rel>

Members of type way repeat their geometry in a sequence of "nd" subelements:

 <rel id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <member type="way" role="" ref="...">
     <nd lat="..." lon="..."/>
     <nd lat="..." lon="..."/>
     <nd lat="..." lon="..."/>
   </member>
   ...
</rel>

Time slices and numbering

The augmented diffs are numbered with consecutive numbers. The zero point is the license change, i.e.

http://overpass-api.de/api/augmented_diff?id=0&debug=yes

will tell you that it is going to fetch the difference between "2012-09-12T06:55:00Z" and "2012-09-12T06:56:00Z". From that date on, the counter is increased once per minute.

You can get the last available augmented diff via the API call

http://overpass-api.de/api/augmented_diff_status

The augmented_diff API call

On overpass-api.de exists an API that allows to filter the Augmented Diffs already on the server. The base URL is

 http://overpass-api.de/api/augmented_diff?

It accepts the following three arguments:

  • id= is mandatory and is the id of the Augmented Diff to process
  • bbox= is optional and is a min_lon,min_lat,max_lon,max_lat limited bounding box of the area of interest

Applications

Software and Services using Augmented Diffs:

  • osmconvert (since 0.7E): The program can now read Augmented Diffs and convert them to .osc. It also can process them and update an existing .osm or .pbf. All this is highly experimental, you should expect one or two bugs... :-)
  • Osm-watch: osm-watch is an OSM contributions almost real-time monitoring tool with filters based alerts that you can created and receive by RSS or email.
  • achavi: Augmented Change Viewer - visualizes updates to OpenStreetMap
  • Show Me The Way on OSMLab