User:Kmpoppe/Automated Edits/Mapillary Update

From OpenStreetMap Wiki
Jump to navigation Jump to search

Rationale

In June 2021 Mapillary retired Version 3 of their API which led to a significant change in how Image IDs were issued. While Version 3 Image IDs had the pattern [A-Z0-9\-\_]{22}/i, new IDs are numeric only with at least 15 digits.

To allow data consumers to work with API Version 4, all tags mapillary=* shall be changed to contain the new Image ID.

Statistics

This statistic is up-to-date per November 28, 2021 (Date of tagInfo data). Values were grabbed from tagInfo and checked multiple times to reach the correct action for each value.

Type Description Proposed action Total #
All distinct values 215,924
Old IDs 22 characters matching the above mentioned pattern Translation into new IDs and update objects with these IDs 201,773
"short" Old IDs Old IDs that probably were valid but "lost" 1 or 2 character through Copy&Paste Wildcard search for probable old IDs, then translation 54
New IDs A numeric ID that was confirmed valid via Mapillary API v4 No change 14,072
Numeric Values A numeric ID that was confirmed invalid via Mapillary API v4 Remove tagging 8
Anything else Possible incorrect contents of tag Manual check, remove/change where possible, use this Maproulette Challenge to support removing those. 17
Successful matches Old Keys that have successfully matched to a new Key. Change mapillary key during edit 197,618
Unsuccessful matches Old keys that are no longer existent in mapillary's infrastructure Remove mapillary key during edit 3,349

History

Frédéric Rodrigo published a list of Old->New IDs as a gist On 1st Aug 2021. As per speaking with him he made it clear to me, that he'll not be doing the automated edit himself.

A discussion on Tagging-ML lead to the vague idea that an automated would be in order.

I checked back with @cbed32 (Chris Beddow) from Mapillary, who'll be providing ID-matches of those Old IDs that have popped up in OSM between August 1st and today as well as checking the "short IDs" if they can be found or have to be discarded.

Plan of Action

Until November 1st 2021 the list of Old IDs that need a translation into New IDs shall be finished and send to Chris Beddow for completion.

Until November 26th 2021 a list of changes (objects with mapillary IDs), grouped by country (or a smaller administrative unit if a country has more than 1000 objects) will be provided here for community information and discussion.

On the first weekend of December 2021 the changes should be made by a special OSM user for this import.

Milestones

  • On 21st Oct, Geofreund1 single-handedly corrected around 70 entries where WUSCH1463 had mistakenly entered a ref-value into the mapillary-key. Thanks!
  • Since Oct 25th, all single translation maps are available and now have to be condensed into one map and a plan to get the corresponding OSM object will have to be drawn up.
  • On 28th Nov, the work table (distinct old->new values) is in a database for further proceedings. I have contacted KUMI SYSTEMS, who host an overpass instance about how they will be used to find the affected OSM objects.

How are the changes done?

A PHP script, running on my root server (hosted in Frankfurt, Germany), will use the OSM API to upload the changes. Discussions are welcome on how Changesets should be split (country- or smaller-wise like mentioned above or one CS for every distinct "old ID" so a max of ~50 objects per Changeset but many Changesets with 1 change).