Facebook AI-Assisted Road Tracing

From OpenStreetMap Wiki
(Redirected from AI-Assisted Road Tracing)
Jump to navigation Jump to search

Facebook have been running a project to perform AI-Assisted Road Tracing within OpenStreetMap. This has taken place in several phases, and involved local OpenStreetMap communities in Thailand and Indonesia.

In 2016, Facebook started researching how various machine learning algorithms could be integrated into a mapping workflow. At State of the Map 2016 Facebook shared initial results on how one could use AI and new mapping techniques, such as advanced deep neural net models, to detect roads from satellite imagery. We have since received many requests to share our process and masks. Our goal is to collaborate with the OpenStreetMap community to create quality road data, validated by trained mappers and local OSM communities. Please send questions to osm@fb.com.

For more information on the project please also check out Mapwith.ai and the github page Blacktocat.svg Facebook OSM Data Team

Schedule

Partnering with the local OSM communities, the pilot mapping countries were Thailand followed by Indonesia. They chose these countries because of their relative low road coverage on OSM.

Thailand Road Network Changes from June 1st 2017, to December 21st 2018:

Completed

Stats for total project length (including community contributions during the time frame):

  • 956,381 New ways added
  • 515,306 km's of roads added

Indonesia Road Network Changes from March 1st 2018, to September 1st 2019

~99% completed

Stats for total project length (including community contributions during the time frame)

  • 1,711,470 New ways added
  • 614,328 km's of roads added

Malaysia

Work begins September 11th 2019

Data Processing

DigitalGlobe source imagery

DigitalGlobe's Vivid+ product provides high-resolution (50cm / pixel, or zoom 18) color-corrected cloud-free satellite imagery with full country coverage. We now have license to extract road data from the imagery and contribute to OSM.

The following images show the benefits of using current high-resolution imagery. For example the quality of satellite image currently available to the OSM community for Mu Ban Don Sak Thani, Thailand is not optimal for mapping.

Mu Ban Don Sak Thani of Thailand. Vivid+ on left, Bing Aerial on right, at zoom 14, which is Bing Aerial's max zoom for the city.

Figure 1.png

The full resolution goes to zoom 18.

Figure 2.png

Another example of Ko Samui where Bing has zoom 18 but it's heavily covered by clouds.

Figure 3.png

Road Extraction

We will only contribute roads to OSM. We use the latest deep learning algorithms to assist road extraction and mapping. Here is an example of a prediction using deep learning for an area in Thailand.

Data Processing Image Example.png

Data Processing Image Example 2.png

We then apply a series of post processing steps to generate .osm file that can be loaded into JOSM or our modified iD for manual correction and enhancement. The images below show the .osm file in our iD, where the grey road is an existing road on OSM, and green roads are new.

Data Processing Image Example 3.png

Data Processing Image Example 4.png

Submission Process

The submission process consists of the following steps summarized in the image below.

Project Workflow.png

(1) We use the OSM Tasking manager to divide up tasks and create a flow of editing and validation for the Facebook mapping team.

(2) The Mapper picks a task and loads an .osm/.xml file generated from the post processing step into our enhanced version of iD (this includes both the roads generated by us and the current OSM data).

(3) The Mapper evaluates the roads we generated (styled in a different color) and inspect them for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fixes these issues manually one by one, using DigitalGlobe satellite imagery.

(4) Our modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint. This allows us to check for quality and conflicts with current OSM data prior to submission. Some examples of what we check for include:

  • Isolated roads that are not connected to rest of the map.
  • Roads crossing waterways.
  • Roads with repeating node IDs.
  • Roads with possible connections after a short extension.
  • Self-intersecting ways.

You can watch the video example of the various data integrity editing features here.

(5) In case of conflicts between our newly added roads and other OSM contributors' changes, our mappers will almost always choose “keep their edits.” If we end up changing another mapper's edits, (for example, to align better with the latest version of DG satellite imagery) we will leave detailed notes to explain why. 

(6) Until all errors are fixed, Mappers will not be able t upload to OSM. Here is an example of a highlighted issue prompting the mapper to fix it.

(7) After fixing all the issues detected in the iD tool, the Mapper clicks the “Save local” button to save their edited roads locally for validation.

(8) A Validator then goes into the same task to verify that the data looks correct by re-validating the data, makes necessary changes as needed, and clicks “Submit” to finally upload the task to OSM. 

(9) The Validator then conducts a post-submission review of the submitted edits in JOSM. If any errors are detected, the validator addresses the errors and submits necessary changes. A custom JOSM paint style is used to help distinguish our submissions from community submissions, so that all roads with the import=yes tag are highlighted in green. Some examples for what we check for include: 

  • Short Unclassified ways (between 10-60 meters)
  • Highway crossing waterway without a bridge tag
  • Highway crossing buildings
  • Overlapping ways
  • Unspecific highway tags

(10) We will continue to watch all comments made on our edits, reply to them in a timely fashion. We are happy to incorporate feedback and local expertise into our validation process and future mapping behaviors. 

Data Preparation

Road Type Tagging

During the submission process, our mappers will assign and fix road types according to the general highway tag guidelines of OSM http://wiki.openstreetmap.org/wiki/Key:highway

Before moving to a new location, we will work with mappers from local OSM community in each country to verify our highway tagging method and make sure they align with local conventions.

Some other methods we use to tag roads include:

  • Referring to local Wiki guides for each country.
  • Looking at commonly used tags in the local area of mapping.
  • When connecting to current OSM roads on the map we try to use the same tag where it makes sense to do so.
  • Referring to local community for advice.

Changeset Tags

We use the following tags:

Changeset Tags:

We use the above changeset tags in every import changeset, where the '#country' tag is the country of the submission. For Thailand the tags are '#nsroadimport #thailand'.

Element Tags:

  • source = digitalglobe
  • import = yes

The 'source' tag indicates that road geometries are based on DigitalGlobe satellite imagery.

Data Merge Workflow

Team Approach

This import (data integration) will be done through a Tasking Manager, by engineers and a group of selected trained map editors. Among the skills required:

  • Good experience with iD and JOSM.
  • Familiarity with editing norms: how to merge nodes (M), join ways (J), combine ways (C) and un-glueing (G)
  • S/he knows how to handle conflicts with both iD tool and JOSM.

The map editing team consists of Mappers and Validators, and their accounts are listed as below.

Team Approach

This import (data integration) will be done through a Tasking Manager, by engineers and a group of selected trained map editors. Among the skills required:

  • Good experience with iD and JOSM.
  • Familiarity with editing norms: how to merge nodes (M), join ways (J), combine ways (C) and un-glueing (G)
  • S/he knows how to handle conflicts with both iD tool and JOSM.

The map editing team consists of Mappers and Validators, and their accounts are listed as below.

Mappers:

Name Username Role Link
Zvone RVR001 Validator https://www.openstreetmap.org/user/RVR001
Jeff RVR002 QA https://www.openstreetmap.org/user/RVR002
Yunzhi RVR004 QA https://www.openstreetmap.org/user/RVR004
Charmaine RVR012 QA https://www.openstreetmap.org/user/RVR012
Rena VLD017 Mapper https://www.openstreetmap.org/user/VLD017
Tina VLD020 Mapper https://www.openstreetmap.org/user/VLD020
Ian VLD023 Validator https://www.openstreetmap.org/user/VLD023
Maureen VLD026 Mapper https://www.openstreetmap.org/user/VLD026
Sophie VLD030 Mapper https://www.openstreetmap.org/user/VLD030
Dylan VLD032 Mapper https://www.openstreetmap.org/user/VLD032
Katherine VLD035 Validator https://www.openstreetmap.org/user/VLD035
Olympia VLD036 Validator https://www.openstreetmap.org/user/VLD036
Rowen VLD040 Mapper https://www.openstreetmap.org/user/VLD040
Mike VLD041 Mapper https://www.openstreetmap.org/user/VLD041
Haley VLD043 Mapper https://www.openstreetmap.org/user/VLD043
Brady VLD045 Mapper https://www.openstreetmap.org/user/VLD045
Kaiti VLD046 Validator https://www.openstreetmap.org/user/VLD046
Maegan VLD048 Mapper https://www.openstreetmap.org/user/VLD048
Gerum VLD052 Mapper https://www.openstreetmap.org/user/VLD052
Jesspher VLD062 Mapper https://www.openstreetmap.org/user/VLD062
Ming VLD066 Mapper https://www.openstreetmap.org/user/VLD066
Jacob VLD067 Mapper https://www.openstreetmap.org/user/VLD067
Joseph VLD072 Mapper https://www.openstreetmap.org/user/VLD072
Christopher VLD073 Mapper https://www.openstreetmap.org/user/VLD073
Ryan VLD074 Mapper https://www.openstreetmap.org/user/VLD074
Michael VLD075 Mapper https://www.openstreetmap.org/user/VLD075
Lewei VLD076 Mapper https://www.openstreetmap.org/user/VLD076
Andre VLD077 Mapper https://www.openstreetmap.org/user/VLD077
Courtney VLD079 Mapper https://www.openstreetmap.org/user/VLD079
Henry VLD080 Mapper https://www.openstreetmap.org/user/VLD080
Adrian VLD081 Validator https://www.openstreetmap.org/user/VLD081
Gwendolyn VLD082 Mapper https://www.openstreetmap.org/user/VLD082
Julia VLD083 Mapper https://www.openstreetmap.org/user/VLD083
Joshua VLD085 Mapper https://www.openstreetmap.org/user/VLD085
Cameron VLD086 Mapper https://www.openstreetmap.org/user/VLD086
Hector VLD087 Mapper https://www.openstreetmap.org/user/VLD087
Steve VLD089 Mapper https://www.openstreetmap.org/user/VLD089
Tukui VLD090 Mapper https://www.openstreetmap.org/user/VLD090
Jeff VLD091 Mapper https://www.openstreetmap.org/user/VLD091
Yanitza VLD092 Mapper https://www.openstreetmap.org/user/VLD092
Je-Ni VLD093 Mapper https://www.openstreetmap.org/user/VLD093
Amin VLD094 Mapper https://www.openstreetmap.org/user/VLD094
Larissa VLD095 Mapper https://www.openstreetmap.org/user/VLD095
Kayla VLD096 Mapper https://www.openstreetmap.org/user/VLD096
Matthew VLD097 Mapper https://www.openstreetmap.org/user/VLD097
Bala VLD100 Validator https://www.openstreetmap.org/user/VLD100
Narendar VLD109 Mapper https://www.openstreetmap.org/user/VLD109
Samba VLD117 Mapper https://www.openstreetmap.org/user/VLD117
Sandeep VLD118 Mapper https://www.openstreetmap.org/user/VLD118
Shakil VLD119 Mapper https://www.openstreetmap.org/user/VLD119
Teesta VLD124 Mapper https://www.openstreetmap.org/user/VLD124
Gunapati VLD126 Mapper https://www.openstreetmap.org/user/VLD126
Nayani VLD127 Validator https://www.openstreetmap.org/user/VLD127
Shyam VLD128 Mapper https://www.openstreetmap.org/user/VLD128
Madduluri VLD129 Mapper https://www.openstreetmap.org/user/VLD129
Srinivas VLD130 Mapper https://www.openstreetmap.org/user/VLD130
Chittipolu VLD131 Mapper https://www.openstreetmap.org/user/VLD131
Mahesh VLD132 Mapper https://www.openstreetmap.org/user/VLD132


We are not bots :) The usernames (VLD001, RVR001, etc.) were made to help our internal process where we have paired Mappers and Validators to ensure quality. Please feel free to send us questions individually about edits or email all of us at osm@fb.com.

Sub-contracted Teams

Humanitarian OpenStreetMap Team (HOT)

For mapping Indonesia, Facebook has subcontracted to HOT employees, as they have an office in Jakarta; in order to better coordinate with the local community and improve the road network of Indonesia. The team is comprised of:

Name Username Role Link
Wulan Wulan IRM-RV Team Lead https://www.openstreetmap.org/user/Wulan%20IRM-RV
Adiatma Adiatma IRM-RV QA https://www.openstreetmap.org/user/Adiatma%20IRM-RV
Hanif Hanif IRM-RV QA https://www.openstreetmap.org/user/Hanif%20IRM-RV
Yeni Yeni IRM-RV QA https://www.openstreetmap.org/user/Yeni%20IRM-RV
Akrimullah Akrimullah IRM-RV QA https://www.openstreetmap.org/user/Akrimullah%20IRM-RV
Adam Adam IRM-ED Mapper https://www.openstreetmap.org/user/Adam%20IRM-ED
Silvia Silvia IRM-ED Mapper https://www.openstreetmap.org/user/Silvia%20IRM-ED
Dary Dary IRM-ED Mapper https://www.openstreetmap.org/user/Dary%20IRM-ED
Irwan Irwan IRM-ED Mapper https://www.openstreetmap.org/user/Irwan%20IRM-ED
Riyadi Riyadi IRM-ED Mapper https://www.openstreetmap.org/user/Riyadi%20IRM-ED
Zainab Zainab IRM-ED Mapper https://www.openstreetmap.org/user/Zainab%20IRM-ED
Deo Deo IRM-ED Mapper https://www.openstreetmap.org/user/Deo%20IRM-ED
Anisa Anisa IRM-ED Mapper https://www.openstreetmap.org/user/Anisa%20IRM-ED
atmahadi atmahadi IRM-ED Mapper https://www.openstreetmap.org/user/atmahadi%20IRM-ED
Dennis Dennis IRM-ED Mapper https://www.openstreetmap.org/user/Dennis%20IRM-ED
Fatri Fatri IRM-ED Mapper https://www.openstreetmap.org/user/Fatri%20IRM-ED
Muty Muty IRM-ED Mapper https://www.openstreetmap.org/user/Muty%20IRM-ED
Iko Iko IRM-ED Mapper https://www.openstreetmap.org/user/Iko%20IRM-ED
Aglis Aglis IRM-ED Mapper https://www.openstreetmap.org/user/Aglis%20IRM-ED
Diana Diana IRM-ED Mapper https://www.openstreetmap.org/user/Diana%20IRM-ED
Fadlilaa Fadlilaa IRM-ED Mapper https://www.openstreetmap.org/user/Fadlilaa%20IRM-ED
Arma Arma IRM-ED Mapper https://www.openstreetmap.org/user/Arma%20IRM-ED
Elisabeth Elisabeth IRM-ED Mapper https://www.openstreetmap.org/user/Elisabeth%20IRM-ED
Diah Diah IRM-ED Mapper https://www.openstreetmap.org/user/Diah%20IRM-ED
Dorica, doricamugusi_ImportAccount Team Lead https://www.openstreetmap.org/user/doricamugusi_ImportAccount
Gwantwa Gwantwa_Andrew_ImportAccount Mapper https://www.openstreetmap.org/user/Gwantwa_Andrew_ImportAccount
Dorice Dorice lucas_ImportAccount Mapper https://www.openstreetmap.org/user/Dorice%20lucas_ImportAccount
Jamal JAMAL AMANI_ImportAccount Mapper https://www.openstreetmap.org/user/JAMAL%20AMANI_ImportAccount
Mariamu Mariamu Crispo_ImportAccount Mapper https://www.openstreetmap.org/user/Mariamu%20Crispo_ImportAccount
Nuru NURU ATHUMANI_ImportAccount Mapper https://www.openstreetmap.org/user/NURU%20ATHUMANI_ImportAccount
Samwel Samwel Kyando_Import Mapper https://www.openstreetmap.org/user/Samwel%20Kyando_Import
Charles charles Frank_ImportAccount Mapper https://www.openstreetmap.org/user/charles%20Frank_ImportAccount
Abou Abou kachongo jr_ImportAccount Mapper https://www.openstreetmap.org/user/Abou%20kachongo%20jr_ImportAccount
Christina Christina Reuben_ImportAccount Mapper https://www.openstreetmap.org/user/Christina%20Reuben_ImportAccount
Edger Edger_ImportAccount Mapper https://www.openstreetmap.org/user/Edger_ImportAccount
Aron Aron Martin_ImportAccount Mapper https://www.openstreetmap.org/user/Aron%20Martin_ImportAccount
Erasto ERASTO&20ELIUD_ImportAccount Mapper https://www.openstreetmap.org/user/ERASTO&20ELIUD_ImportAccount
Asibwene asibwene_ImportAccount Mapper https://www.openstreetmap.org/user/asibwene_ImportAccount
Godfrey Godfrey Faustine_ImportAccount Mapper https://www.openstreetmap.org/user/Godfrey%20Faustine_ImportAccount
Janet janethmlinga_ImportAccount Mapper https://www.openstreetmap.org/user/janethmlinga_ImportAccount
Catherine CatherineG_ImportAccount Mapper https://www.openstreetmap.org/user/CatherineG_ImportAccount
Damon Damon Monyo-ImportAccount Mapper https://www.openstreetmap.org/user/Damon Monyo-ImportAccount
Furaha Furaha L Mrema_ImportAccount Mapper https://www.openstreetmap.org/user/Furaha%20L%20Mrema_ImportAccount
Godfrey Godfrey Hussein_ImportAccount Mapper https://www.openstreetmap.org/user/Godfrey%20Hussein_ImportAccount
Neema NeemaAlphonce_ImportAccount Mapper https://www.openstreetmap.org/user/NeemaAlphonce_ImportAccount


Inactive users

Name Username Link
Aude Aude IRM-ED https://www.openstreetmap.org/user/Aude%20IRM-ED
David RVR003 https://www.openstreetmap.org/user/RVR003
Adrian RVR005 https://www.openstreetmap.org/user/RVR005
Stefani RVR006 https://www.openstreetmap.org/user/RVR006
Mohamed RVR007 https://www.openstreetmap.org/user/RVR007
Sam RVR009 https://www.openstreetmap.org/user/RVR009
Jaclyn VLD001 https://www.openstreetmap.org/user/VLD001
Annie VLD002 https://www.openstreetmap.org/user/VLD002
Alexandra VLD003 https://www.openstreetmap.org/user/VLD003
Sheffield VLD004 https://www.openstreetmap.org/user/VLD004
Mei VLD005 https://www.openstreetmap.org/user/VLD005
Joseph VLD006 https://www.openstreetmap.org/user/VLD006
Kurt VLD007 https://www.openstreetmap.org/user/VLD007
Pablo VLD008 https://www.openstreetmap.org/user/VLD008
Yoko VLD009 https://www.openstreetmap.org/user/VLD009
Klavdija VLD010 https://www.openstreetmap.org/user/VLD010
Michael VLD011 https://www.openstreetmap.org/user/VLD011
Paris VLD012 https://www.openstreetmap.org/user/VLD012
Kailei VLD013 https://www.openstreetmap.org/user/VLD013
Jen VLD014 https://www.openstreetmap.org/user/VLD014
Joseph VLD015 https://www.openstreetmap.org/user/VLD015
Gemma VLD016 https://www.openstreetmap.org/user/VLD016
Lena VLD018 https://www.openstreetmap.org/user/VLD018
Elizabeth VLD019 https://www.openstreetmap.org/user/VLD019
Alayna VLD025 https://www.openstreetmap.org/user/VLD025
Louis VLD027 https://www.openstreetmap.org/user/VLD027
Ismael VLD028 https://www.openstreetmap.org/user/VLD028
Savanna VLD033 https://www.openstreetmap.org/user/VLD033
Christina VLD034 https://www.openstreetmap.org/user/VLD034
Jacob VLD039 https://www.openstreetmap.org/user/VLD039
Jonathan VLD042 https://www.openstreetmap.org/user/VLD042
Casey VLD044 https://www.openstreetmap.org/user/VLD044
Elena VLD050 https://www.openstreetmap.org/user/VLD050
Linzi VLD053 https://www.openstreetmap.org/user/VLD053
Ryan VLD054 https://www.openstreetmap.org/user/VLD054
Yonis VLD055 https://www.openstreetmap.org/user/VLD055
Luxinaree VLD056 https://www.openstreetmap.org/user/VLD056
Katalin VLD057 https://www.openstreetmap.org/user/VLD057
John VLD058 https://www.openstreetmap.org/user/VLD058
Nick VLD059 https://www.openstreetmap.org/user/VLD059
Guowei VLD063 https://www.openstreetmap.org/user/VLD063
Derrick VLD070 https://www.openstreetmap.org/user/VLD070
Johanna VLD078 https://www.openstreetmap.org/user/VLD078
Ashley VLD088 https://www.openstreetmap.org/user/VLD088


References

Following the imports guidelines, these imports will be discussed first in the country specific mailing list, and to get more feedback it will also be shared with the import list.

Workflow

In summary, the overall mapping workflow is:

(1) Extract grey-scale road mask images from high resolution DigitalGlobe satellite imagery.

(2) enerate road vectors based on the road mask images and merge them with latest OSM data to produce .osm files.

(3) The human mapping team does manual validation on the .osm files through our enhanced iD tool and on JOSM prior to submission (as described in the “Submission Process” above).

(4) After conflict resolution, submission, and post-submission review, work with local OSM community to complete local validation as described in the “Validation Steps” below.

Validation Steps

There will be a 4 steps validation process done by experienced OSM editors including local country Mappers.

  • Our method for each country will include starting with small number of edits to gather feedback from the local OSM community before proceeding with larger edits.
  • We then start with human validation from our Mappers as explained in the “Submission Process.”
  • Before submitting our changes, our team will verify that our updates are based on the current dataset.
  • After Submission (as described in the “Submission Process” above, a team of selected Validators and QA's conduct a second post-submission review by passing over chunks of multiple submitted tasks , region by region, by repeating the JOSM validation checks in described in Step 9 under “Submission.”
  • Our validation process primarily only includes elements that our team has contributed, which can be identified by the “import=yes” tag. Elements that our team “inherits,” meaning elements that are authored by one of our usernames but lacking the “import=yes” tag, are not typically included in post-submission validation. For more about inheritance, see footnotes.
    • We understand inherited versus contributed elements by exploring the ownership of nodes and their parent ways. We can identify inheritance based on authorship and tagging. Roads that are authored by one of our usernames, but does not include the Import=yes tag, is likely an inherited way. Because we are a team of remote mappers, using one data source, we want to avoid overstepping our contributions.
  • Community feedback on imported roads is evaluated and incorporated for final edits in this stage as well. All edits can be identified with the changeset tags above and the team will respond to any e-mails or concerns about edits within 48 hours.

To facilitate the validation process, we share our road mask as a tile service that can be used in iD tool and JOSM as a background layer. The URL template of the tile service is distributed on a per-request basis. Please contact osm@fb.com for requesting a validated URL template.

So far we have prepared road masks for selected areas in Egypt and Thailand. We will announce the availability for other countries here as they become ready.

Local OSM Community Engagement - Using the list servers and wikis we contact local OSM community members before starting to edit in a specific country. Through this we invite members to collaborate and provide valuable local insight and feedback.

Legal terms of use for our tile service: see Media:FacebookRoadMaskLicense.pdf. Information regarding DigitalGlobe Imagery: DigitalGlobe confirms the above stated license grants pertaining to DigitalGlobe satellite imagery are accurate.

To use our tile service in iD:

(1) Click the “Edit custom background” button in iD

Legal terms of use for our tile service 1.png

(2) Paste the URL template above in the input box of the pop-up dialog and click “OK”

Legal terms of use for our tile service 2.png

To use our tile service in JOSM:

(1) Click Imagery > Imagery Preferences dialog

(2) In the Imagery providers tab, in the lower right hand side of the dialog box, click on “+TMS” button

Legal terms of use for our tile service 3.png

(3) Enter the URL template in the input box and provide a name for the new layer

Legal terms of use for our tile service 4.png

(4) Once added, this new layer will show up under the Imagery list. You can select this layer to be displayed in JOSM

Legal terms of use for our tile service 5.png


FAQ's

What does AI mean?
Artificial Intelligence (AI) is training a computer to do things that require intelligence when done by humans. In this case it involves creating training data for an area by tracing out satellite imagery the same way you would when editing and feeding it to a computer. The output of this is road masks. Here is an example prediction result using deep learning for an area in Thailand.
Are there limitations with AI?
Yes, AI and algorithms still have a long way to go to figure things such as assigning tags according to OSM region specific guidelines, figure out the correct OSM relation, add bridges over waterbodies or figure out connections with crossing railways among many other things. Currently all of this is done by our mappers manually.
Are you adding roads directly to OSM without humans validating?
Every single road from the AI output will be added only after multiple rounds of human validation. After some processing of the road masks shown above we end up with a .osm file. We then treat this like remote tracing. Using iD editor and/or JOSM mappers make sure that merging with the current data has happened smoothly. They tag each road appropriately, connect them to current OSM edits, fix alignment, when necessary, add sensible changeset comments/notes and resolve conflicts before finally saving where it is merged with the current OSM data.
Why are you using iD Editor versus JOSM?
We know JOSM is efficient at what we are trying to do. In fact, we started primarily with JOSM. This said, to make our internal process more efficient, our engineers improved the iD editor so our version has comparable functionality as JOSM. We are working on making this public so we can share it with the community.
What Imagery are you using?
We are using DigitalGlobe's Vivid+ which is high-resolution (50cm / pixel, or zoom 18) color-corrected and cloud-free
Can you share DG imagery?
Our imagery is very similar to the Maxar imageries available in iD and JOSM. Our exact version used for prediction is available in RapiD as Facebook's Map With AI - Maxar Imagery
Facebook License with Digital Globe
This license is necessary to comply with our obligations to Digital Globe, and we think the benefits of supplying this data outweigh the licensing restraints. The data will be under ODBL once it has been contributed to OSM in compliance with the terms.
There is no contradiction between the OSM terms and the Facebook license with Digital Globe. Our license terms do not apply to data once contributed to OSM and termination of the agreement with Digital Globe would not impact the data that has already been contributed to OSM.
Is the Imagery free of Cloud Cover?
The intended output of +Vivid is cloud free, and in most cases, DG meets this. There are some highly cloudy parts of the world where it’s unavoidable. On average though, this product maintains less than 1% cloud cover.
Who is behind osm@fb.com?
There are 4 people behind this email. This helps us collaborate as a team and is the most efficient way for everyone to be on the same page. More people also ensures that no emails are missed. I (Drishtie Patel) as the Program Manager primarily watch this email and respond to queries along with Ming Gao and Saikat Basu.
Can the Chageset be more specific?
Yes we can absolutely change that to import=facebook-ai-____. We don't specifically have versions but can include the area or task number for example. Open to suggestions.
As for adding more details to our edits: there were no changes to current OSM data in this sample and we plan to add notes when we upload as would normally happen when using the Tasking Manager and iD. The sample shared has not been uploaded yet.
Can you Publish Source Code for ML?
We cannot share the source code for Machine Learning at this time, but we can and will share our internal tools.
Can you Publish Source Code for internal tools?
Yes. We are currently using a version of iD Editor and the HOT Tasking Manager. We are absolutely going to share our tooling. We are currently working on creating a Github where we will post soon.
Can Facebook Share other data?
This is clear violation of privacy so we cannot share this kind of information. We are exploring ways to crowdsource road names for OSM and will share that as we have a more concrete plan.
Why did you choose Thailand?
Facebook has a high number of users in this country and we would like to improve that map for this community. It is also our mission as a company is to make the world more open and connected and one way we can do is by filling in the missing gaps on the map. We also saw a strong OSM community that we could learn from and engage with while we refine our process for mapping. We are hoping for community feedback as we move forward so we can contribute high quality edits.
Are you going to Map other countries?
Yes. We are moving slowly to focus efforts on one country at a time to make sure our process is accepted by the OSM community out of respect for the process.
What is the Overall Process?
PHASE 1 - Generating Road Masks
  1. Training data is created by editors. Example
  2. Training data used by engineers in ML to produce road masks. Example
  3. Road masks are processed to remove low confidence predictions and add connections between short breaks.
PHASE 2 - Creating Road Vectors (.osm files)
1. Road masks are then processed using an algorithm from a black and white image to a vector format specifically a .osm file. Example
2. Our .osm file is then merged with current OSM changesets. During this process the following things take place
All current OSM data for an area is merged, keeping full history.
No changes are made to the current OSM data.
Duplicated roads are deleted from the ML generated data. Current OSM data is always taken over facebook generated data.
New ML identified roads are connected to current OSM roads.
3. We then have a locally stored file containing both our generated roads and the current OSM data set for a specific area.
PHASE 3 - Human Validation
1. We use the Tasking manager to divide up tasks and create a flow of editing and validation for the Facebook mapping team. Example
2. The Mapper picks a task and loads the locally stored .osm file generated from the post processing steps outlined above. Example
3. The task is opened up in our enhanced version of iD. (This includes both the roads generated by us and the current OSM data). Example
4. Our generated roads are highlighted in a different color so editors can inspect it for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fix these issues manually one by one, using Digital Globe satellite imagery background.
5. We do not delete current OSM data, but make typical OSM editing changes where necessary to ensure high quality of the data (more precisely follow imagery, consistent tagging, etc).
6. Our modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint. This allows us to check for quality and conflicts with current OSM data. Some examples of what we check for include:
isolated roads that are not connected to rest of the map. Example-Before Example-After
roads crossing waterways.
roads with repeating node IDs.
roads with possible connections after a short extension. Example-Before Example-After
7. In case of conflicts between our newly created roads and other OSM editor's mapping, our mapper will almost always choose “keep their edits”. If we think our newly added roads are better aligned with the latest version of DG satellite imagery, we'll contact the other mapper offline to reach resolution. If we end up changing other mapper's edits, we will leave detailed notes to explain why.
8. Until all errors are fixed mappers will not be able to save. Here is an example of a highlighted issue prompting the mapper to fix it.
9. After fixing all the issues detected, our mapper clicks the “Save Local” button in the iD tool to save their edited roads locally for validation.
10. A second person, the validator then goes into the same task to verify that the data looks correct, makes necessary changes as needed and clicks “Submit” to finally upload the tile to OSM.


Controversy

  • LukeWalsh (28 August 2018). “https://news.ycombinator.com/item?id=17856687”.