Facebook AI-Assisted Road Tracing
Facebook have been running a project to perform AI-Assisted Road Tracing within OpenStreetMap. This has taken place in several phases, and involved local OpenStreetMap communities in Thailand and Indonesia.
In 2016, Facebook started researching how various machine learning algorithms could be integrated into a mapping workflow. At State of the Map 2016 Facebook shared initial results on how one could use AI and new mapping techniques, such as advanced deep neural net models, to detect roads from satellite imagery. We have since received many requests to share our process and masks. Our goal is to collaborate with the OpenStreetMap community to create quality road data, validated by trained mappers and local OSM communities. Please send questions to firstname.lastname@example.org.
For more information on the project please also check out Mapwith.ai and the github page
Partnering with the local OSM communities, the pilot mapping countries were Thailand followed by Indonesia. They chose these countries because of their relative low road coverage on OSM.
- Work began January 21st, 2020
- Changeset: #mapwithai #Vietnam
- Road mapping will begin once Malaysia is completed
Changeset: #mapwithai #Tanzania
- June 1, 2017 to December 21, 2018
- Stats for total project length (including community contributions during the time frame):
- 956,381 New ways added
- 515,306 km of roads added
- Road Network Changes from March 1, 2018 to September 24, 2019
- 1,095,034 new ways added
- 474,409 km of roads added
- Stats for total project length (including community contributions during the time frame)
- 1,711,470 new ways added
- 614,328 km of roads added
- December 23, 2019 - December 31, 2019
- We added missing roads to area around the city of Leogane at the request of MotoMeds Haiti.
- Ongoing since September 11, 2019
Changeset: #mapwithai #Malaysia
DigitalGlobe source imagery
DigitalGlobe's Vivid+ product provides high-resolution (50cm / pixel, or zoom 18) color-corrected cloud-free satellite imagery with full country coverage. We now have license to extract road data from the imagery and contribute to OSM.
The following images show the benefits of using current high-resolution imagery. For example the quality of satellite image currently available to the OSM community for Mu Ban Don Sak Thani, Thailand is not optimal for mapping.
Mu Ban Don Sak Thani of Thailand. Vivid+ on left, Bing Aerial on right, at zoom 14, which is Bing Aerial's max zoom for the city.
The full resolution goes to zoom 18.
Another example of Ko Samui where Bing has zoom 18 but it's heavily covered by clouds.
We will only contribute roads to OSM. We use the latest deep learning algorithms to assist road extraction and mapping. Here is an example of a prediction using deep learning for an area in Thailand.
We then apply a series of post processing steps to generate .osm file that can be loaded into JOSM or our modified iD for manual correction and enhancement. The images below show the .osm file in our iD, where the grey road is an existing road on OSM, and green roads are new.
The submission process consists of the following steps summarized in the image below.
(1) We use the OSM Tasking manager to divide up tasks and create a flow of editing and validation for the Facebook mapping team.
(2) The Mapper picks a task and loads an .osm/.xml file generated from the post processing step into our enhanced version of iD (this includes both the roads generated by us and the current OSM data).
(3) The Mapper evaluates the roads we generated (styled in a different color) and inspect them for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fixes these issues manually one by one, using DigitalGlobe satellite imagery.
(4) Our modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint. This allows us to check for quality and conflicts with current OSM data prior to submission. Some examples of what we check for include:
- Isolated roads that are not connected to rest of the map.
- Roads crossing waterways.
- Roads with repeating node IDs.
- Roads with possible connections after a short extension.
- Self-intersecting ways.
You can watch the video example of the various data integrity editing features here.
(5) In case of conflicts between our newly added roads and other OSM contributors' changes, our mappers will almost always choose “keep their edits.” If we end up changing another mapper's edits, (for example, to align better with the latest version of DG satellite imagery) we will leave detailed notes to explain why.
(6) Until all errors are fixed, Mappers will not be able t upload to OSM. Here is an example of a highlighted issue prompting the mapper to fix it.
(7) After fixing all the issues detected in the iD tool, the Mapper clicks the “Save local” button to save their edited roads locally for validation.
(8) A Validator then goes into the same task to verify that the data looks correct by re-validating the data, makes necessary changes as needed, and clicks “Submit” to finally upload the task to OSM.
(9) The Validator then conducts a post-submission review of the submitted edits in JOSM. If any errors are detected, the validator addresses the errors and submits necessary changes. A custom JOSM paint style is used to help distinguish our submissions from community submissions, so that all roads with the import=yes tag are highlighted in green. Some examples for what we check for include:
- Short Unclassified ways (between 10-60 meters)
- Highway crossing waterway without a bridge tag
- Highway crossing buildings
- Overlapping ways
- Unspecific highway tags
(10) We will continue to watch all comments made on our edits, reply to them in a timely fashion. We are happy to incorporate feedback and local expertise into our validation process and future mapping behaviors.
Road Type Tagging
During the submission process, our mappers will assign and fix road types according to the general highway tag guidelines of OSM http://wiki.openstreetmap.org/wiki/Key:highway
Before moving to a new location, we will work with mappers from local OSM community in each country to verify our highway tagging method and make sure they align with local conventions.
Some other methods we use to tag roads include:
- Referring to local Wiki guides for each country.
- Looking at commonly used tags in the local area of mapping.
- When connecting to current OSM roads on the map we try to use the same tag where it makes sense to do so.
- Referring to local community for advice.
We use the following tags:
- comment = '#nsroadimport #country'
We use the above changeset tags in every import changeset, where the '#country' tag is the country of the submission. For Thailand the tags are '#nsroadimport #thailand'.
- source = digitalglobe
- import = yes
The 'source' tag indicates that road geometries are based on DigitalGlobe satellite imagery.
Data Merge Workflow
This import (data integration) will be done through a Tasking Manager, by engineers and a group of selected trained map editors. Among the skills required:
- Good experience with iD and JOSM.
- Familiarity with editing norms: how to merge nodes (M), join ways (J), combine ways (C) and un-glueing (G)
- S/he knows how to handle conflicts with both iD tool and JOSM.
The map editing team consists of Mappers and Validators, and their accounts are listed as below.
We are not bots :) The usernames (VLD001, RVR001, etc.) were made to help our internal process where we have paired Mappers and Validators to ensure quality. Please feel free to send us questions individually about edits or email all of us at email@example.com.
Humanitarian OpenStreetMap Team (HOT)
For mapping Indonesia, Facebook has subcontracted to HOT employees, as they have an office in Jakarta; in order to better coordinate with the local community and improve the road network of Indonesia. The team is comprised of:
In summary, the overall mapping workflow is:
(1) Extract grey-scale road mask images from high resolution DigitalGlobe satellite imagery.
(2) enerate road vectors based on the road mask images and merge them with latest OSM data to produce .osm files.
(3) The human mapping team does manual validation on the .osm files through our enhanced iD tool and on JOSM prior to submission (as described in the “Submission Process” above).
(4) After conflict resolution, submission, and post-submission review, work with local OSM community to complete local validation as described in the “Validation Steps” below.
There will be a 4 step
s validation process done by experienced OSM editors including local country Mappers.
- Our method for each country will include starting with small number of edits to gather feedback from the local OSM community before proceeding with larger edits.
- We then start with human validation from our Mappers as explained in the “Submission Process.”
- Before submitting our changes, our team will verify that our updates are based on the current dataset.
- After Submission (as described in the “Submission Process” above, a team of selected Validators and QA's conduct a second post-submission review by passing over chunks of multiple submitted tasks , region by region, by repeating the JOSM validation checks in described in Step 9 under “Submission.”
- Our validation process primarily only includes elements that our team has contributed, which can be identified by the “import=yes” tag. Elements that our team “inherits,” meaning elements that are authored by one of our usernames but lacking the “import=yes” tag, are not typically included in post-submission validation. For more about inheritance, see footnotes.
- We understand inherited versus contributed elements by exploring the ownership of nodes and their parent ways. We can identify inheritance based on authorship and tagging. Roads that are authored by one of our usernames, but does not include the Import=yes tag, is likely an inherited way. Because we are a team of remote mappers, using one data source, we want to avoid overstepping our contributions.
- Community feedback on imported roads is evaluated and incorporated for final edits in this stage as well. All edits can be identified with the changeset tags above and the team will respond to any e-mails or concerns about edits within 48 hours.
To facilitate the validation process, we share our road mask as a tile service that can be used in iD tool and JOSM as a background layer. The URL template of the tile service is distributed on a per-request basis. Please contact firstname.lastname@example.org for requesting a validated URL template.
So far we have prepared road masks for selected areas in Egypt and Thailand. We will announce the availability for other countries here as they become ready.
Local OSM Community Engagement - Using the list servers and wikis we contact local OSM community members before starting to edit in a specific country. Through this we invite members to collaborate and provide valuable local insight and feedback.
To use our tile service in iD:
(1) Click the “Edit custom background” button in iD
(2) Paste the URL template above in the input box of the pop-up dialog and click “OK”
To use our tile service in JOSM:
(1) Click Imagery > Imagery Preferences dialog
(2) In the Imagery providers tab, in the lower right hand side of the dialog box, click on “+TMS” button
(3) Enter the URL template in the input box and provide a name for the new layer
(4) Once added, this new layer will show up under the Imagery list. You can select this layer to be displayed in JOSM
- What does AI mean?
- Artificial Intelligence (AI) is training a computer to do things that require intelligence when done by humans. In this case it involves creating training data for an area by tracing out satellite imagery the same way you would when editing and feeding it to a computer. The output of this is road masks. Here is an example prediction result using deep learning for an area in Thailand.
- Are there limitations with AI?
- Yes, AI and algorithms still have a long way to go to figure things such as assigning tags according to OSM region specific guidelines, figure out the correct OSM relation, add bridges over waterbodies or figure out connections with crossing railways among many other things. Currently all of this is done by our mappers manually.
- Are you adding roads directly to OSM without humans validating?
- Every single road from the AI output will be added only after multiple rounds of human validation. After some processing of the road masks shown above we end up with a .osm file. We then treat this like remote tracing. Using iD editor and/or JOSM mappers make sure that merging with the current data has happened smoothly. They tag each road appropriately, connect them to current OSM edits, fix alignment, when necessary, add sensible changeset comments/notes and resolve conflicts before finally saving where it is merged with the current OSM data.
- Why are you using iD Editor versus JOSM?
- We know JOSM is efficient at what we are trying to do. In fact, we started primarily with JOSM. This said, to make our internal process more efficient, our engineers improved the iD editor so our version has comparable functionality as JOSM. We are working on making this public so we can share it with the community.
- What Imagery are you using?
- We are using DigitalGlobe's Vivid+ which is high-resolution (50cm / pixel, or zoom 18) color-corrected and cloud-free
- Can you share DG imagery?
- Our imagery is very similar to the Maxar imageries available in iD and JOSM. Our exact version used for prediction is available in RapiD as Facebook's Map With AI - Maxar Imagery
- Facebook License with Digital Globe
- This license is necessary to comply with our obligations to Digital Globe, and we think the benefits of supplying this data outweigh the licensing restraints. The data will be under ODBL once it has been contributed to OSM in compliance with the terms.
- There is no contradiction between the OSM terms and the Facebook license with Digital Globe. Our license terms do not apply to data once contributed to OSM and termination of the agreement with Digital Globe would not impact the data that has already been contributed to OSM.
- Is the Imagery free of Cloud Cover?
- The intended output of +Vivid is cloud free, and in most cases, DG meets this. There are some highly cloudy parts of the world where it’s unavoidable. On average though, this product maintains less than 1% cloud cover.
- Who is behind email@example.com?
- There are 4 people behind this email. This helps us collaborate as a team and is the most efficient way for everyone to be on the same page. More people also ensures that no emails are missed. I (Drishtie Patel) as the Program Manager primarily watch this email and respond to queries along with Ming Gao and Saikat Basu.
- Can the Chageset be more specific?
- Yes we can absolutely change that to import=facebook-ai-____. We don't specifically have versions but can include the area or task number for example. Open to suggestions.
- As for adding more details to our edits: there were no changes to current OSM data in this sample and we plan to add notes when we upload as would normally happen when using the Tasking Manager and iD. The sample shared has not been uploaded yet.
- Can you Publish Source Code for ML?
- We cannot share the source code for Machine Learning at this time, but we can and will share our internal tools.
- Can you Publish Source Code for internal tools?
- Yes. We are currently using a version of iD Editor and the HOT Tasking Manager. We are absolutely going to share our tooling. We are currently working on creating a Github where we will post soon.
- Can Facebook Share other data?
- This is clear violation of privacy so we cannot share this kind of information. We are exploring ways to crowdsource road names for OSM and will share that as we have a more concrete plan.
- Why did you choose Thailand?
- Facebook has a high number of users in this country and we would like to improve that map for this community. It is also our mission as a company is to make the world more open and connected and one way we can do is by filling in the missing gaps on the map. We also saw a strong OSM community that we could learn from and engage with while we refine our process for mapping. We are hoping for community feedback as we move forward so we can contribute high quality edits.
- Are you going to Map other countries?
- Yes. We are moving slowly to focus efforts on one country at a time to make sure our process is accepted by the OSM community out of respect for the process.
- What is the Overall Process?
- PHASE 1 - Generating Road Masks
- Training data is created by editors. Example
- Training data used by engineers in ML to produce road masks. Example
- Road masks are processed to remove low confidence predictions and add connections between short breaks.
- PHASE 2 - Creating Road Vectors (.osm files)
- 1. Road masks are then processed using an algorithm from a black and white image to a vector format specifically a .osm file. Example
- 2. Our .osm file is then merged with current OSM changesets. During this process the following things take place
- All current OSM data for an area is merged, keeping full history.
- No changes are made to the current OSM data.
- Duplicated roads are deleted from the ML generated data. Current OSM data is always taken over facebook generated data.
- New ML identified roads are connected to current OSM roads.
- 3. We then have a locally stored file containing both our generated roads and the current OSM data set for a specific area.
- PHASE 3 - Human Validation
- 1. We use the Tasking manager to divide up tasks and create a flow of editing and validation for the Facebook mapping team. Example
- 2. The Mapper picks a task and loads the locally stored .osm file generated from the post processing steps outlined above. Example
- 3. The task is opened up in our enhanced version of iD. (This includes both the roads generated by us and the current OSM data). Example
- 4. Our generated roads are highlighted in a different color so editors can inspect it for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fix these issues manually one by one, using Digital Globe satellite imagery background.
- 5. We do not delete current OSM data, but make typical OSM editing changes where necessary to ensure high quality of the data (more precisely follow imagery, consistent tagging, etc).
- 6. Our modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint. This allows us to check for quality and conflicts with current OSM data. Some examples of what we check for include:
- 7. In case of conflicts between our newly created roads and other OSM editor's mapping, our mapper will almost always choose “keep their edits”. If we think our newly added roads are better aligned with the latest version of DG satellite imagery, we'll contact the other mapper offline to reach resolution. If we end up changing other mapper's edits, we will leave detailed notes to explain why.
- 8. Until all errors are fixed mappers will not be able to save. Here is an example of a highlighted issue prompting the mapper to fix it.
- 9. After fixing all the issues detected, our mapper clicks the “Save Local” button in the iD tool to save their edited roads locally for validation.
- 10. A second person, the validator then goes into the same task to verify that the data looks correct, makes necessary changes as needed and clicks “Submit” to finally upload the tile to OSM.
- Facebook “imported lots of bad data in Egypt without telling anyone, ignoring the Automated Edits code of conduct or import guidelines. The data was of such poor quality that it was entirely reverted.”
- LukeWalsh (28 August 2018). “https://news.ycombinator.com/item?id=17856687”.