Austin, TX/Buildings Import/Contributor Guide
|Most of the content on this page is copied from the very excellent Louisville,_Kentucky/Building_Outlines_Import/Contributor_Guide - we are adopting the basic outline of their import process, but there are a few minor differences. If you notice any errors, please report them or even better, just to jump in and fix them.|
So you want to help out with the Austin building import? Great! We could use the help. Here's how.
- 1 Overview
- 2 1) Select a task and initialize JOSM
- 3 2) Work on Data in JOSM
- 4 4) Upload Data to OSM
- 5 Validation
- 5.1 Did everything import?
- 5.2 Validation errors and warnings to watch out for
- 5.3 If Your Area is Invalidated
- 5.4 Using geofabrik QA tool
- 5.5 Using osmose QA tool
- 6 How to get help
- 7 More info
- This is a work in progress, feel free to add to it or give feedback!
Thank you for your interest in helping with the Austin building import! Here is a guide for how to contribute.
Install and Configure What You Need
The How to start section contains steps that only need to be completed one time at the beginning. This data import process requires a bit more software setup than nomal OSM editing, but we'll try to make it fast and painless.
See our Software Setup Guide.
1) Select a task and initialize JOSM
Now that those preparations are complete, we're ready to start working with the building and address data.
The OSM Tasking Manager is our central tool for coordinating this buildings import project.
Take a task
In order to make sure two people don't work on the same block group at the same time, when you "take" a task the task tracker will mark it as "currently worked on" and not give it to anybody else. Here's how:
- Start with the task manager here: http://tasks.openstreetmap.us/project/7 (Note: you should log in with your
- Now you should be able to browse the map of the city and see what areas are complete and what areas need work. You "take" a task by clicking on a block group on the map.
- Grey areas are still available to be imported. Click on one of these grey areas to see the task for that block group.
- Green areas have been uploaded *and* verified (checked by a second set of eyes)
- Yellow areas have been uploaded but not yet verified
- Start with areas that you are familiar with - neighborhoods where you have lived or worked, for example
- If you don't care what block group you will work on, you can click on "Contribute" and then "Take a task at random".
- Click the green "Start Mapping" button to lock the task. Only one person can lock a task at a time, so this signals to other mappers that you are working on this task. The lock on the task will automatically expire in 2 hours.
Open in JOSM
Before proceeding farther, make sure JOSM is running (with remote control enabled, as described above)
- Now you should see a dropdown to start mapping. Select
Edit with JOSM. This will load the pre-existing data from OSM into JOSM in a new layer.
- Turn on satellite imagery layer to give you a better frame of reference. Select Imagery -> Bing from the menu bar. The Mapbox Satellite imagery is also good. If imagery is badly misaligned you can adjust it by right clicking the layer and choosing New Offset - this won't affect the building data, only your visuals in JOSM.
2) Work on Data in JOSM
- A side note: JOSM can be a bit daunting to use at first. This is OSM's advanced offline editor, which is quite powerful and allows us to integrate nicely with the task manager and import data. So it is the best fit for this import process. But for many other situations we'd likely prefer OSM's more friendly online editors like iD.
This step is the bulk of our work. Now we will use JOSM to review and make any changes or merges to the data before uploading to OSM's database. Until the upload step, no changes are made to the public map, so you can still bail out if you have problems.
General JOSM editing info
If you haven't used JOSM before, there is a nice guide on this wiki that may be helpful. For our purposes JOSM/Basic_editing will contain most of what's needed to get up to speed. And there's another guide here from the folks at learnosm.org. (Of couse you can also just learn by trying, we're here to help)
Tips for Editing
- Use a mouse, not a trackpad if you can.
- Control + Click will drag the map (or right click).
- Two finger slide up or down will zoom the map (or mouse wheel).
- Command + Shift + V will paste all copied tags into selected item.
- Go to preferences and remap the Delete key to something else (like Eject) if you are on a keyboard without a number pad.
- Add commonly used menu items to the top quick bar.
- Shift Click selects multiple items at once, Control Click deselects.
- Shift Click to select multiple tags at once, right click a tag for more options (copy selected, copy all, etc).
You can watch/listen to a demo of this step from the NYC buildings folks: click here for youtube video.
Importing and Editing Data
The City of Austin buildings and addresses have been split into three files for import, found in the "import links:" comment of the task (under the Contribute tab). Not every task will have all three. To open a file in JOSM, just click on its link.
These are the building footprints that are ready to directly import into JOSM. They have been processed to exclude any buildings that intersect with an existing building, but it's possible a building or two has slipped after the data was processed.
These are the address nodes that are ready to directly import into JOSM. These are address points that either do not intersect a building (for example an empty lot), or address points for buildings that have multiple addresses (for example a duplex).
These are the address nodes that need to be conflated to OSM buildings. These are address points that intersect with an preexisting OSM building (not from the CoA dataset). They need to be conflated to their respective OSM buildings using the JOSM Conflation plugin (JOSM/Plugins/Conflation).
For our purposes we'll be using address points as reference layer in the conflation plugin and the existing buildings will be the subject layer.
Steps to do the conflation:
- Make sure the Conflation plugin is installed and the Conflation window has been added to JOSM (via menu UI: Windows -> Conflation).
- Click on the addresses-to-conflate.osm link to load the file into JOSM as a separate layer.
- Activate the addresses-to-conflate layer so that the green checkmark appears to the left of it.
- Select all these address nodes (via menu: Selection -> Select All, or use the equivalent keybind).
- Click the Configuration button in the Conflation window to open up the Configure conflation settings dialog.
- Click the Freeze button of the Reference section of the Configure conflation settings dialog to lock in the address points as the things we want to conflate from.
- Activate the Data Layer layer and open a search dialog (via menu UI: Edit -> Search... or by keybind)
- For the search string, use
type:way building=*, choose the replace selection option and click the Start search button to select all of the existing buildings.
- Click the Freeze button of the Reference section of the Configure conflation settings dialog to lock in the building ways as the things we want to conflate to.
- Click the Generate matches button of the Configure conflation settings dialog to generate the matches between address nodes and building ways. You may see a window pop up that contains a java.lang.NullPointerException - you can ignore this, the conflation will still work and the conflation window should show the results.
- The Conflation window has three sections: Matches, Reference only, and Subject only - we only care about the first two. Select all the matches in the Matches section (shift+click to select multiple) and then click the Conflate button. (A Conflation confirmation dialog will pop up saying the that tags are used conflictingly, you can choose Do not show again (this session) to avoid having to click through this for each match.)
- Then the conflict dialog will show which tags will be applied to buildings. If there are no conflict (already existing addr:* tags), then everything should merge cleanly. Click the Apply button for each match.
- Now check the matches in the Reference only section of the Conflation window. These are address nodes that the conflation plugin failed to match to a building for some reason. These need to be manually checked. Often they can be merged into the nearby building, but can happen in weird cases too. For example: an address node falls in the middle of a courtyard of a bldg that has an address already - we don't want to import the node in that case.
- Hurray! Run the validator and upload your edits.
- Restart JOSM after uploading your edits. It looks like that's the only way to clear out "frozen" Subject and Reference features.
3) Run Validator JOSM tool
JOSM has something called a Validator (click for details) that attempts to automatically find common errors and alert us to fixes before we finish.
- Run JOSM validator and review any errors/warnings it presents. Typically these only come up in very dense areas with lots of buildings. Most will be about duplicate nodes or ways, which are minor and should be automatically fixed.
- May find these warnings: Crossing buildings; Self-intersecting ways; Building inside building
Now fix the Errors automatically with JOSM. Click the Errors folder in the bottom right, and click the Fix button (see image to the right). Then re-validate and there should be no more errors! If there are fix them manually.
If there are still Warnings related to buildings or ways, please fix those manually (right click the warning and click Zoom to see it):
- If a way intersects a building, you can modify the road so it goes around the new building. Reference the satellite imagery.
- If the interior of a building should be cut out (eg, for a courtyard), the problem is usually that there are 2 interior shapes. Left click and drag to select the entire interior shape, the double click the 'building' selection.
- If you have serious doubts about the data's accuracy or suspect a technical glitch, contact the rest of the import team for assistance or a second opinion.
Some errors and warnings are from existing data problems with boundaries, roads, tagging etc. If they were already there and not part of buildings or addresses or roads ovelapping buildings, you cna ignore them.
4) Upload Data to OSM
You can watch/listen to a demo of this step from the NYC buildings folks: click here for youtube video.
Note: If you changed the existing OSM user data (ie, removed buildings or addresses), upload the changes for this layer first! Then upload the city data layer from task manager.
You'll be faced with a dialog listing the modified objects (likely a lot, in the case of our import). It will also ask you for a brief comment and source. We're going to follow a consistent format for these two fields throughout the import, which is important as it allows us to track our work within this rather large chunk of data. It also clearly documents our work so that others in the OSM community to understand how this data got here.
JOSM Upload Changeset Comment and Tags
Make sure you include the following info in changesets to make them more easy to track.
If anything special or out of the ordinary happened, include that in the changeset comment. At the very least, each changset should include this line:
This changeset is part of an import of buildings and addresses for the City of Austin and surrounding areas. Full details on the import can be found at https://wiki.openstreetmap.org/wiki/Austin,_TX/Buildings_Import #atx-buildings-import
Set the import=yes tag on every changeset.
Source should always be set to:
City of Austin
Mark complete in task manager
Back in your web browser, there is a third box with a field for comment. This is a good place to put qualitative thoughts on how the data seemed to you, which parts you are more or less confident of, or any other notes you think the next person looking over this section should see first.
Then click Mark task as done. Now the area will show as done on the task tracker (colored in yellow).
After areas are marked as done on the task manager, a second mapper will check it over to see if everything looks right. This is our final QA/QC check for the import, so be sure to only mark a task as valid if you are completely sure that it is done and error-free. Don't validate your own tasks. This will allow some cross-over so at least two people will need to have signed off on a task. Things to look for before marking a task as valid:
Did everything import?
Kind of obvious, but buildings and addresses for the task should have made it into OSM.
Validation errors and warnings to watch out for
crossing ways OR crossing buildings
meaning: This happens whenever two ways intersect. With imported buildings, this can happen when buildings were imported on top of pre-existing ways or if there was funky import data for whatever reason. If one of the ways involved is not a building we imported, then don't worry about it (you can definitely fix if you want, but it's not a blocker).
how to fix:
- Check imagery to see what's going on.
- Sometimes the features are just off and can be fixed by moving them as appropriate.
- If a way actually passes underneath a building then set layer=* or covered=* tags as needed.
- A toll booth building over a highway. The correct way to map this is to split the highway into segments with nodes joined to the building way where it passes under the building. The segment under the building should have the covered=yes tag set. The building should also have the building=roof tag set.
- A creek crossing a building. This often indicates that the creek runs through a culvert underground here. If that is the case, then the creek should be split into segments where it passes underground, and both layer=-1 and tunnel=culvert tags should be set on that segment.
building node within a building way
meaning: There are many variants on this, but happens if a node already existed for a building before the building was imported and now there is a building way and node in the same area.
how to fix: Either merge the tags from the building way and node into the building way, or just leave the node and remove the building tag. If the tag is more specific than building=yes, then set that building value on the building way.
meaning: This might indicates that addresses nodes were have been imported twice by accident.
how to fix: You can just run auto-fix on this.
meaning: This might indicate that buildings were imported twice by accident. If this happened from a duplicate import, there will usually be a ton of them (as many as there are buildings, so at least a few hundred). If the duplicate ways are not buildings, then you can ignore this.
how to fix: Use the JOSM reverter plugin roll back all but one of the duplicate changesets. See the reverter plugin page at JOSM/Plugins/Reverter for instructions on how to do that.
abbreviated street name
meaning: This is a pre-existing error in OSM but we should fix it now since it affects the address we are importing and is important for routing. This occurs when an OSM street name is abbreviated (for example: Lighthouse Landing Dr should be Lighthouse Landing Drive)
how to fix:
- Expand the abbreviate of the name= value on the street way
- Also update any address node or building that has the abbreviated street name.
- Open Edit -> Search tool: search for any object with the incorrect addr:street tag (example search string: "addr:street":"Lighthouse Landing Dr") and have the replace selection radio button selected. This will select all the objects with the former name.
- Then use the Tags window to update all these tags at once by setting the correct expanded value for the "addr:street" tag
If Your Area is Invalidated
When an area is invalidated, it is no longer highlighted in the Task Manager. Find it again and click it to see the comments from the admin. If you want to try and fix the errors, just download the JOSM - there is no need to use the .osm file any more since all that data has been sent to OSM already. When you've addressed their concerns upload just the JOSM layer and set the task as done again.
Using geofabrik QA tool
Using osmose QA tool
(replace myname_atxbuildings by your user account)
How to get help
- We are hanging out in the #osmimport channel of the ATX OSG slack: atxosg.slack.com
- Contacting Austin mappers organizing this project: https://wiki.openstreetmap.org/wiki/Austin,_TX/Buildings_Import
- There's a great community of OSM helpers available here: https://help.openstreetmap.org/
- How to add a note asking for others to check: https://www.youtube.com/watch?v=w8bfKy-dv-E&list=UU7finIieWNix2qRsSeIOt-A&index=4