Austin, TX/Buildings Import/Contributor Guide

Most of the content on this page is copied from the very excellent Louisville,_Kentucky/Building_Outlines_Import/Contributor_Guide - we are adopting the basic outline of their import process, but there are a few minor differences. If you notice any errors, please report them or even better, just to jump in and fix them.

So you want to help out with the Austin building import? Great! We could use the help. Here's how.

Overview

This is a work in progress, feel free to add to it or give feedback!

Thank you for your interest in helping with the Austin building import! Here is a guide for how to contribute.

Install and Configure What You Need

The How to start section contains steps that only need to be completed one time at the beginning. This data import process requires a bit more software setup than nomal OSM editing, but we'll try to make it fast and painless.

See our Software Setup Guide.

1) Select a task and initialize JOSM

Now that those preparations are complete, we're ready to start working with the building and address data.

The OSM Tasking Manager is our central tool for coordinating this buildings import project.

Take a task

City divided into census block groups in OSM task manager

One task selected and active.

In order to make sure two people don't work on the same block group at the same time, when you "take" a task the task tracker will mark it as "currently worked on" and not give it to anybody else. Here's how:

Start with the task manager here: http://tasks.openstreetmap.us/project/7 (Note: you should log in with your username_atxbuildings account)
Now you should be able to browse the map of the city and see what areas are complete and what areas need work. You "take" a task by clicking on a block group on the map.
- Grey areas are still available to be imported. Click on one of these grey areas to see the task for that block group.
- Green areas have been uploaded *and* verified (checked by a second set of eyes)
- Yellow areas have been uploaded but not yet verified
Start with areas that you are familiar with - neighborhoods where you have lived or worked, for example
- If you don't care what block group you will work on, you can click on "Contribute" and then "Take a task at random".
Click the green "Start Mapping" button to lock the task. Only one person can lock a task at a time, so this signals to other mappers that you are working on this task. The lock on the task will automatically expire in 2 hours.

Open in JOSM

3 layers in JOSM: Austin data, OSM data, Bing Imagery

Before proceeding farther, make sure JOSM is running (with remote control enabled, as described above)

Now you should see a dropdown to start mapping. Select Edit with JOSM. This will load the pre-existing data from OSM into JOSM in a new layer.
Turn on satellite imagery layer to give you a better frame of reference. Select Imagery -> Bing from the menu bar. The Mapbox Satellite imagery is also good. If imagery is badly misaligned you can adjust it by right clicking the layer and choosing New Offset - this won't affect the building data, only your visuals in JOSM.

2) Work on Data in JOSM

A side note: JOSM can be a bit daunting to use at first. This is OSM's advanced offline editor, which is quite powerful and allows us to integrate nicely with the task manager and import data. So it is the best fit for this import process. But for many other situations we'd likely prefer OSM's more friendly online editors like iD.

This step is the bulk of our work. Now we will use JOSM to review and make any changes or merges to the data before uploading to OSM's database. Until the upload step, no changes are made to the public map, so you can still bail out if you have problems.

General JOSM editing info

If you haven't used JOSM before, there is a nice guide on this wiki that may be helpful. For our purposes JOSM/Basic_editing will contain most of what's needed to get up to speed. And there's another guide here from the folks at learnosm.org. (Of couse you can also just learn by trying, we're here to help)

Tips for Editing

Use a mouse, not a trackpad if you can.
Control + Click will drag the map (or right click).
Two finger slide up or down will zoom the map (or mouse wheel).
Command + Shift + V will paste all copied tags into selected item.
Go to preferences and remap the Delete key to something else (like Eject) if you are on a keyboard without a number pad.
Add commonly used menu items to the top quick bar.
Shift Click selects multiple items at once, Control Click deselects.
Shift Click to select multiple tags at once, right click a tag for more options (copy selected, copy all, etc).

You can watch/listen to a demo of this step from the NYC buildings folks: click here for youtube video.

Importing and Editing Data

The City of Austin buildings and addresses have been split into three files for import, found in the "import links:" comment of the task (under the Contribute tab). Not every task will have all three. To open a file in JOSM, just click on its link.

buildings-to-import.osm

These are the building footprints that are ready to directly import into JOSM. They have been processed to exclude any buildings that intersect with an existing building, but it's possible a building or two has slipped after the data was processed.

addresses-to-import.osm

These are the address nodes that are ready to directly import into JOSM. These are address points that either do not intersect a building (for example an empty lot), or address points for buildings that have multiple addresses (for example a duplex).

addresses-to-conflate.osm

Use the Edit -> Search tool to select all the buildings .

The Configure conflation settings dialog with address nodes and buildings locked in.

The Conflation Plugin window after running Generate Matches.

The conflict dialog where you can merge address nodes to buildings or fix any conflicts.

These are the address nodes that need to be conflated to OSM buildings. These are address points that intersect with an preexisting OSM building (not from the CoA dataset). They need to be conflated to their respective OSM buildings using the JOSM Conflation plugin (JOSM/Plugins/Conflation).

For our purposes we'll be using address points as reference layer in the conflation plugin and the existing buildings will be the subject layer.

Steps to do the conflation:

Make sure the Conflation plugin is installed and the Conflation window has been added to JOSM (via menu UI: Windows -> Conflation).
Click on the addresses-to-conflate.osm link to load the file into JOSM as a separate layer.
Activate the addresses-to-conflate layer so that the green checkmark appears to the left of it.
Select all these address nodes (via menu: Selection -> Select All, or use the equivalent keybind).
Click the Configuration button in the Conflation window to open up the Configure conflation settings dialog.
Click the Freeze button of the Reference section of the Configure conflation settings dialog to lock in the address points as the things we want to conflate from.
Activate the Data Layer layer and open a search dialog (via menu UI: Edit -> Search... or by keybind)
For the search string, use type:way building=*, choose the replace selection option and click the Start search button to select all of the existing buildings.
Click the Freeze button of the Reference section of the Configure conflation settings dialog to lock in the building ways as the things we want to conflate to.
Click the Generate matches button of the Configure conflation settings dialog to generate the matches between address nodes and building ways. You may see a window pop up that contains a java.lang.NullPointerException - you can ignore this, the conflation will still work and the conflation window should show the results.
The Conflation window has three sections: Matches, Reference only, and Subject only - we only care about the first two. Select all the matches in the Matches section (shift+click to select multiple) and then click the Conflate button. (A Conflation confirmation dialog will pop up saying the that tags are used conflictingly, you can choose Do not show again (this session) to avoid having to click through this for each match.)
Then the conflict dialog will show which tags will be applied to buildings. If there are no conflict (already existing addr:* tags), then everything should merge cleanly. Click the Apply button for each match.
Now check the matches in the Reference only section of the Conflation window. These are address nodes that the conflation plugin failed to match to a building for some reason. These need to be manually checked. Often they can be merged into the nearby building, but can happen in weird cases too. For example: an address node falls in the middle of a courtyard of a bldg that has an address already - we don't want to import the node in that case.
Hurray! Run the validator and upload your edits.
Restart JOSM after uploading your edits. It looks like that's the only way to clear out "frozen" Subject and Reference features.

3) Run Validator JOSM tool

JOSM Validation errors. All errors should be fixed, and most warnings.

JOSM has something called a Validator (click for details) that attempts to automatically find common errors and alert us to fixes before we finish.

Run JOSM validator and review any errors/warnings it presents. Typically these only come up in very dense areas with lots of buildings. Most will be about duplicate nodes or ways, which are minor and should be automatically fixed.
- May find these warnings: Crossing buildings; Self-intersecting ways; Building inside building

JOSM Fix Validation errors.

Now fix the Errors automatically with JOSM. Click the Errors folder in the bottom right, and click the Fix button (see image to the right). Then re-validate and there should be no more errors! If there are fix them manually.

If there are still Warnings related to buildings or ways, please fix those manually (right click the warning and click Zoom to see it):

If a way intersects a building, you can modify the road so it goes around the new building. Reference the satellite imagery.
If the interior of a building should be cut out (eg, for a courtyard), the problem is usually that there are 2 interior shapes. Left click and drag to select the entire interior shape, the double click the 'building' selection.
If you have serious doubts about the data's accuracy or suspect a technical glitch, contact the rest of the import team for assistance or a second opinion.

Some errors and warnings are from existing data problems with boundaries, roads, tagging etc. If they were already there and not part of buildings or addresses or roads ovelapping buildings, you cna ignore them.

4) Upload Data to OSM

You can watch/listen to a demo of this step from the NYC buildings folks: click here for youtube video.

JOSM Upload dialog

Note: If you changed the existing OSM user data (ie, removed buildings or addresses), upload the changes for this layer first! Then upload the city data layer from task manager.

Click the upload button which looks like this: on the top button toolbar. This will only upload the active layer, so make sure it's selected!

JOSM Upload progress bar

If you get a Conflict notice, click the far right button and try again.

You'll be faced with a dialog listing the modified objects (likely a lot, in the case of our import). It will also ask you for a brief comment and source. We're going to follow a consistent format for these two fields throughout the import, which is important as it allows us to track our work within this rather large chunk of data. It also clearly documents our work so that others in the OSM community to understand how this data got here.

JOSM Upload Changeset Comment and Tags

Make sure you include the following info in changesets to make them more easy to track.

Comment

If anything special or out of the ordinary happened, include that in the changeset comment. At the very least, each changset should include this line:

This changeset is part of an import of buildings and addresses for the City of Austin and surrounding areas. Full details on the import can be found at https://wiki.openstreetmap.org/wiki/Austin,_TX/Buildings_Import #atx-buildings-import

import

Set the import=yes tag on every changeset.

Source

Source should always be set to:

City of Austin

Mark complete in task manager

Back in your web browser, there is a third box with a field for comment. This is a good place to put qualitative thoughts on how the data seemed to you, which parts you are more or less confident of, or any other notes you think the next person looking over this section should see first.

Then click Mark task as done. Now the area will show as done on the task tracker (colored in yellow).

Congratulations!

Validation

After areas are marked as done on the task manager, a second mapper will check it over to see if everything looks right. This is our final QA/QC check for the import, so be sure to only mark a task as valid if you are completely sure that it is done and error-free. Don't validate your own tasks. This will allow some cross-over so at least two people will need to have signed off on a task. Things to look for before marking a task as valid:

Did everything import?

Kind of obvious, but buildings and addresses for the task should have made it into OSM.

Validation errors and warnings to watch out for

crossing ways OR crossing buildings

meaning: This happens whenever two ways intersect. With imported buildings, this can happen when buildings were imported on top of pre-existing ways or if there was funky import data for whatever reason. If one of the ways involved is not a building we imported, then don't worry about it (you can definitely fix if you want, but it's not a blocker).

how to fix:

Check imagery to see what's going on.
Sometimes the features are just off and can be fixed by moving them as appropriate.
If a way actually passes underneath a building then set layer=* or covered=* tags as needed.

examples:

A toll booth building over a highway. The correct way to map this is to split the highway into segments with nodes joined to the building way where it passes under the building. The segment under the building should have the covered=yes tag set. The building should also have the building=roof tag set.
A creek crossing a building. This often indicates that the creek runs through a culvert underground here. If that is the case, then the creek should be split into segments where it passes underground, and both layer=-1 and tunnel=culvert tags should be set on that segment.

building node within a building way

meaning: There are many variants on this, but happens if a node already existed for a building before the building was imported and now there is a building way and node in the same area.

how to fix: Either merge the tags from the building way and node into the building way, or just leave the node and remove the building tag. If the tag is more specific than building=yes, then set that building value on the building way.

duplicate nodes

meaning: This might indicates that addresses nodes were have been imported twice by accident.

how to fix: You can just run auto-fix on this.

duplicate ways

meaning: This might indicate that buildings were imported twice by accident. If this happened from a duplicate import, there will usually be a ton of them (as many as there are buildings, so at least a few hundred). If the duplicate ways are not buildings, then you can ignore this.

how to fix: Use the JOSM reverter plugin roll back all but one of the duplicate changesets. See the reverter plugin page at JOSM/Plugins/Reverter for instructions on how to do that.

abbreviated street name

meaning: This is a pre-existing error in OSM but we should fix it now since it affects the address we are importing and is important for routing. This occurs when an OSM street name is abbreviated (for example: Lighthouse Landing Dr should be Lighthouse Landing Drive)