Austin, TX/Buildings Import

From OpenStreetMap Wiki
Jump to navigation Jump to search

Overview

This page tracks and documents the community effort to import the 2013 building footprints and address points datasets from the City of Austin Data Portal

Code and scripts are in our github repo.

Join discussion at our google group and on slack in the #osmimport channel.

Data

The specific datasets we are looking at are:

  1. 2013 Building Footprints (metadata) - about 585,000 building footprints derived from planimetrics data collected in 2013
  2. Address Points (metadata) - address points (updated monthly)

License

There was some initial confusion about the data license, but that has been cleared up now. These datasets are in the Public Domain and provided without restriction. Quoting the CoA data policy: 

COA data available through Data.AustinTexas.gov is offered free and without restriction.
Data and content created by COA government employees within the scope of their
employment are not subject to copyright protection.

...

Data accessed through Data.AustinTexas.gov do not, and should not, include controls over its
end use.

Participants

This is a list of people who are working on the import, along with each of their import usernames:

Status

Buildings and addresses have been bulk imported and we are currently working on validation and clean up now.

Inspiration

The OSM Import/Guidelines and Automated_Edits/Code_of_Conduct pages spell out the general process and some rules to follow. It might help to look at successful imports from other cities for ideas. The full list is in the Import/Catalogue page, but here are some particularly relevant imports:

The Plan

We'll follow the OSM import guidelines and have our plan documented on this wiki page and reviewed by the folks on the OSM import and OSM US mailing lists.

The general plan is to run the source files through an automated data pipeline that will map features and attributes to their counterparts in the OSM data model. The pipeline will also break the buildings up into smaller pieces that can be manually QA'd and imported by individual volunteers. Then, our team of exquisite volunteers will use JOSM to review, adjust and import the processed data. This review / import step will be the most labor intensive part so the hope is to get lots of people involved and make a party of it.

The buildings dataset and addresses dataset are different enough that it is easier to treat them as independent, rather than try to merge and import the two datasets in the same workflow. Only building footprints that are not already in OpenStreetMap will be imported, but we want to assign addresses to buildings in OSM that do not yet have them set. Also, addresses are continuously updated (at least once a month), so we may want to repeat the address import workflow on a more regular basis.

Mailing List Discussion

Relevant OpenStreetMap mailing list discussions can be found in the following threads:

Import workflow

This section outlines the workflow for volunteers who will be working on the import.

A map of building footprints from the City of Austin, colored by census block group.
Building footprints from City of Austin, color-coded by census block group.

Prerequisites (get cosy with JOSM)

After a thorough review, it looks like the JOSM editor is currently the best tool for an import like this. It provides a platform for efficiently reviewing, adjusting, and uploading data to OSM. The interface is busy and takes some getting used to, but it doesn't take long to ramp up and become a powerful OSM editor, able to import tall buildings in a single bound.

So, install and get comfortable with JOSM. Here are some particularly great resources:

Account and Software Setup

Check out the instructions at the Software Setup page for details on how to install and configure JOSM for the import. You'll also find details on how to create a separate dedicated import account, which will make it easier to keep track of changes.

Importing

See the Austin,_TX/Buildings_Import/Contributor_Guide for a detailed walkthrough of the import process.

The data have been processed and divided up into ready-to-import files for each census block group (one for buildings, one for addresses). These files are in an OSM XML format that can be directly imported into JOSM. Most of the setup/download steps can be automated via the [http://wiki.openstreetmap.org/wiki/JOSM/Plugins/RemoteControl JOSM remote control API to streamline the import process. The general process is this:

  1. Open JOSM, zoom to the census block area and download existing OSM data
  2. Import the file containing building footprints into JOSM, validate and upload
  3. Import the file containing single address nodes into JOSM, validate and upload
  4. Import the file containing address points that need to be merged with preexisting (non-CoA data) OSM buildings into JOSM, merge with the conflation plugin, validate and upload

Changesets

Changesets made during this import should include #atx-buildings-import in the comment and have the following tags set:

import=yes

Source=City of Austin

Data processing

This section details the data processing steps that prepare the data for import.

Buildings

The table below lists the OpenStreetMap tags that are set by the data processing scripts. buildings.FIELD_NAME refers to the field FIELD_NAME of the building footprints shapefile.

OSM tag value
building "yes"
height buildings.MAX_HEIGHT (convert from feet to meters, rounded to cm)
ele buildings.ELEVATION (need to convert from NAVD88 feet to WGS84/EGM96 meters, rounded to cm) [1]

To prepare buildings for import, the data processing scripts work through the following steps:

  1. Download buildings shapefile from City of Austin
  2. Convert buildings shapefile to GeoJSON and transform datum to WGS84
  3. Subdivide buildings into their respective census block group
  4. For each block group, process the data as follows:
    1. Filter out non-building shapes, keeping only shapes where ("Feature" == "Structure")
    2. Filter out any City of Austin buildings that intersect an existing OpenStreetMap building - we only want to import buildings that are not already in OSM
    3. Set property for OSM tag "building=yes"
    4. Use MAX_HEIGHT field to derive OSM tag "height", converting from feet to meters and rounding to cm
    5. Simplify building geometries
    6. Drop any properties that are not OSM tags
    7. Convert GeoJSON to OSM XML using osm-and-geojson

Addresses

The table below lists the OpenStreetMap tags that are set by the data processing scripts. addresses.FIELD_NAME refers to the field FIELD_NAME of the address points shapefile.

OSM tag value
addr:housenumber addresses.ADDRESS
addr:street addresses.PREFIX_DIR + addresses.PRE_TYPE + addresses.STREET_NAM + addresses.STREET_TYP + addresses.SUFFIX_DIR (matched + normalized to nearby highway=*)
coa:place_id addresses.PLACE_ID [2]

To prepare address points for import, the data processing scripts work through the following steps:

  1. Download address points shapefile from City of Austin
  2. Convert address points shapefile to GeoJSON and transform datum to WGS84
  3. Subdivide address points into their respective census block group
  4. For each block group, process the data as follows:
    1. Filter out non-building addresses, only keeping points where ("ADDRESS_TY" == 1)
    2. Set OSM tag "coa:place_id=<PLACE_ID>" to keep track in OSM of CoA-imported addresses
    3. Download OpenStreetMap streets for the census block, and find the OSM street name that matches the address fields for the address. If one can't be found, ignore the address point.
    4. Drop any properties that are not OSM tags
    5. Split and merge addresses as necessary (see: #Merging addresses and buildings for more)
    6. Convert GeoJSON to OSM XML using osm-and-geojson


Merging addresses and buildings

Addresses will be merged with buildings where if there is a 1:1 match (based on spatial intersection) with a building, or imported as individual nodes if not. This means there will be 3 possible files to merge per census block:

  • buildings-to-import.osm: these are the CoA building footprints, with address tags merged to the buildings that there is a match for. If there is no address point or multiple address points that intersect the building, then it won't have address tags merged into it.
  • addresses-to-import.osm: address points (nodes) to import directly into OSM. These are the points that either do not intersect a building or points where multiple points intersect a building.
  • addresses-to-conflate.osm: address points that should be merged into buildings that are pre-existing in OSM (not part of the CoA import). These points need to be explicitly merged with the buildings. They have been pre-filtered so there should be a 1:1 relationship with the existing OSM buildings. The conflation plugin should make this pretty easy.

Follow up

Smallish tasks we can do after the main import:

  1. Apply building height from CoA buildings to pre-existing OSM buildings that do not have a height=* set.
  2. Clean up bad building heights for already-imported buildings (trim to only 2 decimal places)
  3. Merge any free-standing address nodes not imported from CoA - there should only be a few of these see github issue
  4. Clean up any 'building=*' POIs (nodes) - we should have building footprints for all of these after the import

Related Information

Public Art Dataset

Here is a tool that converts the City of Austin public art dataset to OSM format: https://github.com/chip-rosenthal/osm-import-atx-public-art

The resulting OSM file loads into the JOSM application without error, but is not yet ready for import. (There is a README file in the repo that lists some of the issues.)

That data could be imported too, once the rights and process issues are worked out.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Austin,_TX/Buildings_Import&oldid=1869271"