Airdrie Alberta Import

From OpenStreetMap Wiki
Jump to navigation Jump to search

Current Status: Completed

Done:

  1. Errors such as duplicate buildings, and buildings crossing highways or landuse were corrected.
  2. Building exists, and its shape resembles, what is in available imagery. Primary imagery sources were Esri World Imagery (2017 image), and the Airdrie 2018 Aerial Photo from the Airdrie open data.
  3. Building type is correctly tagged. Residential buildings were further categorized as apartments, terrace, semidetached_house, and static_caravan where applicable.
  4. Correct address assigned to each building. This was visually inspected by comparing the housenumbers in the OSM data, with the original address data file as a background layer in JOSM.
  5. Addresses for schools were moved from the building outline to the amenity=school area.

ToDo:

  1. Some areas contain a mix of retail, commercial, and industrial businesses. Coverage from Mapillary and OpenStreetCam is sparse, so a detailed ground survey will be needed to tag these buildings in more detail.
  2. There has been a significant amount of detailed mapping done in Airdrie, with the details aligned to Esri World Imagery, with an offset of -2.64;1.02. Ideally, the buildings will be adjusted to match this. That work will be ongoing, but is of low priority.

Goals

The goal of this project is to import the 'Airdrie 2018 Building Footprints' and related 'Airdrie Address Points' data into Open Street Maps.

Schedule

Starting Nov 2, 2019

There are 22351 buildings that will be imported or updated as required. These will be done in blocks of 5000 buildings with one week between each block for quality checks and review.

1 - 5000 - Nov 2, 2019 - COMPLETED - verified

5001 - 10000 - Nov 9, 2019 - COMPLETED - verified

10001 - 15000 - Nov 16, 2019 - COMPLETED - verified

15001 - 20000 - Nov 23, 2019 - COMPLETED - verified

20000 - 22351 - Nov 30, 2019 - COMPLETED - verified


Verification finished Dec 31, 2019. (Verification was carried out neighbourhood by neighbourhood, rather than the imported order.)

Import Data

There are 22351 buildings and 29755 addresses defined in the City of Airdrie data.

The data for buildings comes from the 'Airdrie 2018 Building Footprints' dataset located at: http://data-airdrie.opendata.arcgis.com/datasets/airdrie-2018-building-footprints.

The data for addresses comes from the 'Airdrie Address Points' dataset located at: http://data-airdrie.opendata.arcgis.com/datasets/airdrie-address-points.

A detailed explanation of the import process is noted in 'Data Transformation' section of this wiki page.

Background

Data source site (Buildings): http://data-airdrie.opendata.arcgis.com/datasets/airdrie-2018-building-footprints
Data source site (Addresses): http://data-airdrie.opendata.arcgis.com/datasets/airdrie-address-points
Data license: http://data-airdrie.opendata.arcgis.com/pages/our-open-licence
Contains information licensed under the Open Data Licence – City of Airdrie

OSM attribution: Noted above and verified with City of Airdrie GIS department
ODbL Compliance verified: yes

OSM Data Files

No data files are created. All updates are sent to OSM via the OSM REST API.

Import Type

This is a one time import.

Data will be sent to OSM via the OSM's REST API version 0.6. (https://wiki.openstreetmap.org/wiki/API_v0.6)

Data Preparation

Data Reduction & Simplification

The custom c# program checks to see if the data already exists and if so it will update the way with address and FID data. See Data Transformation section for more details.

Tagging Plans

The following are the tags that will be used for the building ways.

Tag Description Values
addr:city The city the building way belongs to always "Airdrie"
addr:housenumber The civic address for the building (if known) ex: "12354", "121,123,125", "100-121" (multi-civic address)
addr:postcode The Canada Post postal code for the address ex: "T9V 0E4", "T9V" (if multi-postal code address)
addr:province The province the building way belongs to always "AB"
addr:street The full street name for the building ex: "Wayview Crescent SW", "Kings Heights Road NE"
addr:unit The unit number for the building if applicable ex: "10", "20", "1A,2A,3,4,5" (multi-unit address)
building The type of building possible values: "yes","house","garage","shed"
building:levels The number of levels for the building This is a simple formula to calculate the levels -> levels = Math.Round(height/5)
ref:fid The feature id of the building from the Airdrie dataset ex: 1, 16029
height The height of the building in meters ex: 5.7, 5,65 (only to 2 decimal places)

Changeset Tags

<osm>
  <changeset>
    <tag k="created_by" v="David Nicholson"/>
    <tag k="comment" v="Auto creation of missing buildings from City of Airdrie data"/>
    <tag k="source" v="City of Airdrie Open Data - http://data-airdrie.opendata.arcgis.com/datasets/airdrie-2018-building-footprints"/>
    <tag k="bot" v="yes"/>
    <tag k="attribution" v="Contains information licensed under the Open Data Licence – City of Airdrie"/>
    <tag k="license" v="http://data-airdrie.opendata.arcgis.com/pages/our-open-licence"/>
  </changeset>
</osm>

Data Transformation

I have built a custom C# program that does the following steps.

Airdrie Building Footprint Data (Step 1)

  1. Loads in the 22351 building footprints from an REST API call to the Airdrie GIS data.
  2. Stores the above footprint data into a JSON structured file (this will allow me to replay the data with out going back to the API).
  3. Iterates through all building data and finds the address that best matches the building (based on closest lat/lon points to anyone node or 'In The Box').

Sync OSM Program (Core)

  1. Loads in the 29755 Airdrie addresses from a download excel file.
  2. Converts the StreetTypes in the data into valid street types to match the current OSM street names for the City Of Airdrie. ex: WY -> WAY, RD -> ROAD, etc.
  3. Loads in the previously saved (from step 1) Airdrie Building Footprints in to memory.
  4. Iterates through every Building as follows:
    1. Creates an in-memory object to load the building data (includes nodes latitude / longitude of corners) and basic tags - height, area, etc.).
    2. Finds the FID in the address list for this buildings location (if no address found it will still create the building without address information).
    3. Retrieves the nearest objects to the building location from existing OSM data.
      1. Checks if the building already exists or if this buildings footprint will overwrite another building way.
        1. if it might overwrite we record this in a log with the Building's ref:fid and the OSM way id that it might have overwritten and do not create the building.
      2. if the existing OSM building way has an buildings ref:fid tag we compare the tags data and nodes to see if they match.
        1. if they do not match we record this in a log with the buildings ref:fid and the OSM buildings way id and mark the OSM building way for update.
    4. Based on the area size we tag the building as either 'yes','house','garage' or 'shed'. This is !buildingtype!
    5. We add the tags to the in-memory object:
      1. addr:city=Airdrie
      2. addr:housenumber=address:civicnumber
      3. addr:province=AB
      4. addr:street=address:street
      5. addr:unit=address:unit if available
      6. building=!buildingtype!
      7. height=!height!
    6. We look up the postal code for the address (it's not part of the Airdrie Address data) and assign it to the addr:postcode=* tag
    7. We deserialize the in-memory object to the XML payload and call the OSM REST API for create or update.

Data Transformation Results

There is no OSM XML file involved. We are using the OSM REST API to create/update OSM node and way data.

Data Merge Workflow

N/A

Team Approach

Crowd Source Data Verification

Workflow

The changeset size will be set to ~5000 elements (this includes ways, nodes and tags)

A block of 5000 buildings will be imported per process noted below with a period of 1 week between imported block. Total weeks to import all buildings: 5 weeks.

We will run the overall implementation steps will be as follows:

  1. Import 10 buildings (ways) then verify the 10 buildings in OSM are correct and review the log (1 hour)
  2. Import 25 buildings (ways) then verify the 25 buildings in OSM are correct and review the log (2 hours)
  3. Import 100 buildings (ways) then spot check 25 random buildings in OSM are correct and review the log (2 hours)
  4. Import 500 buildings (ways) then spot check 100 random buildings in OSM are correct and review the log (4 hours)
  5. Import remaining buildings (ways) (4365) then spot check 100 random buildings in OSM are correct and review the log (4 hours)

Any issues identified will be logged and a determination of severity will be undertaken. If the severity is critical we will stop the process, determine the issue and retest in the development OSM. A re-plan will be determined later..

We will leave this data in OSM for 1 week before importing the next 5000 buildings. The remaining 5000 buildings batches will perform the exact same steps as that noted above.

If the data is not good we have the process to remove all the ways, nodes and tags.

The creation and the reversal process has been already been executed (2) times in the DEV OSM instance.

Conflation

This import will not overwrite any existing ways/nodes/relations but it may update existing buildings by adding missing tags. It will not update any existing nodes or tags.

This program can be rerun to update any existing ways/nodes that it has created.

The tag 'ref:fid' on the way (building) will be the indicator of a way that was created/updated by this program.

The program also makes a record in a database of the ways and nodes OSM id that it creates for future reference.

QA

The following is a list of manual checks that will be done with for the targeted buildings.

  1. Verify the way is complete and looks to be in the correct placement in OSM.
  2. Verify the tagged data is accurate to the original data.
  3. Verify that the address is correct - City of Airdrie Address Points website - http://data-airdrie.opendata.arcgis.com/datasets/airdrie-address-points/
  4. Verify no errors are showing in OSM for the building.
  5. Verify no other building ways were overwritten or doubled up.

NOTE: We have identified that the current ground maps are not perfectly aligned so the underlying building image may not line up with the building way.

Post Completion Tasks

The following are some of the issues that will need to be addressed by the Airdrie Community users.

  1. There was a (98.5%) hit rate for the address of the building. The remaining buildings will need to be looked at and tagged with the correct addresses. See Verification Website noted below.
  2. Businesses established at buildings cannot be determined from the data so those buildings will need to be updated with the current commercial occupants and other pertinent information. Please update via the OSM website or your OSM editor of choice.
  3. As we are using area size to determine if the building is a house, garage or shed, some buildings may get tagged with the wrong building type. See Verification Website noted below.
  4. The levels are determined by the height of the building and it's pretty rudimentary -> levels = Math.Round(height / 5). Generally this is correct but there maybe instances were it is not.
  5. There may be some apartment and multi-dwelling units that have multiple addresses but only one building. The program should update these buildings correctly but these buildings will need to be verified. See Verification Website noted below.
  6. UPDATE: Street names that are prefixed with their civic address range (example: 100 Bayside Landing SW) are not coded correctly in the City Of Airdrie Address data and the street names in OSM are incorrect. These will be corrected as soon as an ideal resolution is devised before buildings 5001 - 10000 import is started.

Verification Website - We have created a website to verify or reject building ways added by this program so we can crowd-source volunteer verification's of all the building ways added. verification site just created: http://www.nicholsoft.ca/

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Airdrie_Alberta_Import&oldid=1940353"