Canada Building Import

From OpenStreetMap Wiki
Jump to navigation Jump to search

This import is on hold pending discussion and review.

This page describes a plan to import building footprints in all major Canadian metro areas. Many buildings have already been imported in Ontario, and others are planned for other provinces/metros. Many felt that the import was begun without properly following the import guidelines and without sufficient input from the OSM community[1][2][3]. As a result, it was stopped about two weeks after it started and no new data is being imported while we decide how to proceed.

Now that there are helpful step-by-steps documented in Import steps, below, more-local (in each of the provinces where there are data available to import) volunteer talent can emerge and say "I/we can and will begin this in (Alberta, New Brunswick...)" and the Tasking Manager can move forward. Use the Discussion tab above ("Talk Page") and feel free to more richly detail the Import steps below, if needed.

Updates about Data Quality and Simplification should also be offered in the appropriate sections below.

Background and Introduction

Originally a Statistics Canada endeavor to add to OSM details of non-residential buildings in Ottawa, Canada, OSM has taken initiative to import these national-scale data. It is probably best if a mapper physically "looks" at each building, but that isn't realistic with these (and such a large amount of) already-digital data; this is OSM mapping from imported data, not "on the ground" analysis of buildings and mapping from scribbled notes. The intent is not to wave a magic wand and pour all sorts of data into OSM, it is to get the best blend of these digital data and volunteer contributions of high-quality mapping skills becoming high-quality data in OSM. Canada already enjoys some (limited, but growing) success at doing this (some of these data have already been imported), so let's complete the task! One intent is to add tags to existing buildings; there are many existing buildings on the map (in Ottawa and elsewhere) that can be worked on now. Stats Canada "announced" at http://www.statcan.gc.ca/eng/crowdsourcing, but this is now firmly an OSM project, launching with this Import/Plan.

From work done with Stats Canada's building import for Ottawa, the building data are available under a Canadian Government Open Data License for other municipalities. (See Ottawa's Import wiki Canada:Ontario:Ottawa/Import/Plan). This license has been formally approved by OSM's Legal Working Group. It might not yet have made its way to the Federal Government's Open Data Portal, but that is the intention. The City of Ottawa now has 320,000 buildings of which 48,000 are tagged building=yes. Other buildings have been enriched by the local community with an average of three or four tags per building. From Ottawa, it was determined that buildings mapped using iD are not of sufficiently high quality, so to avoid future cleanup work, use JOSM instead of iD. Recent data sampling indicate the present data are quite accurate (as were Ottawa data), which bodes well for this Plan.

This project morphed from and grew into the Buildings Canada 2020 "initiative" (WikiProject_Canada/Building_Canada_2020). This suffered from a lack of direction and was partially re-written; the results of those efforts turned into this wiki. So it may be instructive to read WikiProject_Canada/Building_Canada_2020 and see if it offers additional useful historical or technical context. Some sections, especially "The data that could be mapped" may still be relevant during the present era of this project. All are welcome to copy/move sections of that (older) wiki into this one, provided they are presently relevant and achieve consensus.

It will be very ambitious to add these data to OSM, but ongoing good direction and communication should yield high-quality results. Use this wiki, its Discussion tab, the Tasking Manager and if necessary, talk-ca to communicate. Thank you for your efforts!

Permission

From Alessandro Alasia (StatCan), posted on talk-ca. Subject: "Open Building Data: an exploratory initiative":

This exploratory initiative aims at enhancing the use and harmonization of open building data from government sources for the purpose of contributing to the creation of a complete, comprehensive and open database of buildings in Canada. The outcome of this exploratory work is a first version of the Open Database of Buildings (ODB), a centralized and harmonized repository of building data made available under the Open Government License - Canada.

This initiative originates from insights taken from the Statistics Canada pilot project on data crowdsourcing, which used OpenStreetMap as a platform for integrating data on building footprints. In addition to the possible benefits of crowdsourcing, that project highlighted the potential of integrating open data from municipal, regional, and provincial governments to meet the needs of official statistics. In its current version (version 1.0), the ODB contains approximately 4.3 million building footprints. https://www.statcan.gc.ca/eng/open-building-data/index

Open Database of Buildings (ODB)

Licence

These data are published by Stats Canada under the Open Government Licence 2.0.

The OSMF Licensing Working Group determined in their meeting on 2017-03-02 that data under the Ottawa Open Data, Licence Version 2.0 (Ottawa ODL 2.0) can be included in the OpenStreetMap dataset and distributed on ODbL 1.0 terms. Quoting from the draft minutes:

The LWG has determined [(1)] that the attribution requirements of the Ottawa ODL 2.0 can be met by adding the required text to the wiki contributor page and corresponding changeset source attribute values, and that there is no downstream attribution requirement, [(2)] that we are not using "Personal Information" as defined in the licence and referenced legislation, and that so licensed material can be included in the OpenStreetMap dataset and distributed on ODbL 1.0 terms.

— OSMF LWG draft minutes, 2017-03-02 Note: The LWG's acceptance of the Ottawa ODL 2.0 or the Canada Open Data Licence 2.0 does not mean that near-identical licences are also acceptable for OSM import. For example, if the fictional City of Rotonto took the exact text of the Ottawa ODL 2.0 and merely replaced instances of “Ottawa” with “Rotonto”, the above minute indicates that the Rotonto ODL would still need LWG approval. The draft minute goes on to say:

In the past the local variants of the OGL Canada have varied widely and have in some cases included additional terms that have made them incompatible with the ODbL and in some instances non-open. For this reason we are not making a blanket statement on other such localised versions of the OGL at this point in time and will continue to review them on a case by case base.

Attribution

Attribute will be tagged in the import changesets:

source=Statistics Canada - Open Building Database

Data

The data used for this import are released by Stats Canada and will be served through a data service in MBTiles Vector Tile 2.0 format, thus you must provide a z,x,y to obtain data. Building Footprints (Polygons) https://data.osmcanada.ca/{z}/{x}/{y}/statscan-buildings.osm

OSM Data Files

The data will be served by https://data.osmcanada.ca/ as JOSM xml files via a link in the tasking manager. See data source site for more information.

Quality Issues in Source Data

There are several data quality issues in the original dataset that need to be addressed before import. These include (but are not limited to):

  • Overlapping buildings with complex geometries, (possibly from a 3d mapping, but there is not enough data to reconstruct this in OSM)
  • Unsimplified ways representing simple, orthogonal buildings (e.g. https://www.openstreetmap.org/way/661770515)
  • Numerous buildings that share walls in dense retail areas are mapped as one giant building spanning the block
  • Orthogonal buildings not squared[4]
  • Oversimplified, poor quality footprints (e.g. https://www.openstreetmap.org/way/661643188)

The scope and extent of these issues will need to be explored and documented more fully, and a plan made to correct or remove bad data from the import.

Simplification

The data from StatsCan may have many duplicate nodes which will need to be cleaned up prior to import with a simplification procedure. How exactly this will be done has not yet been determined. A restart of importation should not begin until this is documented and/or completed. Preliminary analysis indicates that simplification could save several hundred megabytes in Ontario alone[5].

Orthogonalization

As much as 2/3 of the building footprints may lack properly orthogonal angles[6]. Ideally, orthogonalization will only square angles that are very close to 90 degrees, unlike the squaring tool in JOSM which squares all angles. Better algorithms do exist[7], and seem to be implemented in open source GIS software[8].

3D Buildings

Some buildings have what appear to be 3D geometries, though the attributes in the source data are not clear on this. If included, this data will have to follow the guidelines outlined in Simple 3D buildings.

un-simplified building in JOSM
An example of an unsimplified building from StatsCan. 11 nodes are used to represent a rectangle.

Proposed Schedule

An import of these data began in Ontario in late December 2018 and continued until concerns were raised about the quality of the data and the speed of the import[1]. The importers agreed to pause the import while the issues were discussed. That discussion is ongoing at the time of this writing and is now taking place mostly on the talk-ca mailing list. It also takes place right in this wiki and its Discussion tab (Talk Page).

Tag Mapping

The building footprint is the only thing being pulled in; all new buildings will be tagged only building=yes. Tags on buildings with updated geometries should be retained.

Import Plan

Since it requires JOSM to do the import once the data have been cleaned up the actual import will be done by experienced OSM Mappers via the OSMCanada Tasking Manager. The first pass will import the buildings outlines, the second will validate the work. The hope is we can set up a team for each area:

  • coordinator
  • import data preparation
  • QA
  • import execution
  • data enrichment (commercial, residential, etc... tagging)
  • etc..

Then we can see where we have gaps and how to fill them.  Perhaps some municipalities have local mappers who have little or no technical capability to execute the actual import, though will do the tagging of building type and validation if the buildings "look right."  And maybe some folks who did imports before will help in areas where we have no technical expertise.

If bad geometry is present, it should be updated or left alone. The import process can separate existing buildings in OSM vs new buildings and the main goal will be to add missing buildings as opposed to updating all geometries. Sheds will be excluded via a soft coded filter at the end of the URL(a.k.a is modifiable by importer) so that anything that is smaller than 300 sq feet is ignored.

Quality assurance will be done by people using the Tasking Manager as well as Osmose and other QA tools to ensure proper compliance.

Conflation

Buildings already exist in OSM.

We will replace geometry (if really bad), using the replace geometry function in JOSM, with the new data if we determine it is better than the current. This will keep the entire history for the current item as well as keep the same OSM id. The data service can sort out existing buildings vs new ones and there will be two separate links for import. (Huh?)

To avoid assuming, a note should be opened on OpenStreetMap i.e. https://www.openstreetmap.org/note/741666 to request a local survey to avoid erroneous data.

Import process

OSM's building=* wiki documents over 60 values for this key in six categories. Click on building=* and read up on these (2-3 minutes). You might stumble across a building you know about and be able to apply exactly the right tag value! In rare cases the man_made=* key is a more correct tag (for example, lighthouse, wastewater_plant, windmill).

Import is to be performed with OSM Canada Tasking Manager. Below is the recommended workflow for a typical task.

  1. Select and open a task square in JOSM. If it's too big (i.e. too much work or request is too big to load in JOSM) - go back and split the task into smaller squares. Load imagery layer. For most places Esri World Imagery works best.
  2. Load buildings data for your square - click "Click to Download - Buildings" link under the Contribute Tab. Now let's work with the data and prepare it for merging into main OSM layer by tagging buildings with appropriate tags, removing overlapped outlines and fixing any other JOSM errors and warnings.
    1. Apply two filters: building=yes (inverse) and type:node. This will help us hide nodes and already tagged buildings.
    2. Run validation for the entire import data layer. Typically you would see some errors and warnings. You can easily fix all or most errors automatically by clicking on Fix button.
    3. Once all errors are fixed run validation again. Fixing remaining warnings is usually straight-forward. Only exception is "Building inside building" warning. Some municipalities provide outlines for each building level so buildings outlines will appear to be overlapped. Leave the building=yes tag for the main outline and change the building tag to building:part=yes for those building parts that are inside the main outline.
    4. Once all warnings are fixed, classify buildings with appropriate building=* tags based on imagery. Use lasso selection to select multiple buildings if needed. Here are the most common building tags you'll encounter:
  3. Now we are ready to merge buildings layer. Switch back to the data layer and do Edit->Merge Layer. You should have one layer with OSM data and only visible buildings. Select All (Ctrl-A) and run Validation. Now deal with all building-related warnings. Go one-by-one and conflate new outlines with existing buildings. Use Replace Geometry tool from utilsplugin2 plugin to preserve tags and history - select both new and existing outline and press Ctrl-Shift-G.
  4. Run Validation again and fix any remaining highway/building related warnings, such as crossing buildings/highways, etc
  5. Upload changes to OSM.

Task Status

This import is on hold pending a wider review of the above steps and building community of intermediate and advanced mappers to do the importing. Please do not import any data via the links below right now (January 25, 2019). Rather, use the Discussion tab ("Talk Page") to discuss "readiness" to begin importing with a Task in any given province. When one, two or more intermediate or advanced JOSM editors familiar with the high-quality of data that can be achieved by well-importing these data into OSM step up, request that the Tasking Manager activate that province's Task. Then importing can begin in that province using that Task.

Province Tasking Manager Link Completion
Ontario http://tasks.osmcanada.ca/project/145 Stopped
Québec http://tasks.osmcanada.ca/project/146 Stopped
Alberta http://tasks.osmcanada.ca/project/147 Stopped
British Columbia http://tasks.osmcanada.ca/project/148 Stopped
New Brunswick http://tasks.osmcanada.ca/project/149 Stopped
Nova Scotia http://tasks.osmcanada.ca/project/150 Stopped

The Tasking Manager's "home page" (which lists all available tasks) displays progress of "completed" (orange bar to 100%) and "validated" (green bar covering the orange bar). If these are not displayed, searching for "building import" should select them.

References