Import/Maine E911 Addresses

From OpenStreetMap Wiki
Jump to navigation Jump to search

The Maine E911 Address Import is an import of Maine E911 Addresses (covering Maine in the United States). The import is being directed by Alex who is welcoming to feedback and collaborators.

Goals

  • Import addresses from the Maine E911 address set into OpenStreetMaps to improve local routing
  • Meet community expectations of quality, transparency, and collaboration

Current Status

Import is in progress

You can see status of a specific municipality by looking for ChangeSetIds in MaineMunicipalities.json

Schedule

  • July, 2019 - Reviewed existing conflation tools, started developing OsmPipeline for this import
  • August, 2019 - Evaluated the data quality and license of the data source
  • December, 2019 - Finish OsmPipeline project, produce sample results, draft this proposal, and formally request review
  • January, 2020 - Begin processing the import

Outreach, QA, Review & Feedback

I am formally requesting review from these channels

Please critique all parts which you have expertise or interest

  • this proposal
  • OsmPipeline (the software that I wrote to automate most of the data processing)
  • the sample results (see Sample Data)

If there are no unresolved issues, the import will begin in January 2020

How to Respond

I'll be watching for feedback on:

  • the discussion side of this page
  • the email thread
  • the #Maine channel in OSM US Slack

I can be contacted directly by email, OSM message, @blackboxlogic in the OSM US slack, or issues on my github.

Data Source

Website: https://www.maine.gov/geolib/, specifically: https://gis.maine.gov/arcgis/rest/services/Location/Maine_E911_Addresses_Roads_PSAP/MapServer/1
Data license: "Access: Public Use: User assumes risk. NOT FOR EMERGENCY DISPATCH USE."
Type of license (if applicable): Public Domain
Link to permission (if required): https://www.mainelegislature.org/legis/statutes/1/title1sec538.html
OSM attribution (if required): http://wiki.openstreetmap.org/wiki/Contributors#Maine
ODbL Compliance verified: yes

Permissions

Hello, State of Maine GIS data that can be accessed via the Internet without first authenticating as a named user to gain access, is considered information in the public domain. Reasonable efforts have been made to ensure data is complete and accurate at time of publishing, however it is the end user's responsibility to determine suitability for use. Reference: <https://www.mainelegislature.org/legis/statutes/1/title1sec538.html>

You have made reference to Maine Parcels. Please note that Maine Parcels Organized Towns <https://geolibrary-maine.opendata.arcgis.com/datasets/maine-parcels-organized-towns>, Maine Parcels Organized Towns ADB <https://geolibrary-maine.opendata.arcgis.com/datasets/maine-parcels-organized-towns-adb>, and Maine Parcels Unorganized Territory <https://geolibrary-maine.opendata.arcgis.com/datasets/maine-parcels-unorganized-territory> are not the authoritative source for address information. Address information is available in Maine E911 Addresses <https://geolibrary-maine.opendata.arcgis.com/datasets/maine-e911-addresses>

Thank you for contacting the Maine Office of Information Technology. Sincerely, Todd M

Data Processing Plan

All data manipulation is performed using OsmPipeline (a c# tool I wrote for this import). The import tools and process allow for future imports of updated data without conflict. The following is an explanation of its design and intended use.

Partitioning the Region

This process will be run individually per municipality in Maine (one town at a time). Processing a large town can take a few hours to manually review the change. There are nearly 700 municipalities in Maine, but most are very small. After completing a municipality, I will note the progress.

Data Source Translation

  • The user selects a target municipality and a folder is created with this name to hold the resulting data files
  • The reference data for this municipality is fetched from Maine's E911 API and saved in ReferenceRaw.GeoJson
  • Points are omitted if they are on the municipality's Blacklist in the progress report
    • This is a curated list of points that failed any validation step and, upon manual review, were declared undesirable (invalid, inaccurate or redundant)
  • The data is validated (see Translation Validation below). Errors are printed to console for manual review
  • Points are translated from GeoJson features into OSM nodes (see Translating Tags below) and saved in ReferenceTranslated.osm
  • Similar addresses are combined (if tags only differ by addr:unit=* and level=* and are within 3 meters of each other)
    • The effect is to turn an apartment building with multiple units into a single address, but not to change a trailer park
    • Nodes at the exact same location (but with tags preventing them from being combined) are nudged slightly to de-stack them
  • Reference processing is finished. The results are saved in Reference.osm

Translating Tags

Translation Validation

Elements that don't pass these validation rules produce output in console for manual review

  • OBJECTIDs must be unique
  • The [street] SUFFIX, PREDIR, and POSTDIR must be in the list of expected values
  • STREETNAME must be entirely spaces and alphabetical characters
  • FLOOR and BUILDING can be made into a number
  • ZIPCODE is 5 numerals
  • ADDRESS_NUMBER is numeric and great than zero
  • LAT, LON must not be zero

Conflating into OSM

  • The reference's bounding box (plus a small margin) is calculated, and this map region is fetched from OSM's API and saved in Subject.osm
    • If the region is too large to fetch (over 50,000) nodes, then the region is automatically subdivided and retried.
  • Subject elements (nodes, ways and relations) are matched and merged with Reference nodes (see Conflation Validation below)
    • Matches are found by address tags addr:street=* and addr:housenumber=*, then by exclusive location within a building=* without addr:*=* tags
      • When a match is found, the new tags are applied to the existing element. If the existing element is a node, it is moved to the position of the reference node
    • Otherwise, unmatched nodes are just added
    • Three files are created for easy reviewing, but these files should never be committed to OSM because they contain Review Tags
      • Conflated.Review.osm has elements which failed validation
      • Conflated.Create.osm has new elements which are added
      • Conflated.Modify.osm has existing elements which were modified, (including child elements for ways and relations for proper rendering)
      • Review Tags are customize tags to make manual review simpler. They are never committed to OSM
        • maineE911id={id} - The original objectID(s) from the E911 data-set
        • maineE911id:{key}=added - This {key} was added
        • maineE911id:{key}=change from {oldValue} - This key was changed
        • maineE911id:moved={distance} meters {direction} - The element has been moved this far in this direction
        • maineE911id:info={validation message} - Details which might help a reviewer understand why software made a decision
        • maineE911id:warn={validation message} - The reason that the element failed validation. This element will be included unless blacklisted
        • maineE911id:error={validation message} - The reason that the element failed validation. This element will be excluded unless whitelisted
        • maineE911id:whitelist=yes - Indicates that the element has been manually included in the change even though failing validation
    • An OsmChange file is created Conflated.osc to be committed to the OsmApi after the user has finished manual review

Conflation Validation

Elements that don't pass these validation rules are "exceptional" and placed into their own map layer for manual review

  • Matches by address tags must be within 30 meters (configurable, this may change) of the centroid of the other element
  • Matches by address tags most not match multiple subject elements
  • Nodes must have a positive, non-zero addr:housenumber=*
  • While merging, tag conflicts are a validation error, except for the building tag, which can be changed to a more specific descendant tag
  • The finished change file is checked for being under 10,000 changes (the OSM API limit)
  • Measure if most points were moved in the same direction and warn that the data seems shifted
  • Validate that there is a highway=* with the same name=* as this addr:street=*

Manual Review

  • A human reviews the console output and three review files as layers in JOSM, along side Subject.osm and background imagery
    • The JSOM validation will be run for each layer
    • Special attention is taken to the review file, as these elements have failed validation. They can be manually blacklisted, whitelisted, edited in iD, or ignored. From experience, many of the exceptions come from incorrect data in OSM, which I correct during this step.
    • If adjustments are made, the user selects specific steps to re-run and re-evaluation (minimizing load on exterior resources)

Committing the Change

Only after the change has been reviewed, the user will direct OsmPipeline to commit the change file to OSM's API. The commit author will be OSM user [osmuser:blackboxlogic+bot/ blackboxlogic+bot] using basic authentication. If the commit is too large for OSM, it is broken into pieces, then committed.

ChangeSet Tags

Sample Data

This is an example of the results (covering only Westbrook Maine) which has not yet been committed to OSM. The full import will cover the entire state of Maine.

Here is the Westbrook OsmChange file which is [a sample of] the final product but I found it was not easy to review an osc file. I recommend reviewing the following .osm files which group the changes by change type. They also include the child elements for modified ways and relations (which wouldn't render in JOSM otherwise), and include Review Tags.

  • Nodes in Westbrook which will be Added
  • Existing elements in Westbrook which will be Modified
  • A layer of Review (Elements that failed validation) Each element has a 'warn' or 'error' tag describing the problem

Risks & Known Issues

  • New address nodes may be placed near existing Points Of Interest without combining them since I have no way to confidently determine that the POI is related to this address
  • I have no practice using the JOSM revert plugin, which I hope not to need
  • I could be blocked or throttled from OSM API or Maine's API for exceeding "normal" use patterns
  • Each municipality may have its own data quality issues. I may need to refine the process or skip entire towns if they are too mangled. For example: I've noticed that Lewiston has many addresses missing ADRESSS_NUMBER
  • While matching addresses which appear inside an address-less building, the program may fail to calculate inside in edge cases where the building is defined by a relation and the outer ring is defined by more than 2 open ways which combine to form a single closed way. There are only 50 address-less buildings in Maine defined by relations. And only a small fraction of these are defined with multiple open outer ways. Fixing this isn't worth the development time
  • If the import is re-run (for example: a year later to include fresh data), more development may be needed to prevent a situation where a point was added, an OSM mapper deleted the point, and the point will be re-added. This could be mitigated by pre-filtering the data source by last-modified date.
  • Many roads in Maine are missing or mis-named. I'm mostly ignoring the issue for this import. The [GeoFabric Address Validation](https://tools.geofabrik.de/osmi/#) layer is going to get very messy.

Further Readings

A similar import in MA: Import/Catalogue/MassGIS Addresses