Mexico Road Name Import Project

From OpenStreetMap Wiki
Jump to navigation Jump to search



Mexico Road Name Import Project is an import of Marco_Geoestadístico_Nacional_(MGN) datasets which are shapefiles from the INEGI datasets covering Mexico.

About

Kaart will conflate data provided by Mexico’s National Institute of Statistics and Geography (INEGI) with OSM. The team dedicated to this import will use Cygnus to conflate the Marco_Geoestadístico_Nacional_(MGN) data set.
Kaart is following the import guidelines and throughout the duration of the project will continue efforts to ensure effective communication with Mexico’s OSM community.

Import Plan Outline

  1. Determine which INEGI data set is most effective for an import → MGN (June 2018)
  2. Develop tools to efficiently and accurately convert .shp files to .osm.pbf and accurately map attributes
  3. Develop processes to edit and validate data using JOSM (including custom paint styles and validators
  4. Conflate MGN data with OSM data using Cygnus one state at a time
  5. Create projects for each state using an instance of the OSM_Tasking_Manager

Goals

To contribute up-to-date data for the road networks of Mexico by updating missing geometry and road names.

Schedule

  • Discussion with Mexico's OSM community - September 2018
  • Announcement to Imports mailing list - September 2018
  • Develop data processing tools - September 2018
  • MGN data conflation - October 2018 → Completion(TBD)
  • Manual editing and validation using JOSM - October 2018 → Completion(TBD)

Progress

Tabasco

Tasking Manager

View Changesets

  • Started Mapping: October 2018
  • Finished Mapping: December 4, 2018

Oaxaca

Tasking Manager

  • Started Mapping: December 4, 2018
  • Ongoing

Mexico City

Tasking Manager

  • Started Mapping: December 12, 2018
  • Finished Mapping: January 8, 2019

Import Data

Background

States and their id's

The data is delivered as a .zip (~2.6GB) of the entire country of Mexico which contains a .zip for each state. Each state has a corresponding number(id) prefixing the name of the state (ex. 01_aguascalientes) as shown in the PDF on the right.

The state .zip contains 3 directories:

Directory Contents
catalogos PDF and .txt files describing the data sets
conjunto de datos "Data Sets" .shp files
metadatos Metadata information

There are 13 different data sets within conjunto de datos for each state (information describing them can be found in contenido.pdf in catalogos).
The data set containing road data is Ejes de vialidad which will be identified by the id prefixing the letter "e" (ex. 01e.shp).

Data source site: MGN data set
Data license: Terms of Use
Type of license (if applicable): LIBRE_USO_MX
OSM attribution (if required): INEGI
ODbL Compliance verified: yes

OSM Data Files

OSM data will be processed and delivered on a state by state basis. If there are any questions or concerns regarding the OSM data, please contact User:Zlavergne (examples of data may be made available).

Import Type

This is a one-time import using a combination of automated tools for data processing/conflation and manual editing/validation prior to upload.

Import Methods

  • Automated tool to prepare data for conflation
  • Automated tool to conflate data and output OSM_XML files
  • Progress tracking will be done using an instance of the OSM_Tasking_Manager
  • Manual processes for editing and validating data in JOSM

Data Preparation

Data Reduction & Simplification

Our import focuses on road names and missing road geometries (we will also add oneway information to missing geometries). As described in the Import Data #Background section, we will be using the Ejes de vialidad data set.

In order to use the Cygnus conflation tool, the .shp files must first be converted to .osm.pbf files that are <20MB and no larger than 50km x 50km areas.
Our team developed a tool to automate the process of splitting and converting. Simply explained below:

Arguments

  • Shapefile
  • Translation file (compatible with ogr2osm)
  • Size of area (in meters)

Tools/Modules

Process

  1. Creates a grid of squares covering extent of Shapefile
  2. Executes spatial analysis to extract Shapefile data within each grid into individual Shapefiles
  3. Converts Shapefile to .osm and translates attributes to OSM tags
  4. Converts .osm file to .osm.pbf


We then run each .osm.pbf file through Cygnus twice:

  • Once with name as the tag of interest without missing geometry
  • Once to generate missing geometry without any tags of interest

We will end up with 2 OSM_XML files for each square. We will use JOSM to merge the name files for all squares of a state into one OSM_XML and missing geometry into another.
Once we have two conflated OSM_XML per state (one for names, one for missing geometry), we can then create an OSM_Tasking_Manager project for the state and use the task bounds to take the appropriate Bounding_box of the conflated files as separate layers into JOSM.

All of these processes have been extensively tested and their outputs will be closely monitored throughout the import process.

Tagging Plans

MNG Field OSM Tag Description (Sp) Description (En)
NOMVIAL name NOMBRE DE VIALIDAD Name of Road
TIPOVIAL highway TIPO DE VIALIDAD Type of Road
SENTIDO oneway SENTIDO DE LA VIALIDAD Sense of the Road
CVE_SEG CLAVE DEL SEGMENTO DE VIALIDAD CONFORME A LAS MANZANAS Key of road segment according to blocks
CVE_VIAL inegi:road_key CLAVE DE LA VIALIDAD Road Key
CVE_LOC CLAVE DE LOCALIDAD Locality Key
CVE_MUN CLAVE DE MUNICIPIO Municipality Key
CVE_ENT CLAVE DE ENTIDAD FEDERATIVA Federal Entity Key

Changeset Tags

We will use the following changeset comments: Importé geometría faltante utilizando datos de INEGI. #Kaart for the missing geometry and Actualicé nombres de vías utilizando datos de INEGI. #Kaart for the road name validation. We will use the changeset source: source=INEGI:MGN:06-2018

Data Transformation

Prior to using the tool described above, we make sure the data is encoded using ISO-8859-1 and is projected to WGS84 using QGIS.

Data Transformation Results

OSM XML files will be created on a state by state basis to keep the data as accurate as possible.

Data Merge Workflow

Team Approach

The development of the tools and the actual import of the data will be handled by the Kaart Team with constant collaboration with the local Mexico community.

Current Editors

Andrew The Mapgician_import
Jon JonKrato_import
Leanne LSkalayo_import
Nick daFisch_import
Robin FreestyleR15_import
Sharon Sherbare_import
Zack the_node_less_traveled_import

References

Workflow

Once the user selects a tasks they will follow the following workflows:

Missing Geometry

  1. Download OSM data within task area using an Overpass query
  2. Merge the missing-geometry layer with the OSM data layer
  3. Manually verify connections
  4. Run JOSM/Validator with the custom validator
  5. Upload changeset using the comment: Mexico Road Geometry Import #Kaart

Road Names

  1. Use the name layer to manually verify road names
  2. Run JOSM/Validator with the custom validator
  3. Upload changeset using the comment: Mexico Road Name Project #Kaart

Any necessary reverting will be handled with JOSM Reverter Plugin.

Conflation

Conflation will be handled by Cygnus and manually verified by our team.

QA

QA will be incorporated into the OSM_Tasking_Manager's validation. We will be using custom validators as well.

You can view changesets using OSMCha.

See also

The email to the Imports mailing list was sent on 2018-09-18 and can be found in the archives of the mailing list at [1].

The email to the Talk-mx mailing list was sent on 2018-10-2 and can be found in the archives of the mailing list at [2].