Ramani Huria OMK Field Data Cleaning Workflow
- 1 Preparing for OMK field data collection
- 2 Field Data Cleaning Procedure
Preparing for OMK field data collection
General documentation from the OpenMapKit website.
- Develop data model.
- Examples for Dar es Salaam, Uganda, Liberia
- Create ODK/OMK (OpenDataKit, OpenMapKit) forms based on step 1 - the Data Model.
- More documentation on xlsform.
- Conversion from .xls to xml: form formart conversion
- Example forms available from Uganda and Liberia
- Constraints on (specific) questions can be applied by creating/adapting the `constraints/default.json` file (found in the OpenMapKit installation directory on a phone), according to the specification at specific questions constrains
- Get a base map (through humanitarian map, or any other way to create an mbtiles map. Such as tl, or Tilemill). An online map can also be used, if connectivity is good.
- Get base layer OSM data (either from the export tool, Overpass turbo or through JOSM) in .osm format
- If through JOSM: make sure it includes feature types you’d like to be able to edit through the forms. ‘File’->’Download from Overpass`. Select proper area, use query: `building=* OR amenity=* OR office=* OR shop=* OR man_made=* OR highway=* OR waterway=*` (or applicable parts of that query).
- Install ODK and OMK on a smartphone
- On the phone, copy forms to ‘odk/forms’, base map to `openmapkit/mbtiles`, base layer osm to `openmapkit/osm`. If custom constraints are used, copy to `openmapkit/constraints`.
- Off you go.. fill in some forms
- Once data collected, grab data from the phone; copy `odk/instances`
- Grab all the .osm files, pull into JOSM, merge layers, check data, upload
Field Data Cleaning Procedure
After field data collection with OMK, clean the data before .
Step 1 - Create folders
Create the folder structure for the data cleaning process.
- Create primary working folder: <omk_field_data_cleaning>
- Create a container folder:
<ward_name>_<current-date>. For example: sinza_2017-09-22.
- Create a container folders:
- Add a folder for the raw data: /01_raw/<your OSM name>
- Add a folder for the working data: /02_working/<your OSM name>
- Add a folder for the final data: /03_final/<your OSM name>
Step 2 - Raw Files
Your supervisor or team leader will download data from the server, and select sections of data for you to clean up.
Note to Supervisor:
- While dividing out this data to the different team members, purge everything else from except what you want to give to a specific team member, and then ‘Save As…’ that section. Do not copy a section of the data downloaded from the OMK server into another new file, as this will not sync while downloading data from OSM database to start cleaning field data. Read about the purge action
- Do this for every team member you are giving data to clean.
- You will need the time-dependent filter to select data on the server over a particular period and give that to the team for cleaning. See some examples
- ‘timestamp:2017-09-22/’ - Will find any object that was modified after date
- ‘timestamp:2017-09-22/201-09-25’ - Will find any object that was modified after date but before other date
For team members:
- The file your team leader gives you will be named as: ‘<ward_name>_<current-date>_raw/<yourOSMusername_raw>’. For example: ‘sinza_2017-09-22_raw_lusdavo.osm’.
- Copy the file your team leader gives you to the ‘<ward_name>_<current-date>/01_raw>’ folder. Format dates as YYYY-MM-DD. For example: ‘sinza_2017-09-22/01_raw/‘sinza_2017-09-22_raw_lusdavo.osm’’.
Step 3 - Add OSM Files to JOSM
Add all the .osm files to JOSM
- Drag all the .osm file into JOSM
Step 4 - Sync Field Data with OSM Data
Do a series of checks on the data to see if there is any issues with the data.
You will be using several filters during your cleaning process. Add the following filters:
- ‘type:node untagged’
- ‘new OR modified’
To prepare your data for cleaning:
- Select collected data using the JOSM filter: ‘type:node untagged’
- Select all results (Ctrl+A)
- Add the selection to the TodoList plugin: on the plugin, click ‘+ Add’
- Deselect results by clicking in the Map View
- Download OSM data in your area, to ensure we merge properly with existing data
- Press ‘2’ on the keyboard to zoom to the extents of your collected data
- Select ‘File -> Download in current view’
- Now save this file to your working folder
- File -> Save As… -> ‘<ward_name>_<current-date>_raw/<working_yourOSMusername>’For example: ‘sinza_2017-09-22_working_lusdavo.osm’.
Step 5 - Cleaning Data
- You are now cleaning the ‘working’ file in JOSM.
- Add Imagery being used to clean data
- From the ToDoList, inspect each feature (select the first one in the TodoList)
- Are the locations correct?
- Are there any spelling mistakes?
- Are the tags capitalised correctly?
- Are the tags fields filled correctly?
- Are we not duplicating existing OSM data?
- If okay, click ‘Mark’ in TodoList to proceed to the next item.
- Add a filter ‘new OR modified’.
- Add imagery
- Uncheck the ‘type:node untagged’ filter
- Select only new or modified data using the JOSM filter: ‘new OR modified’, and check if everything looks okay
- Run ‘Validation’ in the ‘Validate Results’ window.
- Fix any ‘Errors’ and ‘Warnings’ that may be found in the data you have been cleaning
5.1 Record Errors Encountered
Capture common problems that come up
- Team members should record any errors and problems encountered (both while mapping and validating).
- Please record this in a text file with screenshots if possible
- Save to ‘<ward_name>/<current_date>/02_working’, for example ‘sinza/2017-09-22/02_working’, folder.
5.2 Final Data Check
Double check the validated data
- Team Leaders are to review their teams data validation
- If satisfied, get the Group Supervisor to double check the data
5.3 Upload Data
If Group Supervisor authorises, upload the validated data to OSM
- Add your OSMUsername and Password to the JOSM preferences
- Upload adding the #RamaniHuria #MissingMaps comment, and specify Survey as the datasource.
Step 6 - Save Final Files
Save the final loaded layer file to the external hard drive
- Right-click the final layer that was uploaded and select ‘Save As..’
- Save to ‘<ward_name>/<current_date>/03_final’, for example ‘sinza/2017-09-28/03_final’ folder on external hard drive.
* File name: sinza_2017-09-28_final_lusdavo.osm
Step 7 - Get another dataset from you team leader to clean.