- 1 About
- 2 Goals
- 3 Schedule
- 4 Import Data
- 5 Data Preparation
- 6 Conflation
- 7 Uploaded
- 8 QA
This page is about importing Alberi Monumentali (natural monument trees) dataset for regione Friuli Venezia Giulia (RAFVG).
In this page we consider dataset downloadable at RAFVG catalogue page fetched searching "Alberi Monumentali".
The import is being discussed on the national OSM mailing list and will be the result of consensus there.
This import aims to have a complete and updated set of trees denoted as "natural monument" for the italian territory. Survey took in consideration historic, ecological and dimensional values. This import has been set on a regional size, as a pilot for other italian regions (admin_level=4).
First import will be performed after community audit on shared json preview files. Audit progress will be trackable in project page. Import limited size should take 10-30 days to be accomplished.
Source dataset contains <200 punctual objects (mostly provided by "Regione FVG") which quality is good and spatially accurate. Other sources have less or basic details for tagging. During a pre-audit, few minor spatial errors has been detected (<20m).
Dataset is presented as ESRI shapefiles compressed zip in two proections; UTM 33N has been choosen.
- Language: ita
- Date: 24/04/2018
- Subject: RAFVG - Direzione generale - Servizio paesaggio e biodiversità
- Role: originator
- Web site: www.regione.fvg.it
- Classification: unclassified
- Other constraints: public
Record format and tagging plan
RAFVG dataset table structure will be adapted and pruned thru OpenRefine.
|1||numero_sch||codice univoco scheda albero||unique tree code||01/D014/GO/06||ref:mipaaft|
|3||specie||specie||species||Larix decidua Mill.||taxon|
|4||nome_volga||nome volgare||common name||Larice||species:it|
|5||circonfere||circonferenza (cm)||circumference (cm)||340||circumference/100|
|6||h_stimata||altezza stimata (m)||estimated height (m)||24||height+note|
|7||h_misurata||altezza misurata (m)||measured height (m)||0||height|
|8||tipo||candidato per||candidate for||monumentale/notevole||denotation=natural_monument/landmark|
|10||data_rilie||data rilievo||survey date||2016/03/13||2016-03-13|
- Data source site: RAFVG
- Data license: public as defined in Regione Autonoma Friuli Venezia Giulia site
Prior to uploading, the dataset will be prepared and checked by an audit process.
Source data is presented as shp shape ESRI file in a collection of punctual elements, one for each tree, projection is UTM zone 33N (ETRS89). The following operations has been performed thru QGIS application:
- reprojection using WGS84 datum
- creation of lat & lon columns in DD.DDDDD format
Prior to OSM JSON conversion, some issues require refining operations, documented herein. A summary of actions performed thru OpenRefine:
- Text to number for geo and dimensional columns
- Text to date for timeline columns
- Rename column numero_sch to ref:mipaaft
- Rename column specie to species
- Rename column genere to genus
- Rename column nome_volga to taxon:it
- Rename column circonfere to circumference
- Create column height based on column complementary fields h_stimata + h_misurata
- Create column note if h_stimata has value
- Rename column fonte to source
- Rename column data_rilie to survey:date
- Rename column tipo to denotation
- Mass edit cells in column denotation
- Rename column vincolo to protected
- Mass edit cells in column protected
Conflator input requires json format. Dataset conversion to json is performed thru OpenRefine template documented herein.
Further validation (ie: typos detection) on output json files can be performed thru jsonlint (npm -g install jsonlint).
Up to 2.8 version, Openrefine doesn't manage null values; workaround to remove lines containing nulls:
pi@raspberrypi:~/OSM sed -i -e '/ : null/d' <Openrefine-output-file>.json
Conflation is performed by OSM Conflator. Objects tagged ad natural=tree will be extracted from OSM in a bounding box defined by source dataset. Existing OpenStreetMap data within a range is merged and tags will be added/replaced accordingly to conflator parameter file profile.py. Prior to import, 9 objects has been detected in AOI querying landmark or natural_monument trees.
Conflator output example
pi@raspberrypi:~/OSM conflate -i AlberiMonumentali-FVG-csv.json -o AM.osm -c previewAM.json profile.py 12:02:28 Read 139 items from the dataset 12:02:40 Downloaded 2457 objects from OSM 12:02:42 Matched 14 points 12:02:42 Adding 125 unmatched dataset points 12:02:42 Deleted 0 and retagged 0 unmatched objects from OSM pi@raspberrypi:~/OSM
Dedicated upload account
The account attilaimport will be used to upload community revised .osm files.
Changeset will be tagged with:
- source=RAFVG - Regione Autonoma Friuli Venezia Giulia
- source:date=yyyy-mm-dd (as defined in latest dataset survey)
Import will be managed by the following OSM users:
Step by step operations:
- dataset download
- Qgis reprojection and csv export
- OpenRefine operations
- OpenRefine json export & null's removal
- run conflator
- audit map announcement & publication
- wait for community validation
- conflation re-run
- Upload changeset(s) in OSM
In case of import problems, changeset(s) involved will be reverted using proper reverter
In case some problems will be detected after upload:
This import has been performed using wikidata reconciliation service to better set "species" tag. KEW specific botanical service should be tested.