Import/Catalogue/AlberiMonumentali

From OpenStreetMap Wiki
Jump to navigation Jump to search

About

This page is about importing Alberi Monumentali (natural monument trees) from Ministero Politiche Agricole e Forestali (MIPAAF) dataset, which is planned to be updated yearly. Monumental criteria took in consideration historic, ecological and dimensional values.

Notice: until MIPAAF licence update, source data is not compatible with OSM, hence consider this wiki as a preliminary document.

After the Google Map annunced by MIPAAF, some research has been done in order to fetch some better compiled sources. Upon request, a national Excel list has been provided by MIPAAF.

Dataset shall be adapted in order to generate OSM files suitable to be imported in planet.osm. It shall not be a blind import: source data shall be checked by mappers thru an audit support map (example).

The import is being discussed on this thread and results will be considered for consensus.

Goals

This import aims to have a complete and updated set of trees denoted as "natural monument" for the italian territory.

Schedule

First import will be performed after community audit on shared json preview files. Audit progress will be trackable in a project page (example). Import should take 100-200 days to be accomplished, depending on mappers involved in auditing. In order to stimulate local mappers, overall import could be splitted on a regional (admin_level=4) base.

Import Data

Background

Source dataset, provided by italian regional authorities, contains +3500 objects. Spatial accuracy and has to be verified, since it's seems on map preview some errors has been detected. Taking this in account, an overview of the audit map(s) has to be performed.

Resource

Dataset is presented as an Excel Table and a metadata definition pdf document, provided upon email request.

Metadata

Relevant metadata:
elemento item IT EN
titolo della risorsa resource title Elenco degli alberi monumentali d'Italia List of monumental trees of Italy
risorsa online Sito MIPAAF, Pagina 11260 MIPAAF site, Page 11260
descrizione description Dataset degli alberi singoli e insiemi omogenei che compongono l'Elenco degli alberi monumentali d'Italia,censiti ai sensi della Legge n. 10/2013 e del Decreto 23 ottobre 2014 Dataset of sigle or homogeneous cluster tree, surveyed in accordance with Law n.10/2013, Decree 23rd october 2014
informazioni supplementari further info La prima versione del dataset, pubblicata con Dereto dipartimentale del 19/12/2017 - prot.n. 5450 (G.U. 12/02/2018), viene aggiornata con intervallo almeno annuale: i dati in esame fanno riferimento al terzo aggiornamento dell'elenco nazionale approvato con Decreto dirigenziale prot.n. D.Dir. n. 9022657 del 24/07/2020 e pubblicato in G.U. n. 195 del 5-8-2020 del 08/08/2020 First dataset version, publication decree 19/12/2017 - prot.n. 5450 (G.U. [Official Journal] 12/02/2018), shall be update with (at least) annual frequency: herein data refers to third update, approved by executive decree prot.n. D.Dir. n. 9022657 dated 24/07/2020 and published in G.U. [Official Journal] n. 195 08/08/2020
sistema di riferimento dei dati data reference system EPSG:4326 - WGS 84 - geografico 4326 EPSG:4326 - WGS 84 - geographic 4326
data di pubblicazione della risorsa resource publication date 19/12/2017 2017-12-19
data dell'ultima revisione last revised on 20/07/2020 2020-07-20
frequanza aggiornamento update frequency annuale yearly

Record format and tagging plan

MIPAAF dataset table structure will be adapted and pruned thru OpenRefine.

Alberi Monumentali - record format
Name Description:it Description:en Example tagged as
ID PUNTO codice univoco scheda albero unique tree code 01/D014/GO/06 ref:mipaaf=01/D014/GO/06
STATO usato per filtrare used for pruning not used
TIPOLOGIA se non singolo (gruppo o filari)

usato per valorizzare tag note

if note single (cluster or row)

used to set note tag

gruppo omogeneo note=please, map as polygon or way
GENERE genere genus Larix genus(redundant)
SPECIE specie species Larix decidua Mill. taxon=Larix decidua Mill.
NOME VOLG nome volgare common name Larice species:it=Larice
LATITUDINE latitudine EPSG:4326 latitude EPSG:4326 42.026364662872 n/a
LONGITUDIN longitudine EPSG:4326 longitude EPSG:4326 14.369373452387 n/a
QUOTA quota (m) elevation (m) 520 ele=520
AL IS - H albero isolato, altezza (m) single tree height (m) 5.5 height=5.5
AL IS - C albero isolato, altezza (cm) single tree circumference (cm) 125 circumference=1.25
INS - Hmed gruppo o filare, altezza media (m) average height (m) 24 height=24
INS - Cmed circonferenza media (cm) average circumference (cm) 340 circumference=3.4
CR
  • ETA'
  • CIRCONF
  • ALTEZZA
  • AMPIEZZ
  • FM O PO
  • VALORE
  • RARITA'
  • ARCHIT
  • PAESAG
  • STORICO
 criterio della candidatura candidate criteria SI/NO natural_monument:criteria
  • age
  • cricumference
  • height
  • crown
  • shape
  • value
  • rarity
  • plant_architecture
  • landscape
  • historic

Legal

Import Type

Prior to uploading, the dataset will be prepared, conflated and submitted to an audit process. A pilot audit map has been generated for Piemonte region (admin_level=4).

Data Preparation

Source data is presented as xlsx Microsoft Excel file in a collection of punctual elements, one for each tree.

Refining

Prior to OSM JSON conversion, some issues require refining operations, documented herein. A summary of actions performed thru OpenRefine:

  • latitude and longitude decimals reduced to 5
  • removed records which STATO is not "iscritto in elenco" (not registered)
  • note tag set based on TIPOLOGIA
  • leaf_cycle and leaf_type tags set, based on this table compiled from Wikipedia with integration of missing genus
  • circumference tags from centimeters to meters
  • natural_monument:criteria tag set based on CR fields

Exporting

Conflator input requires json format. Dataset conversion to json is performed thru OpenRefine template documented herein.

Further validation (ie: typos detection) on output json files can be performed thru jsonlint (npm -g install jsonlint).

Up to 2.8 version, Openrefine doesn't manage null values; workaround to remove lines containing nulls: 

pi@raspberrypi:~/OSM sed -i -e '/ : null/d' <Openrefine-output-file>.json

Conflation

Conflation is performed by OSM Conflator. Objects tagged ad natural=tree and denotation=natural_monument will be extracted from OSM in a bounding box defined by source dataset. Conflator output shall generate a public audit map for visual review.

Matching trees

Any matching between input dataset and OSM element within a range (defined in profile.py) shall be considered and a proposal for change will be displayed in an audit map as a blue pin.

New trees

Any input dataset tree which has not OSM matches around the above range, will generate a proposal for a new OSM tree and will be displayed in an audit map as a green pin.

Not in dataset

Existing OSM elements which don't have an input dataset match will generate a proposal for a fixme tag; text shall be 'albero non presente in dataset MIPAAF: abbattuto/declassificato?' ('tree not found in MIPAAF dataset: cut down/downgraded?'). They will be displayed in an audit map as a blue pin.

Conflator output example

pi@raspberrypi:~/OSM conflate -i AlberiMonumentaliPiemonte.json  -v -c previewAM.json profile.py
09:27:27 Downloaded 256 objects from OSM 
09:27:28 Matched 163 points 
09:27:28 Adding 56 unmatched dataset points 
09:27:28 Deleted 0 and retagged 93 unmatched objects from OSM
pi@raspberrypi:~/OSM

Conflator re-run

Once audit is completed, online data is downloaded from conflator project page (example) and reprocessed. 

pi@raspberrypi:~/OSM conflate -i AlberiMonumentaliPiemonte.json  -a audit_previewAM.json -o AlmeriMonumentaliPiemonte.osm profile.py
[some echoes...]
pi@raspberrypi:~/OSM

Dedicated upload account

The account attilaimport will be used to upload community revised .osm files.

Changeset Tags

Changeset will be tagged with:

  • source=MIPAAF
  • source:date=2020-07-24

source:description=D.M. N.9022657 del 24/07/2020 - Terzo aggiornamento dell'elenco nazionale degli Alberi Monumentali

Team Approach

Import will be managed by the following OSM users:

  • Cascafico

Workflow

Step by step operations:

  1. dataset download
  2. OpenRefine operations
  3. OpenRefine json export & null's removal
  4. run conflator
  5. audit map (example) announcement & publication
  6. wait for community validation
  7. conflation re-run with audited data
  8. Upload changeset(s) in OSM

In case of import problems, changeset(s) involved will be reverted using proper reverter

Uploaded

Import progress shall be trackable below:

changeset objects notes
12345678 xxx foo

QA

In case some problems will be detected after upload:

Widespread:

  • TBD

Limited:

  • TBD

Possible improvements

This import has been performed using wikidata reconciliation service to better set "species" tag. KEW specific botanical service should be tested.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Import/Catalogue/AlberiMonumentali&oldid=2109941"