Import/Catalogue/TorriTelefonia-RAFVG

From OpenStreetMap Wiki
Jump to navigation Jump to search

About

This page is about importing mobile phone communication towers dataset published by regione Friuli Venezia Giulia (RAFVG), Italy.

Dataset shall be adapted in order to generate OSM files suitable to be imported in planet.osm. It shall not be a blind import: source data shall be checked by mappers thru audit support maps.

The import is being discussed on the regional OSM mailing list. The import will be the result of consensus there.

Goals

This import aims to have a complete and updated set of mobile phone communication towers for regional territory.

Schedule

This import has been splitted in 4 provinces (admin_level=6): Pordenone, Udine, Gorizia, Trieste. The first as a pilot import for remaining Friuli Venezia Giulia provinces. Pordenone import shall be performed after community audit on shared json preview files. Audit progress will be trackable in project page. Import limited size should take 1 week to be accomplished. Remaining provinces schedule may vary, depending on towers density.

Import Data

Background

Source dataset contains 2500 punctual objects; locations provided by "Regione FVG" source, at a first review seem spatially accurate. During audit process, few minior spatial errors may be detected and adjusted.

Metadata

  • Language: ita
  • Date: 11 marzo 2021
  • Subject: Radiazioni Impianti di telefonia mobile
  • Role: originator
  • Web site: www.dati.friuliveneziagiulia.it

Legal

Record format and tagging plan

RAFVG torri telefonia mobile - record format
Field Name Description:it Description:en Example tagged as
1 OPERATORE nome dell'operatore operator's name TELECOM ITALIA MOBILE operator
2 DATA data inizio operatività operation start date 19 March 2018 start_date
7 ATTIVAZIONE stato status REALIZZATO used as filter
8 PROVINCIA provincia del sito site province PN used as filter
9 COMUNE comune del sito site municipality SAGRADO not used
10 ETRS89EST longitudine proiettata projected longitude 347822.3 converted to DD
11 ETRS89NORD latitudine proiettata projected latitude 5060555.2 converted to DD

Besides operator and start_date tags, the following shall be added:

Whenever appropriate, a fixme "man_made=tower" shall be reported during the audit phase.

Import Type

The dataset will be imported on a provincial base (OSM admin_level=6). Prior to upload, each osm preview file will be published and linked in this page to be manually checked by local teams.

Data Preparation

The data is presented as csv file in a collection of punctual elements, one for each tower. Required reprojection has been performed by QGIS.

Refining

Prior to osm>json conversion, some issues require refining operations (OpenRefine), documented herein. A summary of actions performed thru OpenRefine:

  • date conversion to OSM format
  • coordinate rounding to 6th decimal
  • operator titlecase
  • local id (required by conflation process)

Exporting

Conflator input requires json format. Dataset conversion to json is performed thru OpenRefine template documented herein.

Further validation on output json files can be performed thru jsonlint (npm -g install jsonlint).

Up to 2.8 version, Openrefine doesn't manage null values; workaround to remove lines containing nulls: 

pi@raspberrypi:~/OSM sed -i -e '/ : null/d' <Openrefine-output-file>.json

Conflation

Conflation is performed by OSM Conflator. Objects tagged as

  • "man_made"~"tower|mast" AND
  • "tower:type"="communication" AND
  • "communication:mobile_phone"="yes"

will be extracted from OSM in a bounding box defined by source dataset. Existing OpenStreetMap data within a range is merged and tags will be added/replaced accordingly to conflator parameter file profile.py.

Json file resulting from conflation shall be community revised on an audit map. Upon audit completion, an osm file shall be generated by further conflator run.

Conflator output example

pi@rpi3: conflate -i pordenone.json -v -c preview_pordenone.json --osm pordenone.osm  profile.py
10:14:37 Loading profile <_io.TextIOWrapper name='profile.py' mode='r' encoding='UTF-8'>
10:14:37 Dataset points duplicate each other: 2 and 1608
10:14:37 Dataset points duplicate each other: 24 and 278
10:14:37 Dataset points duplicate each other: 44 and 138
10:14:37 Dataset points duplicate each other: 47 and 458
10:14:37 Dataset points duplicate each other: 50 and 317
10:14:38 Found 145 duplicates in the dataset
10:14:38 Read 429 items from the dataset
10:14:38 Downloaded 48 objects from OSM
10:14:39 Matched 40 points
10:14:39 Removed 145 unmatched duplicates
10:14:39 Adding 244 unmatched dataset points
10:14:39 Deleted 0 and retagged 0 unmatched objects from OSM

Conflator re-run after audit

pi@rpi3: conflate -i pordenone.json -a audit_FVG-PN-towers.json -o pn-ready.osm --osm pordenone.osm profile.py
10:39:26 Dataset points duplicate each other: 2 and 1608
10:39:26 Dataset points duplicate each other: 24 and 278
10:39:26 Dataset points duplicate each other: 44 and 138
10:39:26 Dataset points duplicate each other: 47 and 458
10:39:26 Dataset points duplicate each other: 50 and 317
10:39:26 Found 145 duplicates in the dataset
10:39:26 Read 429 items from the dataset
10:39:26 Downloaded 48 objects from OSM
10:39:27 Matched 40 points
10:39:27 Removed 145 unmatched duplicates
10:39:27 Adding 244 unmatched dataset points
10:39:27 Deleted 0 and retagged 0 unmatched objects from OSM

Upload

Dedicated upload account

The account cascafico will be used to upload community revised .osm files.

Changeset Tags

Changeset will be tagged with:

Team Approach

Import will be managed by the following OSM users:

  • Cascafico

Workflow

Step by step operations:

  1. dataset download
  2. OpenRefine operations
  3. OpenRefine json export (iterate provinces)
    1. run conflator
    2. audit map announcement & publication
    3. wait for community validation
    4. conflation re-run
    5. Upload changeset(s) in OSM

In case of import problems, changeset involved will be reverted using proper reverter

Candidates

file province uploaded changeset
pn-ready.osm Pordenone yes 102174566
go-ready.osm Gorizia yes 102483861
ts-ready.osm Trieste yes 103184192
ud-ready.osm Udine yes 103185936

QA

In case some problems will be detected after upload:

Widespread:

  • TBD

Limited:

  • TBD
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Import/Catalogue/TorriTelefonia-RAFVG&oldid=2149937"