Digiroad/Digiroad bus stop import

Goal

Digiroad is national road and street database administrated by the Finnish Transport Infrastructure Agency (FTIA). Digiroad is the main database for public transport stop information in Finland. The data is updated continuously by various public officials in municipalities and in the Regional Centers for Economic Development, Transport and the Environment. The purpose of this project is to complement OSM’s bus stop data with data from Digiroad.

According to a rough estimate, 60% of Finnish bus stop data is missing from the OSM database. In this project, the missing data will be provided to the OSM community for further development and application purposes.

Schedule

This project is divided into two phases:

1. The Digiroad bus stop data gathered from the test location (city of Jyväskylä) will be imported during fall 2018. The exact time of the import depends on the OSM community’s feedback.
2. The Digiroad bus stop data gathered from the rest of Finland will be imported after successful and well-received import of the test data. Edit: Import was done successfully in early 2019.

Dates will be added before the imports.

Import Data

Background

Data source site: https://vayla.fi/en/transport-network/data/digiroad

Data license: Creative Commons By 4.0 https://vayla.fi/en/transport-network/data/open-data/terms-of-use

OSM attribution: https://wiki.openstreetmap.org/wiki/Contributors#Finnish_Transport_Agency

ODbL Compliance verified:  yes

OSM Data files for test set

Original data

OSM Data files for full import

Link to the data will be provided after successful import of the test data.

Permissions

This project is organized by the Digiroad operating service and commissioned by the Finnish Transport Infrastructure Agency (FTIA). The Digiroad bus stop data is available on the Digiroad website in multiple different formats, and the entire data set can be downloaded under the Creative Commons By 4.0 license which is the recommended open data license for public authorities in Finland according to the Public Administration Recommendation JHS-189. This license gives the right to distribute, reuse, modify, and further develop your work, even commercially. FTIA must be credited appropriately as the original source of data. After the import to OSM, the stops from Digiroad will be licensed under the Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF). The content of the ODbL is described here. Permission to Incorporate CC BY Data into OpenStreetMap can be found here.

Data description

The Digiroad bus stop data is available on FTIA’s Digiroad bus stops web site in multiple different file formats including Shape and .CSV in the ETRS-TM35FIN coordinate system and GTFS in the WGS84 coordinate system. The data consists of approximately 94 390 bus stops and 33 attributes.

List of attributes for import:  

1. National unique ID
2. Digiroad stop ID
3. Name of the stop (official language of the municipality)
4. Direction
5. Timetable
6. Shelter with advertising bill board
7. Pole (only virtual stops)
8. Bench     
9. Bicycle parking option
10. Passenger information board
11. Lighting
12. Accessibility by wheelchair (indicated in the note tag)

See also Tagging plans below.

The Digiroad bus stop types are listed below.

One single bus stop may be of many stop types:  

1. Local transport (tag: highway=bus_stop)
2. Long-distance transport (tag: highway=bus_stop)
3. Express bus (tag: highway=bus_stop)
4. Virtual stop (tag: highway=bus_stop): Digiroad virtual stops have no physical bus stop (construction or sign) in terrain. Virtual stops are often used by school transport, and for example hotel bus stops are marked as virtual stops in Digiroad. Therefore, the locations of the virtual stops are considered to be useful information for the user. Virtual stops will be tagged with pole=no and shelter=no and info also in note tag.
5. Tram (will not be imported as the data in OSM is already complete)

The date of the data release from the Digiroad database is September 17, 2018 for the test data and **/**, 2018 for the full data.

*The exact date will be confirmed closer to the import date.

Import type

This is a one-time import with preliminary test data set followed by an import of the complete data set. Maintenance of the imported data is assumed to be carried out mainly by the OSM community. For this purpose, a list of the imported stops’ ref:findr tags is shared with the data sets. Ref:findr tag is a unique national ID allocated to each Digiroad stop, and thus, it can be used to identify the right stops.

JOSM will be used to check, validate, upload, and revert changesets if needed by the Digiroad operator that carries out the import.

Data preparation  

Data reduction and simplification

The OSM bus stop data consists of approximately 34 000 stops in Finland. During the import, OSM stops will be treated as master data. Thus, OSM stops will not be removed or edited in anyway. Missing attribute information will not be added to existing OSM stops. For data reduction and simplification, that is based on spatial analysis, coordinate systems are harmonised. To avoid overlapping data, the Digiroad bus stop data will be filtered using GIS software according to the following steps:

Creating common unique ID:

This step is needed for avoiding possible problems with the incorrectly marked or typed ID information, which were found in both data sets.

• If an OSM stop has a stop code that resembles the Digiroad stop code attribute information (Digiroad MATK_TUNN stop code aka OSM ref: tag, or in a few cases Digiroad VALTAK_ID national ID code aka OSM ref:findr tag), the OSM stop code will be combined with the municipality number to create a unique identification number.
• Similar combination ID will also be allocated to each Digiroad stop.
• In addition, the Digiroad MATK_TUNN and the OSM ref:findr or ref: tag will be changed from string to integer (includes removing whitespaces, letters and any other characters than numbers) in order to make comparison easier.

Stops with common ID that match Digiroad

• If an OSM and a Digiroad bus stop in the same municipality have a common unique ID created in step 1-3, and are located less than 200 meters apart (within), the Digiroad stop will be excluded.
• If the distance is greater than 200 meters, the stops will be manually checked to identify doubles and distinguish between separate stops.

Stops without common or matching ID

• As regards the remaining unidentified stops, if an OSM bus stop has a Digiroad stop within a 50-meter radius, the Digiroad stop will be excluded.
• If the distance between an unidentified OSM and a Digiroad stop is greater than 50 meters, the Digiroad stop is assumed to be a separate stop and will be imported.

Tagging plans

The Digiroad attribute information will be harmonized with OSM as comprehensively as possible. Attribute information will be changed to match with OSM (1, 2, 99 to “no”,“yes” and Null). JOSM editor will automatically drop rows with Null tags during conversion to .OSM. The attribute language is changed to English except with the Note: tag which will remain in the original language. Unlike in the OSM bus stop data, the Digiroad stops are located on the center line geometry of the roads. Thus, the direction of travel will be indicated in the direction tag. Direction will be used for moving the stops from the original Digiroad center line geometry to the actual stop shelter or pole location as described in the work flow.

In tagging plans, the following tags found in Taginfo and MapFeatures will be used:

OSM key	Digiroad column name	Description	Type: example
ref:findr	VALTAK_ID	Nationally unique identifier for the stop. This tag has been used in OSM to refer to the Digiroad VALTAK_ID in some cases before, but is generally missing from most of the OSM stops.	Integer: 142790
ref	MATK_TUNN	Digiroad stop ID aka traveller identifier	String: Jy1221
name, name:fi and name:sv	NIMI_SU OR NIMI_RU	Name of the stop in Finnish or Swedish according to the main language in the municipality. Also both name tags.	String
direction	L_SUUNTIMA	Degree between 0 and 360. Describes the stop validity direction.	Integer: 145
departures_board	AIKATAULU	Paper copy of a timetable in a frame attached to the stop wall or post. Provides information on departure times for the routes serving the stop and, where applicable, stop-specific estimated passing times.	String: yes or no
passenger_information_display	S_AIKATAULU	An electronic timetable board in the stop area which provides real-time information for the routes serving the stop.	String: yes or no
shelter	KATOS or MAINOSKAT	A weather shelter located in the stop area for the use of waiting public transport passengers. Shelter can also be a weather shelter displaying advertising, located in the stop area for the use of public transport passengers waiting for transport. In this case, the advertiser will be responsible for the stop maintenance.	String, yes or no
bench	PENKKI	A bench located in the stop area for the use of waiting public transport passengers. Usually placed under the stop shelter.	String: yes or no
pole	PYS_TYYPPI	Only virtual stops and value always no	String: no
lit	VALAISTUS	Lighting	String: yes or no
note	LISATIEDOT and ESTETTOMYY	Additional information about the stop (LISATIEDOT) in the same language as in Digiroad and also accessibility information (special needs and accessibility by wheelchair). The notes and the accessibility information are separated from each other with a semicolon “;” as follows: "LISATIEDOT:Lasi, roska-astia;ESTEETTOMYY: some text". If only one is known there is no semicolon and is marked as follows: "LISATIEDOT:Lasi, roska-astia". Digiroad ESTEETTOMYY: stop is accessible if it enables independent travelling and waiting by special passenger groups, such as wheelchair users. Stop accessibility is affected by the stop and platform structures, station and terminal structures, and accessibility of the timetable information. Virtual stops are also signed here.	String: max length 200
highway=bus_stop	-	See description of Digiroad bus stop types	String
source	-	Digiroad.fi	String
zone:HSL=*	VYOHYKE	Ticketing zone information. HSL is the regional official. * Naming systems of the zones varies in different regions
zone:Oulu=*	VYOHYKE	Ticketing zone information. Oulu is the regional official. * Naming systems of the zones varies in different regions
zone:Linkki=*	VYOHYKE	Ticketing zone information. Linkki is the regional official. * Naming systems of the zones varies in different regions
zone:Foli=*	VYOHYKE	Ticketing zone information. Foli is the regional official. * Naming systems of the zones varies in different regions
zone:Pori=*	VYOHYKE	Ticketing zone information. Pori is the regional official. * Naming systems of the zones varies in different regions
zone:LSL=*	VYOHYKE	Ticketing zone information. LSL is the regional official. * Naming systems of the zones varies in different regions
zone:Nysse=*	VYOHYKE	Ticketing zone information. Nysse is the regional official. * Naming systems of the zones varies in different regions
zone:Vilkku=*	VYOHYKE	Ticketing zone information. Vilkku is the regional official. * Naming systems of the zones varies in different regions
zone:Jojo=*	VYOHYKE	Ticketing zone information. Jojo is the regional official. * Naming systems of the zones varies in different regions
zone:Kotka=*	VYOHYKE	Ticketing zone information. Kotka is the regional official. * Naming systems of the zones varies in different regions
zone:Meri-Lappi=*	VYOHYKE	Ticketing zone information. Meri-Lappi is the regional official. * Naming systems of the zones varies in different regions

Changeset tags

The following changeset tags will be used:

comment=*

created_by= ID, JOSM/version 14178

source=Digiroad.fi

bot = no

source:date = 2018-11-01 or 2019-01-26

type=import

Data transformation

For the reduced Digiroad bus stops data, JOSM/OpenData  plugin will be used to transform .shp to .OSM.

Data transformation results

Here is the link to the test set data in both .shp and .OSM formats.

Please note, that due using .shp file the following column names representing future OSM tags are not complete in test set data .shp file. Tag names are changed manually in JOSM editor after transformation.

• 'ref_findr' (tag: ref:findr)
• 'departures' (tag: departures_board)
• 'advertisin' (tag:advertising=billboard)
• 'passenger_' (passenger_information display)

Here is the link to the full data set:

link will be provided after test set import!

Data merge workflow

Team

The team consists of operators in the Finnish Transport Infrastructure Agency's (FTIA) Digiroad operating service and the local OSM community contact, Erno, who has been the consultant for the project regarding preliminary OSM community opinion.

Please, contact Digiroad operator if you wish to collaborate.

References

This import was announced on the OSM imports@openstreetmap.org mailing list and the talk-fi@openstreetmap.org mailing list,and Finnish OSM forum as well as in the OpenStreetMap Finland Facebook group. During this project, the team has provided the Finnish OSM community with a possibility to work in close cooperation.

Workflow

Test set

• Digiroad operator will create a test set from the Digiroad bus stop data.
• Test set data is reduced according to the Data reduction and simplification plan.
• Overlapping stops are removed from the Digiroad bus stop test data set.
• Digiroad operator verifies that the data reduction has been successful and that the reduced data is ready to be converted to .OSM.
• Digiroad operator converts the reviewed data from .shp to .OSM using JOSM/OpenData plugin .
• Digiroad operator validates and uploads the data using JOSM/version 14289.
• After the import, the Digiroad operator organizes a mapathon event together with the Finnish OSM community. The plan is to have the mapathon in the GIS Expo on 5th of November, 2018. The aim of the mapathon is to begin moving the Digiroad stops from the original center line geometry location to the actual bus stop shelter or pole location as a stop with the highway=bus_stop tag should be. The note information can also be used for adding more specified tags.

Complete data set

• Results of the test set upload will be reviewed with peers and the Finnish OSM community by open conversation in forums.
• Digiroad operator will exclude the test set data
• Resulting data will be reduced according to the Data reduction and simplification plan section earlier in this document. 
• Operator uses a randomly selected sample of data to verify that the data reduction has been successful and that the reduced data is ready to be converted to .OSM.
• Digiroad operator divides the complete data set into smaller files for import.
• Digiroad operator converts the reviewed and divided data from .shp to .OSM using JOSM/OpenData plugin .
• Digiroad operator validates and uploads the data using JOSM/version 14178.
• After the import, the Digiroad operator organizes mapathon events together with the local OSM community in Turku and Lappeenranta and in other cities as well if seen necessary. The aim of the mapathons is to move the Digiroad stops from the original center line geometry location to the actual bus stop shelter or pole location as a stop with the highway=bus_stop tag should be. The note information can also be used for adding more specified tags.

Changeset size policy

The maximum size of one changeset is 10 000 bus stops as explained in the changeset page, and the import will be done in multiple parts.

Revert plans

If anything goes wrong, JOSM/Plugins/Reverter will be used to revert the change set by the Digiroad operator and possibly with help from the local OSM community contact Picea abies

Conflation

The process of merging Digiroad and OSM data is described in detail in the Data reduction and simplification plan. The goal of the merge is to remove all Digiroad stops overlapping with OSM stops before importing the data. 

Test data:

• In the reduced Digiroad import data set, there are 232 bus stops new to OSM.
• 1104 bus stops were excluded from the Digiroad data.

Please note, that the test data set is from urban area, where Finnish OSM community is active thus the number of missing stops is low. In full data set we anticipate that approximately 53 000 stops to be imported mostly in rural areas.

Complete data set

More detailed information on the complete data set will be updated after the test import.

• In the reduced Digiroad import data set, there are 45 000* bus stops new to OSM.*
• 50 000* bus stops were excluded from the Digiroad data.

*) Not accurate

Quality assurance

The Digiroad bus stop locations and attribute information are defined in the original data by the public officials.

The imported bus stop data is reviewed and, if necessary, revised by the Digiroad operator. This will be done in the JOSM with each OSM file before uploading the changes to the OpenStreetMap.

Moving the stops to correct position

*This section should be moved as its own page?*

Test set stops should be moved from the original center line geometry location to the actual bus stop shelter or pole location as a stop with the highway=bus_stop tag should be.

Here are some instructionsfrom mapathon for moving the stops. With full data we will be marking stops that need moving with tag FIXME=Position_estimated

In overpass turbo, zoom to Jyvaskyla (Central Finland) and run query below, to find the imported stops, that have not yet been moved:

node
  [highway=bus_stop]
  [source="Digiroad.fi"]
  [!note]
  ({{bbox}});
out;

For the test set, we have used note=moved tag to indicate that the stop is moved to correct position. For the full data import we are going to tag stops before hand with FIXME=Position_estimated and the tag is deleted after moving.

Status:

Digiroad stops - moving to correct location

Municipality	total imported stops	Left	Status
Jyväskylä	232	0	100% ready. Tested the moving

List of changes in the data after the test import

In the wiki, the data is described as it was after the full import. In below is the list of the changes made. Changes were made due comments from OSM-community and some practical issues. All changed tags are fixed in the test set also.

Tags that were changed:

1. Shelter with advertising bill board tag was not included, because it over writes visualisation of the Highway=bus_stop tag. Whole purpose of this import was to get the stops visualised so this was not done.
2. Name:fi and Name:sv tags were added, due comments from OSM-community
3. FIXME=Position_estimated tag replaces the practise of using note-tag for signalling that stops needs to be placed again, due comments from OSM-community
4. Zone tags were also added: zone:HSL=A where HSL is the official company and A is the actual ticket zone. Practise was copied from Belgia