OpenStreetMap:Wikibase

From OpenStreetMap Wiki
Jump to: navigation, search
Available languages — Wikibase
Afrikaans Alemannisch aragonés asturianu azərbaycanca Bahasa Indonesia Bahasa Melayu Bân-lâm-gú Basa Jawa Baso Minangkabau bosanski brezhoneg català čeština dansk Deutsch eesti English español Esperanto estremeñu euskara français Frysk Gaeilge Gàidhlig galego Hausa hrvatski Igbo interlingua Interlingue isiXhosa isiZulu íslenska italiano Kiswahili Kreyòl ayisyen kréyòl gwadloupéyen kurdî latviešu Lëtzebuergesch lietuvių magyar Malagasy Malti Nederlands Nedersaksies norsk norsk nynorsk occitan Oromoo oʻzbekcha/ўзбекча Plattdüütsch polski português português do Brasil română shqip slovenčina slovenščina Soomaaliga suomi svenska Tiếng Việt Türkçe Vahcuengh vèneto Wolof Yorùbá Zazaki српски / srpski беларуская български қазақша македонски монгол русский тоҷикӣ українська Ελληνικά Հայերեն ქართული नेपाली मराठी हिन्दी অসমীয়া বাংলা ਪੰਜਾਬੀ ગુજરાતી ଓଡ଼ିଆ தமிழ் తెలుగు ಕನ್ನಡ മലയാളം සිංහල ไทย မြန်မာဘာသာ ລາວ ភាសាខ្មែរ ⵜⴰⵎⴰⵣⵉⵖⵜ አማርኛ 한국어 日本語 中文(简体)‎ 吴语 粵語 中文(繁體)‎ ייִדיש עברית اردو العربية پښتو سنڌي فارسی ދިވެހިބަސް

Goal

This page documents how to store structured tag metadata on this wiki using Wikibase extension - the same software that runs Wikidata. (initial discussion)

Wikibase allows OSM community to store multilingual tag descriptions and community-defined metadata on the OSM wiki in a way useful to both humans and tools.

  • Tools, such as iD editor and Taginfo are now able to get tag information without complex and error-prone parsing of the wiki markup. Eventually the data may include tag suggestions, validation rules, common pitfalls, and more.
  • Data consumers will be able to get structured metadata to help process main OSM database
  • This wiki will be able to show data as info cards and tables, without duplicating and complicated template hackery.

This project's goal is NOT to replace the primary tag storage for the OSM database, nor to use opaque IDs instead of the human readable key=value strings to tag features. We are only trying to improve this wiki's Key:* and Tag:* pages, making them more useful to various tools.

How can I help?

Looking for volunteers...

Community and content
  • Set up a wiki portal, possibly similar to Wikidata's community portal (but simpler), where community can:
    • propose new properties
    • write guidelines/docs
    • discuss Wikibase data structures
  • Create Lua modules to generate tag tables, such as {{Template:Bridge:movable}}, {{Map Features:highway}}, or {{Template:Religions}}.
    • Implementation note: Wikibase only links Tags to the corresponding Key, but Keys do not list all possible Tags. To generate a table, we must have a list of items somewhere. We could create a new WB key property that lists all tags, and use a bot to maintain it, or we could list all needed tags as a template parameter, e.g. for highway, {{...|motorway|trunk|primary|secondary|...}}. List as a template parameter does not need to be localized, and it could specify proper ordering of items (not available in WB). Lua code would use mw.wikibase.getEntityIdForTitle("Key:highway=motorway") to find the right data.
Technical
  • Add Wikibase support to external tools. Simple usage: get key/tag localized description. Complex usage: allow user to add missing or even edit description, especially when user is creating a new key.
  • Port simple validation rules, e.g. regex-based, to use Wikibase data.
  • Help parse various tables of tag data. Even if you can only generate plain files with data, user:Yurik can quickly import them.
tasks in progress
done!
  • Add helper templates, e.g. {{O|Q2}} (link to Tag (Q2)), {{Label|Q2}} (label of the Tag (Q2)). See also Wikidata's Q, label, and other similar templates. Ideally we should have exactly the same functionality, except that we may need to have different template names. Thanks @Teester!!!
  • Create {{Desc|Q2}} (description of the Tag (Q2)) template Thanks @Teester!!!

Tag Keys

Each unique OSM Key is stored as a separate page in the Item namespace:

  • There must be only one item per tag key, e.g. highway=* or landuse=*.
  • The key string will be stored as a permanent key ID (P16)
  • The key string will also be stored as a sitelink, e.g. highway becomes a link to Key:highway page (note that "Key:" is part of the title, and not a wiki namespace)
    • Sitelink will be shown in the upper right corner as a link
    • Ensures each sitelink is unique
    • Allows item's data to be used on the linked page with {{#statements:...}} and Lua
    • Unlike Wikipedia, sitelinks to non-existent Key:* pages is allowed
  • Item must have instance-of property set to Key
  • A bot will create all keys with a significant usage (see below)

Example

See bridge:movable (Q104) that describes a bridge:movable=*:

property type value example description
local sitelink string Key:bridge:movable link to the key:* pages, even if they do not exist
instance of (P2)
table
Item Key (Q7) indicate the type of the item
permanent key ID (P16)
table
String "bridge:movable" shows the exact form of the key as used in OSM. Must never be changed once the item is created.
meant for use on (P5)
table
Items Way (Q4), Area (Q5), Relation (Q6) what kind of OSM objects this key should be used on - e.g. relation, way, area, node
not meant for use on (P24)
table
Items Key (Q7) what kind of OSM objects this key should NOT be used on
image (P4)
table
Commons file MovableBridge roll.gif An image from Wikimedia Commons (Wikibase cannot use images stored on OSM wiki itself)
group (P25)
table
Item bridges (Q4712) The group this item belongs to. In the current model, each key belongs to just one group. In theory we could use it to attach multiple ones, changing the meaning of the "group" to something of a "label"/"meta-tag".
status (P6)
table
  proposal discussion (P11)
  table
Item approved (Q15)
  reference link
community's approval status, together with a reference link to the discussion page (optional)
key type (P9)
table
Item well known values (Q8) what type of key is this? one of well known values, external id, etc.

Optional properties

Here are some additional properties that may be needed in some cases.

name type description
rename-to reference(s) for simple cases (e.g. frequent typos), indicates that this key should be replaced with another key, or possibly one of the other keys(?)
value validation regex (P13) string table
See population (Q574) example.
limited to region qualifier (P26) item table
(must be used as a qualifier of another statement)
See noexit (Q501) example.
excluding region qualifier (P27) item table
(must be used as a qualifier of another statement)
See noexit (Q501) example.

Item Creation Process

An automated tool creates all significantly used tag keys when they are detected in the OSM database (taginfo API). The bot should:

  • create an item for any key with 10+ usages if it matches ^[a-z0-9]+([-:_\.][a-z0-9]+)*$, or for any 1000+ usages regardless of the key syntax (see talk page)
  • set item's label to be the same as the key (can be changed by the user, e.g. name:enEnglish Name )
  • set item's description from the corresponding wiki page's info card (if available, from all languages)
  • set used-by, recommended tags, implies, and any other easy-to-figure-out data from the info cards.
  • should NOT update any fields already set, e.g. if description does not match with wiki, leave it as is.

If the item's key was incorrect, the item should be edited with some metadata to indicate the proper name. Merging items will be prohibited because that would remove the original key (sitelink).

Eventually, it would be better for OSM tools (iD, JOSM, ...) to ask the user for the metadata, and use MW API to create new items.

Tag values

For keys like Key:highway, there is a list of the well-known (enum) values, e.g. highway=residential, highway=service, highway=footway. These items will be stored the same way as for Keys, except that sitelink will point to a Tag:key=value page, the instance-of will be Tag instead of Key, and it will have a link to the corresponding Key item, and it will use permanent tag ID (P19) instead of permanent key ID (P16).

Example

See bridge:movable=bascule (Q888) that describes a bridge:movable=bascule. See all items that link to bridge:movable.

property type value example description
local sitelink string Tag:bridge:movable=bascule link to the tag:* pages, even if they do not exist
instance of (P2) Item Key (Q7) indicate the type of the item
permanent tag ID (P19) String "bridge:movable=bascule" shows the exact form of the tag as used in OSM. Must never be changed once the item is created.
meant for use on (P5) Items Way (Q4), Area (Q5) what kind of OSM objects this key should be used on - e.g. relation, way, area, node
image (P4) Commons file MovableBridge draw.gif An image from Wikimedia Commons (Wikibase cannot use images stored on OSM wiki itself)
tag key (P10) Item bridge:movable (Q104) link to the corresponding key item

Meta item

There will be a number of items that are neither Key nor a Tag. For example, there needs to be several items to represent meta concepts themselves, like Node, Way, Area, Relation, Key, Tag, and perhaps other. All such items are sub-classes of the OSM concept (Q10) meta item.

Additionally, we may want to label each Key with one or more values, e.g. classify keys as belonging to roads / buildings / business / etc, in which case those labels will also have to be meta items.

API access and querying

  • The easiest way for an external tool to get all the data about a key is to use this API call:
https://wiki.openstreetmap.org/w/api.php?action=wbgetentities&sites=wiki&titles=Key:bridge:movable&languages=en|fr
Use languages to filter labels and descriptions to the needed languages.
Add &format=json to get the actual JSON instead of HTML.
Due to MediaWiki limitations, the titles value should be ("Key:" + key).replace('_', ' ').trim(). Use permanent key ID (P16) to get the actual format of the key.
  • Soon the Wikibase data should be made available as downloadable dumps. This will allow us to set up a SPARQL endpoint similar to https://query.wikidata.org, which in turn will help any tools to run complex queries against it.

Quality Control

There are several additional extensions designed to validate Wikibase data, and find items that do not pass validation. Installing such capabilities may not be done in the first deployment stage.

Limitations

  • This Wikibase cannot yet reference local images. So for the moment, all its images must come from the Wikimedia Commons.
  • The sitelink in the upper right corner does not show whether the Tag:* or a Key:* page exists or not
  • All sitelinks must use spaces instead of underscores. API sitelink search does not work otherwise. See permanent key ID (P16) and permanent tag ID (P19) for the correct value. Note that regular Mediawiki Key:* and Tag:* pages have the same issue, and use a special hack to change the title.
  • MediaWiki removes spaces/underscores from the key, so Key:_abc_ would become Key: abc. There are no way to have two items with sitelinks Key:_abc and Key:_abc_ -- they are treated as the same, and fail.

See also