JOSM/Plugins/Wikipedia/GSoC 2018

From OpenStreetMap Wiki
< JOSM‎ | Plugins‎ | Wikipedia
Jump to: navigation, search

Over the course of the Google Summer of Code 2018, the JOSM plugin for Wikipedia got extended, especially with regards to Wikidata. On this page the progress of this project is documented.

Work product

Here is a quick overview over my work on the Wikipedia plugin for JOSM over the course of Google Summer of Code 2018. A more detailed summary can be found below in the #Timeline section, for a git changelog listing each and every commit see the before-after comparison on GitLab (and the same comparison for the gradle-josm-plugin).

One of my first steps was to move the project to Git (hosted on GitHub, since July also on GitLab) to make collaboration and working on different features at once easier. The build system was switched to Gradle, using the gradle-josm-plugin (which I created and maintained already before GSoc), which encapsulates common build logic and especially in combination with git reduces the setup required by developers to git clone the code and run Gradle (only requirement to be installed is Java). Translations are now managed in Transifex instead of Launchpad, since Launchpad is regularly slow (see JOSM Logo 2014.svg #8645).

Validator results showing some new rules from the wikipedia plugin

The first coding period had the main theme of adding validation rules for wikipedia=* and wikidata=*:

  • check if the item in wikidata=* is linked to the article in wikipedia=*
  • check if the item in wikidata=* actually exists (not only that the format is valid)
  • if wikipedia=* contains a redirect, suggest the page to which the redirect leads

To implement these validator checks I generalized the classes that create and execute API queries, so the class hierarchy can be easily extended to cover additional APIs or other queries. Also there is now an abstract class for validation tests, that makes it easier to create new rules that make network requests, it deals with splitting the list of objects into chunks and then testing each chunk separately. Another important improvement was the new website that allows to always get the latest development version of the JOSM plugins wikipedia and pt_assistant. In the meantime this has moved to where now only the wikipedia plugin is served.

The wiki layer showing Wikidata items (and some clusters of markers)

In the second coding phase I implemented the wiki layer that displays Wikidata items and Wikipedia articles, with clustering of markers where there are many of them. Also I cleaned up the interface (buttons in Wikipedia dialog, merge properties panels).

The Wikidata info panel showing statements, labels and sitelinks for the selected Wikidata item.

During the third coding phase, I added a Wikidata info panel that displays labels, statements and sitelinks for each Wikidata item that you select (either an OSM object with wikidata=* or an item from the Wikipedia panel). Also I added a new validator check, that checks for Wikidata items that have an unusual type for an OSM object.


Community bonding

April 23rd — April 29th

(all commits to the repository this week)

April 30th — May 6th

May 7th — May 13th

Coding Ⅰ

May 14th — May 20th

(all commits to the repository this week)

May 21th — May 27th

(all commits to the repository this week)

May 28th — June 3rd

  • test the API queries with WireMock // d388cc3 eaedcd0
  • Generalize the classes that make API requests to allow POST requests with the query not in the query string but in the request body (doing batch queries as GET requests could soon reach URL length limits) // 90525b0 71ba157
  • for the gradle-josm-plugin:
    • try to get required JOSM plugins from the Nexus repo, currently not possible due to this Gradle issue (or the corresponding JOSM ticket JOSM Logo 2014.svg #16347) // db4d9f7 d413efd
    • fix a long standing bug, where the JOSM preferences were always reset between two invocations of ./gradlew runJosm. // 74cc640

(all commits to the JOSM-wikipedia repository this week)

(changes to gradle-josm-plugin this week)

June 4th — June 10th

  • add sitematrix query, that checks against the Wikidata API, which Wikimedia sites exist
  • add caching for API queries (at the moment only used for the sitematrix) // e2cd12c 136205e
  • opened tickets JOSM Logo 2014.svg #16373 (pre-fill component field of bug report) and JOSM Logo 2014.svg #16374 (don't always advise to disable plugin) for the bug report process
  • adapt gradle-josm-plugin to changes in Gradle 4.8, use jGit to determine plugin version, #movingtogitlab. Versions v0.4.8 was released // d413efd...c3e7fdc

First evaluation

Coding Ⅱ

June 11th — June 17th

  • make the ApiQueryClient more generic (for future reuse with other APIs than the Wikidata Action API) // b0bc2d0 5eebcb4
  • move the source directories to directories in Maven/Gradle style (src/main/java) // 1bdde16
  • create WikipediaSite for easier usage of the sitematrix query // 963dce4 1d48578
  • add API query to find Wikipedia articles that redirect (will become a validator test later) // dcde48f 4e22aa0
  • add WikiLayer, that displays Wikipedia articles and Wikidata items // eb6a0e4
  • add Notification when download of Wiki elements fails (e.g. area too big) // 0b374eb
  • combine the download buttons of the WikipediaToggleDialog into one single button // 18f2414

June 18th — June 24th

  • fix the Jenkins build, that was broken by moving the source directories and using WireMock 2.18.0 for unit tests // 23f42a7
  • add validator check for Wikipedia redirects // cdd972f fedfa8a
  • gradle-josm-plugin: Run CI also with Java 10, move the remaining tasks from Java to Kotlin // a5f7307...88e7e42

June 25th — July 1st

  • cluster the markers of the wiki layer // 43e7c34
  • highlight selected element in wiki layer // fb534a8
  • move the settings to one common tab in the JOSM preferences // 07bda9d

July 2nd — July 8th

  • move preferences (Sophox API URL and Wikipedia language) to one common preferences panel // Git icon.svg07bda9d
  • switch from CircleCI to GitLab for continuous integration // Git icon.svgd8ecbe4
  • add API query to find instances of a Wikidata "class" from a given list of Wikidata items (using Wikidata Query Service) // Git icon.svg1cad5e6
  • add validator check that uses the API query for unusual Wikidata "classes" // Git icon.svg5659dca
  • use the correct API names in logging messages of the ApiQueryClient (which is reused for the several APIs) // Git icon.svg4430bf7 Git icon.svgd4b09e9
  • fix repaint issue of the preferences panel when switching expert mode on or off // Git icon.svg83ff387

(all commits to the repository this week)

Second evaluation

Coding Ⅲ

July 9th — July 15th

  • fix the validator check for unusual types // Git icon.svg6c4ff3d

(all commits to the repository this week)

July 16th — July 22th

  • add API query to get all labels for a Wikidata item // Git icon.svg6ac45e9
  • reorganize API queries to allow them to return a different type than the root type of the deserialization schema // Git icon.svgf4d4ed4

(all commits to the repository this week)

July 23rd — July 29th

  • add API query to get all claims for a Wikidata item Git icon.svg252efeb
  • add new dialog, which will display info about the selected Wikidata item // Git icon.svg99ae410

(all commits to the repository this week)

July 30th — August 5th

(all commits to the repository this week)

August 6th — August 14th: Code submission & Final evaluation

(all commits to the repository this week)

Ideas for the GSoC project (from before it started)

This list contains some ideas for improvements over the course of the GSoC.

If you have ideas not on this list that you would like to see implemented, you are very welcome to suggest them On the Talk page.


  • Validate "type" (highway, building, tree, …) of objects with wikidata=* against P31 (instance of) on Wikidata
  • Validate that the Wikidata item used in wikidata=* on an OSM object has coordinates JOSM Logo 2014.svg #11541
    • Validate that the coordinates of the OSM object and the Wikidata item are close (relative to the size of the object: say, 5km for a city, 10m for a small building)
  • check that the website=* matches the one on the Wikidata item
  • check that IDs in OSM match those on Wikidata (uic_ref=*954, ref:bag=*981, ref:whc=*757, …)


  • Add Wiki layer
    • show Wikidata items
    • show images from Commons JOSM Logo 2014.svg #8472
  • Show aliases, description and properties from Wikidata in JOSM JOSM Logo 2014.svg #9886 JOSM Logo 2014.svg #14518
  • filter Wikidata items by type (tumuli, runestones, events, things that no longer exist, …), as suggested on the talk page


  • SAXParseException when querying XML-API (broken doctype) JOSM Logo 2014.svg #14000
  • Unhandled timeout exception JOSM Logo 2014.svg #14051