Talk:Sophox Editor

Missing URL

I think a link to this tool is missing - or hidden, which would also be a problem. --Kocio (talk) 06:44, 7 November 2017 (UTC)

Kocio, what do you think would be the best way to link to it? http://sophox.org/sophox/ is the "power-tool" - editor to create and test sparql queries. But the majority of the less techy users would use one of the Quick fixes - because there they all link to the result view of the queries. --Yurik (talk) 07:11, 7 November 2017 (UTC)

For me it's good now. --Kocio (talk) 10:32, 7 November 2017 (UTC)

automated edit box

While providing some degrees of automation, this tool does not allow automated editing, as defined in Automated Edits code of conduct and Automated edits. I have removed the Import bot warning, as this is not applicable here. We may want to introduce a slightly different box, specifically for task-oriented tools (like MapRoulette), warning users to read-up about each task before attempting to make any edits. --Yurik (talk) 22:38, 14 November 2017 (UTC)

User:Imagico, please don't add something without any discussion. I have posted here and on your user page. You keep adding it without any discussion. Note that JOSM can be used for bot editing to a much higher degree that Sophox. Unless you also introduce it there, and in all other task-oriented fixing tools, it sounds like you are prejudice towards this specific tool. --Yurik (talk) 23:38, 14 November 2017 (UTC)

Sophox

Sophox https seems to be down and all links to it are broken. http seems to be fine. --Zache (talk) 11:50, 25 June 2018 (UTC)

Yes, sadly, Sophox main engine - Blazegraph - has been misbehaving. It refuses to load the entire OSM and WD data all at once, so we are rebuilding it. No known ETA yet, but hopefully we can get it up and running again. Turns out Blazegraph is not very happy with the indexes approaching 1TB of data :( --Yurik (talk) 16:58, 25 June 2018 (UTC)

Any news about this topic? I really miss sophox. --Hofoen (talk) 04:46, 25 October 2018 (UTC)

Hardware arrangements have been made, coming soon! --Yurik (talk) 10:27, 25 October 2018 (UTC)

Hofoen, Sophox is back up. --Yurik (talk) 18:59, 28 November 2018 (UTC)

Great to hear - but the wikidata tables are empty, aren't they? (e.g. "?item wdt:P31 ?instance" gives no relsuts) --Hofoen (talk) 12:10, 3 December 2018 (UTC)

Hofoen, yes, you now have to use federation to access Wikidata (or any other source). See docs. Use https://sophox.org/sparql as the endpoint. We should update our docs to reflect this change (help is welcome -- we will have to revise many of the examples, and some might not work the same way due to performance/timeout limitations). Lastly, just for the sake of accuracy - there is no such thing as wikidata tables - RDF databases have just a single 3-column table for everything (unless you have multiple ones in different namespaces, but in that case this is no different than federation). In this case I imported OSM data, OSM meta data (keys/tags), and Wikipedia pageview stats. Here's an example of getting "instance of":

SELECT * WHERE {
  ?osmid osmt:place "city";
         osmt:wikidata ?wd.
  SERVICE <https://query.wikidata.org/sparql> {
    ?wd wdt:P31 ?instanceof.
  }
}
LIMIT 10

--Yurik (talk) 16:39, 3 December 2018 (UTC)

Due to performance federate queries are no real option for me. In my opinion, the main advantage of your service was the fact, that both osm and wikidate were in the same RDF database. Do you have plans to integrate wikidata data again into your RDF database? --Hofoen (talk) 16:43, 6 December 2018 (UTC)

Hofoen, I have been considering it - the issue is that Blazegraph index seems to be growing far too big and the initial data import process just comes to a crawl after about 70% of Wikidata import. The same happens when I try to import OSM data after Wikidata. Perhaps there are some magical java memory optimization params or Blazegraph settings that would allow me add them both, but I have spent weeks on this problem and haven't found the solution. You are welcome to try it - all the code is in https://github.com/sophox/sophox - you can even build your local one with a small data subset within minutes. --Yurik (talk) 19:41, 6 December 2018 (UTC)

Thanks for advice, but I have no suitable machine to do such tests. Nonetheless: I could not find the scripts to sync wikidata in your repo. --Hofoen (talk) 11:14, 10 December 2018 (UTC)

Hofoen, the import scripts are part of the Wikidata setup itself, see loading the dump. --Yurik (talk) 15:55, 10 December 2018 (UTC)

Sophox question

@Yurik: Is this a question for you?

Regards --Ryzen (talk) 21:51, 3 July 2018 (UTC)

@Ryzen:, thanks for the ping! Yes, but sadly Sophox is still being rebuilt, trying to find spare time for it - have hit a hard problem. --Yurik (talk) 06:27, 5 July 2018 (UTC)

@Ryzen:, updated the answer with a working example. Sophox is back up. --Yurik (talk) 18:59, 28 November 2018 (UTC)

Unstable Endpoint

The endpoint seems to be unstable. I'm accessing it from wikidata-query-service. Yesterday I could get result, but today I get mostly internal errors "Response = 404 page not found". Sometimes I could get results using "LIMIT 1". Even accessing it directly with "curl -X POST https://sophox.org/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' --data-urlencode 'format=json'" results in "404 page not found". With wikidata I use the following query:

PREFIX osmt: <https://wiki.openstreetmap.org/wiki/Key:>
SELECT * WHERE {
  ?wd wdt:P31 wd:Q12280.
  ?wd wdt:P17 wd:Q183.
  FILTER(NOT EXISTS { ?wd wdt:P576 ?x1. })
  FILTER(NOT EXISTS { SERVICE <https://sophox.org/sparql> { ?osmid osmt:wikidata ?wd. } })
}
LIMIT 1

--Hofoen (talk) 11:04, 10 December 2018 (UTC)

Hofoen the endpoint is (relatively) stable, but it does not allow POST requests, only GET. I will need to check with Wikidata people why POST is required for federation, and if I can safely enable it (I don't want the whole world to be able to modify the data directly :) --Yurik (talk) 16:02, 10 December 2018 (UTC)

So if the query above returns results, wikidata used GET to access sophox, otherwise POST? This would explain, that sometimes "limit 1" works, as only very few data has to be transferrred (possible with a GET request). Bigger data sizes should be transferred with a POST request. Am I right? --Hofoen (talk) 16:59, 10 December 2018 (UTC)

Hard to say, but it could be. I do not know too much about the code responsible for forming the GET/POST request. I am still exploring it -- apparently there is a X-BIGDATA-READ-ONLY header that disabled write access via POST -- this is something that WDQS uses itself. I tried to enable it, but I think something else is missing... Still exploring. --Yurik (talk) 17:47, 10 December 2018 (UTC)

Fixed. --Yurik (talk) 19:59, 10 December 2018 (UTC)

Thanks for the great work. --Hofoen (talk) 05:38, 11 December 2018 (UTC)

osmID with wikidata link not being reported in query

user:Yurik , Please see https://www.mediawiki.org/wiki/Talk:Wikidata_Query_Service#osmID_with_wikidata_link_not_being_reported_in_query and respond here or there. --Arjunaraoc (talk) 10:03, 27 December 2019 (UTC)

How do I restrict a task/query to a specific area like e.g. country?

This is the query. --PangoSE (talk) 11:02, 13 January 2020 (UTC)

@PangoSE: Currently, it’s only possible to restrict a query to a bounding box or a radius around a point using the built-in wikibase:box and wikibase:around services, respectively. This issue requests geographic functions that would make it possible to filter a query down to features within a nonrectangular polygon literal, but that request would imply some additional spatial indexing that would probably have to be figured out first. – Minh Nguyễn ^💬 17:16, 10 July 2020 (UTC)

Queries need to be updated to use federation

Hi, it seems the queries under Current Challenges and Using from JOSM need to be updated to work now that we have no local WD-data anymore. Can someone provide a working example?--PangoSE (talk) 14:38, 13 January 2020 (UTC)

PREFIX wd: <http://www.wikidata.org/entity/>
#PREFIX wds: <http://www.wikidata.org/entity/statement/>
#PREFIX wdv: <http://www.wikidata.org/value/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>

#defaultView:Map
SELECT ?osmId ?wdLabel ?wd ?wp ?loc WHERE {
  # Limit to subjects that have a tag called "wikidata", and show its location
  ?osmId osmt:wikidata ?wd ;
         osmm:loc ?loc .

  # ?wd must be an "instance of" a disambiguation page, or an instance
  # of some type, which itself is a (sub-)*subclass of a disambig page.
  SERVICE <http://query.wikidata.org/sparql> 
          {?wd wdt:P31/wdt:P279* wd:Q4167410 .}

  OPTIONAL { ?osmId osmt:wikipedia ?wp . } 
}
LIMIT 10

Run it (edit query)

This timed out for some reason.--PangoSE (talk) 14:57, 13 January 2020 (UTC)

You should use https, then it works for me at least. The following query does not time out and returns more then dismabiguation pages, but feel free, to only include those, that works as well here:

#defaultView:Map
select ?object ?item ?coordinates where {
  ?object osmt:wikidata ?item;
          osmm:loc ?coordinates.

  service <https://query.wikidata.org/sparql> {
    values ?forbiddenTypes {
      wd:Q4167410  # disambiguation page
      wd:Q12139612 # list inkl. Q13406463
      wd:Q5        # human
      wd:Q36774    # webpage
      wd:Q215627   # person
      wd:Q729      # animal
      wd:Q8253     # fiction
      wd:Q16521    # taxon
      wd:Q167270   # brand
      wd:Q507619   # chain store
      wd:Q732577   # publication
    }
    ?item wdt:P31/wdt:P279* ?forbiddenTypes.
  }
}
limit 10

Run it (edit query) CamelCaseNick (talk) 00:24, 17 January 2020 (UTC)

Also take a look at the query optimizations/hints, i.e. hint:Query hint:optimizer "None" . could be useful in many cases. --Yurik (talk) 00:38, 17 January 2020 (UTC)

I tried this query but it times out with or without optimizer and with or without the images. Can Sophox render images from image= or wikimedia_commons= tags? Can we increase the timeout limit?

#defaultView:Map
select distinct ?object ?item ?itemLabel ?image ?coordinates where {
  # disable join order optimizer for this group graph pattern.
  hint:Query hint:optimizer "None" .
  
  ?object osmt:wikidata ?item;
          osmm:loc ?coordinates.

  service <https://query.wikidata.org/sparql> {
    values ?forbiddenTypes {
      wd:Q4167410  # disambiguation page
      wd:Q12139612 # list inkl. Q13406463
      wd:Q5        # human
      wd:Q36774    # webpage
      wd:Q215627   # person
      wd:Q729      # animal
      wd:Q8253     # fiction
      wd:Q16521    # taxon
      wd:Q167270   # brand
      wd:Q507619   # chain store
      wd:Q732577   # publication
    }
    ?item wdt:P31/wdt:P279* ?forbiddenTypes;
          #wdt:P18 ?image;
          rdfs:label ?itemLabel.
		  FILTER(LANG(?itemLabel) = "en").
  }
}
limit 10

Run it (edit query)--PangoSE (talk) 10:39, 17 January 2020 (UTC)

Sophox database updates

The sophox database was updated for the acual OSM data every few minutes till last week and wasn't since then. Does anybody know, if the update cycle is now a week or more or is there a problem with it? CamelCaseNick (talk) 00:44, 17 January 2020 (UTC)

Thanks @CamelCaseNick:, i restarted the updater - seems like it froze without any errors. Should catch up in the next few hours. --Yurik (talk) 00:50, 17 January 2020 (UTC)

#defaultView:Editor doesn't work

The following query works as expected but when adding:

 #defaultView:Editor{"taskId":"fill_wikipedia_from_wikidata", "comment":"Add wikipedia tag from wikidata info."}

as a first line, there is no way to visualitze nor edit the results. What's wrong?

SELECT DISTINCT 
  ?osm  ?wd ?name ?nameCA ?wpCAname ?loc ?distance
  (osmt:wikipedia as ?tag_1) (?wpCAname as ?val_1) 
WHERE{
  ?osm osmt:wikidata ?wd ;
       osmt:name:ca ?nameCA ;
       osmt:name ?name ;
       osmm:loc ?loc .
  FILTER NOT EXISTS{ ?osm osmt:wikipedia ?wikipedia . }

  # disable join order optimizer for this group graph pattern.
  hint:Query hint:optimizer "None" .

  service <https://query.wikidata.org/sparql> {
    ?wd wdt:P625 ?wikiLoc .
    ?article schema:about ?wd .
    ?article schema:inLanguage ?lang.
    ?article schema:name ?wpName.
    FILTER(?lang = "ca")
  }

  BIND(CONCAT("ca:", ?wpName) AS ?wpCAname)
  BIND(geof:distance(?wikiLoc, ?loc) as ?distance)
  FILTER(?distance < 1)
} LIMIT 10

Run it (edit query)

WARNING: it can take 2'5 minutes to run the query.

--Jmaspons (talk) 10:42, 19 October 2020 (UTC)

@Jmaspons: The reason it does not work is, that the OSM element to be edited has to be named ?id in the result. Secondly your query can contain Wikinews articles and not just Wikipedia articles and thirdly the query optimizer helps in most cases. The following query works, only returns Wikipedia articles, runs in about 15 s and returns 50 results instead of 10

#defaultView:Editor{ "taskId": "fill_wikipedia_from_wikidata", "comment": "Add wikipedia tag from wikidata info." }
select distinct (?osm as ?id) (?coords as ?loc) (osmt:wikipedia as ?tag_1) (?wikipedia as ?val_1) ?wd ?name ?nameCA ?distance where {
  ?osm osmt:wikidata ?wd;
       osmt:name:ca ?nameCA;
       osmt:name ?name;
       osmm:loc ?coords.
  minus { ?osm osmt:wikipedia []. }

  service <https://query.wikidata.org/sparql> {
    ?wd wdt:P625 ?coordsWD.
    ?article schema:about ?wd;
             schema:isPartOf <https://ca.wikipedia.org/>;
             schema:name ?title.
  }

  bind(concat("ca:", ?title) as ?wikipedia)
  bind(geof:distance(?coords, ?coordsWD) as ?distance)
  filter(?distance < 1)
}
limit 50

Run it (edit query) --CamelCaseNick (talk) 19:46, 19 October 2020 (UTC)

Thank you, amazing speed up!--Jmaspons (talk) 21:13, 19 October 2020 (UTC)