Talk:Sophox Editor

From OpenStreetMap Wiki
(Redirected from Talk:Sophox)
Jump to: navigation, search

Missing URL

I think a link to this tool is missing - or hidden, which would also be a problem. --Kocio (talk) 06:44, 7 November 2017 (UTC)

Kocio, what do you think would be the best way to link to it? http://sophox.org/sophox/ is the "power-tool" - editor to create and test sparql queries. But the majority of the less techy users would use one of the Quick fixes - because there they all link to the result view of the queries. --Yurik (talk) 07:11, 7 November 2017 (UTC)
For me it's good now. --Kocio (talk) 10:32, 7 November 2017 (UTC)

automated edit box

While providing some degrees of automation, this tool does not allow automated editing, as defined in Automated Edits code of conduct and Automated edits. I have removed the Import bot warning, as this is not applicable here. We may want to introduce a slightly different box, specifically for task-oriented tools (like MapRoulette), warning users to read-up about each task before attempting to make any edits. --Yurik (talk) 22:38, 14 November 2017 (UTC)

User:Imagico, please don't add something without any discussion. I have posted here and on your user page. You keep adding it without any discussion. Note that JOSM can be used for bot editing to a much higher degree that Sophox. Unless you also introduce it there, and in all other task-oriented fixing tools, it sounds like you are prejudice towards this specific tool. --Yurik (talk) 23:38, 14 November 2017 (UTC)

Sophox

Sophox https seems to be down and all links to it are broken. http seems to be fine. --Zache (talk) 11:50, 25 June 2018 (UTC)

Yes, sadly, Sophox main engine - Blazegraph - has been misbehaving. It refuses to load the entire OSM and WD data all at once, so we are rebuilding it. No known ETA yet, but hopefully we can get it up and running again. Turns out Blazegraph is not very happy with the indexes approaching 1TB of data :( --Yurik (talk) 16:58, 25 June 2018 (UTC)
Any news about this topic? I really miss sophox. --Hofoen (talk) 04:46, 25 October 2018 (UTC)
Hardware arrangements have been made, coming soon! --Yurik (talk) 10:27, 25 October 2018 (UTC)
Hofoen, Sophox is back up. --Yurik (talk) 18:59, 28 November 2018 (UTC)
Great to hear - but the wikidata tables are empty, aren't they? (e.g. "?item wdt:P31 ?instance" gives no relsuts) --Hofoen (talk) 12:10, 3 December 2018 (UTC)
Hofoen, yes, you now have to use federation to access Wikidata (or any other source). See docs. Use https://sophox.org/sparql as the endpoint. We should update our docs to reflect this change (help is welcome -- we will have to revise many of the examples, and some might not work the same way due to performance/timeout limitations). Lastly, just for the sake of accuracy - there is no such thing as wikidata tables - RDF databases have just a single 3-column table for everything (unless you have multiple ones in different namespaces, but in that case this is no different than federation). In this case I imported OSM data, OSM meta data (keys/tags), and Wikipedia pageview stats. Here's an example of getting "instance of":
SELECT * WHERE {
  ?osmid osmt:place "city";
         osmt:wikidata ?wd.
  SERVICE <https://query.wikidata.org/sparql> {
    ?wd wdt:P31 ?instanceof.
  }
}
LIMIT 10

--Yurik (talk) 16:39, 3 December 2018 (UTC)

Due to performance federate queries are no real option for me. In my opinion, the main advantage of your service was the fact, that both osm and wikidate were in the same RDF database. Do you have plans to integrate wikidata data again into your RDF database? --Hofoen (talk) 16:43, 6 December 2018 (UTC)
Hofoen, I have been considering it - the issue is that Blazegraph index seems to be growing far too big and the initial data import process just comes to a crawl after about 70% of Wikidata import. The same happens when I try to import OSM data after Wikidata. Perhaps there are some magical java memory optimization params or Blazegraph settings that would allow me add them both, but I have spent weeks on this problem and haven't found the solution. You are welcome to try it - all the code is in https://github.com/sophox/sophox - you can even build your local one with a small data subset within minutes. --Yurik (talk) 19:41, 6 December 2018 (UTC)
Thanks for advice, but I have no suitable machine to do such tests. Nonetheless: I could not find the scripts to sync wikidata in your repo. --Hofoen (talk) 11:14, 10 December 2018 (UTC)
Hofoen, the import scripts are part of the Wikidata setup itself, see loading the dump. --Yurik (talk) 15:55, 10 December 2018 (UTC)

Sophox question

@Yurik Is this a question for you?

Regards --Ryzen (talk) 21:51, 3 July 2018 (UTC)

@Ryzen, thanks for the ping! Yes, but sadly Sophox is still being rebuilt, trying to find spare time for it - have hit a hard problem. --Yurik (talk) 06:27, 5 July 2018 (UTC)
@Ryzen, updated the answer with a working example. Sophox is back up. --Yurik (talk) 18:59, 28 November 2018 (UTC)

Unstable Endpoint

The endpoint seems to be unstable. I'm accessing it from wikidata-query-service. Yesterday I could get result, but today I get mostly internal errors "Response = 404 page not found". Sometimes I could get results using "LIMIT 1". Even accessing it directly with "curl -X POST https://sophox.org/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' --data-urlencode 'format=json'" results in "404 page not found". With wikidata I use the following query:

PREFIX osmt: <https://wiki.openstreetmap.org/wiki/Key:>
SELECT * WHERE {
  ?wd wdt:P31 wd:Q12280.
  ?wd wdt:P17 wd:Q183.
  FILTER(NOT EXISTS { ?wd wdt:P576 ?x1. })
  FILTER(NOT EXISTS { SERVICE <https://sophox.org/sparql> { ?osmid osmt:wikidata ?wd. } })
}
LIMIT 1

--Hofoen (talk) 11:04, 10 December 2018 (UTC)

Hofoen the endpoint is (relatively) stable, but it does not allow POST requests, only GET. I will need to check with Wikidata people why POST is required for federation, and if I can safely enable it (I don't want the whole world to be able to modify the data directly :) --Yurik (talk) 16:02, 10 December 2018 (UTC)
So if the query above returns results, wikidata used GET to access sophox, otherwise POST? This would explain, that sometimes "limit 1" works, as only very few data has to be transferrred (possible with a GET request). Bigger data sizes should be transferred with a POST request. Am I right? --Hofoen (talk) 16:59, 10 December 2018 (UTC)
Hard to say, but it could be. I do not know too much about the code responsible for forming the GET/POST request. I am still exploring it -- apparently there is a X-BIGDATA-READ-ONLY header that disabled write access via POST -- this is something that WDQS uses itself. I tried to enable it, but I think something else is missing... Still exploring. --Yurik (talk) 17:47, 10 December 2018 (UTC)
Fixed. --Yurik (talk) 19:59, 10 December 2018 (UTC)
Thanks for the great work. --Hofoen (talk) 05:38, 11 December 2018 (UTC)