Humanitarian OSM Team/Working groups/Technical/meeting 2014-08-11

From OpenStreetMap Wiki
Jump to: navigation, search

Meeting to discuss Tech topics on Monday 11th August 2014

IRC log:

dodobas pierregiraud: would you like to start... OSMTM2 ? 17:02
pierregiraud OK 17:02
harry-wood hi all 17:02
skorasaurus hi 17:02
ptressel o/ 17:02
*** wonderchook ( has joined #hot 17:02
pierregiraud so since WG meeting the V2 has been deployed on production 17:02
pierregiraud with V1 data migration 17:03
pierregiraud it went more or less well 17:03
pierregiraud but we finally got there 17:03
ptressel \o/ 17:03
pierregiraud thanks to the help from dodobas 17:03
pierregiraud I then continued on integrating new features asked by people who contracted with HOT and camptocamp 17:04
dodobas it's kinda my job, you know :) 17:04
*** FredB ( has joined #hot 17:04
pierregiraud there were several releases from then 17:04
pierregiraud 17:04
pierregiraud with a lot of new features and bug fixes 17:04
pierregiraud I suppose everyone here know about this 17:05
harry-wood Well done everyone involved in that! 17:05
pierregiraud we had a critical issue last week but this seems to be fixed now 17:06
pierregiraud anyone have questions about the new features or fixed bugs? 17:06
dodobas for everyone who wants to follow or add something ... this is the agenda... 17:07
pierregiraud if not I have some specific issues I would like to discuss a bit with you 17:07
dodobas it's an unordered list, if you were wondering :) 17:07
pierregiraud 17:07
pierregiraud this issue is about osm users, their names and ids 17:08
pierregiraud in V1, the issue was that only the username was stored in the TM’s DB 17:08
pierregiraud when the user changed its name, it appeared as a new user if they logged in again in TM 17:09
pierregiraud this issue is fixed since we now store ids 17:09
pierregiraud however, if the username changes we have no way to handle it 17:10
dodobas pierregiraud: so the problem is 'how to regulary update usernams ? 17:10
pierregiraud yep 17:10
pierregiraud exactly 17:10
pierregiraud and there’s no service to get the name for a user by its id 17:10
harry-wood oh really? 17:10
harry-wood hmmm someone should totally add that 17:11
skorasaurus nothing in the OSM api? 17:11
pierregiraud the question has been asked several times 17:11
harry-wood or the API should stop letting people change usernames :-) 17:11
pierregiraud there’s a long thread about this I think 17:12
pierregiraud 17:12
dodobas pierregiraud: maybe overpass api? 17:14
harry-wood yeah that works: 17:14
pierregiraud howdy 17:14
pierregiraud I didn’t read my mails correctly 17:14
pierregiraud my bad 17:14
pierregiraud I got an answer from simonpoole 17:14
dodobas harry-wood: yeah 17:15
pierregiraud sorry for the noise 17:15
* pierregiraud slaps himself 17:15
harry-wood Still quite a pain you have to call this and then update the name in our DB 17:16
pierregiraud or maybe « spanks » ? 17:16
pierregiraud so now the question is, when should we do this? 17:16
dodobas pierregiraud: so... don't link directly to the user page... create a proxy page that will check the user_name and... redirect to the OSM page 17:16
pierregiraud but the user may want to see his name on the top right corner 17:17
clara not necessarily 17:17
harry-wood do when the user logs in 17:17
harry-wood if they change their username half way through a session… they can expect a broken link 17:18
clara it's more important that the user knows that they are logged in - but it doesn't need to say the name 17:18
dodobas pierregiraud: is ther something in the Oauth metadata ? 17:18
pierregiraud the user who changes his name expects a broken link 17:18
pierregiraud others don't 17:18
pierregiraud I receive oauth metadata only when the user logs in 17:18
pierregiraud but username is remember for a long time 17:19
pierregiraud users don’t have to re-log in often 17:19
harry-wood the user who changes his name …is a pain the ass. They deserve some broken links :-) 17:19
skorasaurus could a warning saying that you must log-in or reauthorize if you change your user name enough? 17:20
pierregiraud the tasking manager doesn’t know that the name has changed 17:20
pierregiraud unless the user re-logs in 17:20
clara how often do users actually change their name? 17:20
harry-wood No I think we just say that to people. If somebody complains that it's showing the wrong name, we tell them to try logging out and in again 17:20
skorasaurus ah, ok, i understand now. 17:20
dodobas then just show uid :) 17:21
dodobas problem sloved :) 17:21
pierregiraud hum 17:21
clara no - it's very difficult to find out your uid 17:21
clara so users won't recognize themselves in the number 17:21
pierregiraud OK, I’ll think about it a bit more and come back with proposals 17:22
clara and then they get confused trying to change that to their name :) 17:22
* dodobas is just kidding :) 17:22
pierregiraud problem is half solved with the link (I didn’t pay attention to) given by simon poole 17:23
pierregiraud next point is less technical 17:23
pierregiraud but more about contributions to OSM DB and information about how the data was produced 17:23
pierregiraud 17:24
pierregiraud it looks like I didn’t pay enough attention (once more) to the difference between changeset comments and « source » tag 17:24
dodobas pierregiraud: it's a long issue, can you summarize ? 17:25
pierregiraud we can leave this question to the github thread unless someone here want to give his point of view 17:25
pierregiraud ok 17:25
pierregiraud when an imagery is added automatically by the remote link in JOSM 17:26
pierregiraud a new layer is created 17:26
pierregiraud JOSM uses the layer « title » to add a source tag to elements uploaded to OSM DB 17:26
pierregiraud it’s not clear how it’s done currently 17:27
pierregiraud the user asks us to change the TM code so that the title is set properly at once 17:27
harry-wood yeah I agree that would be good 17:27
pierregiraud but it looks like there are possible caching problems 17:27
pierregiraud so currently the way to solve the issue is not well defined 17:28
pierregiraud I think we should leave this for now 17:29
harry-wood Well if the project admin sets the name of imagery to be clashing with some previous name of imagery… then there's a caching problem. Not otherwise surely? 17:29
pierregiraud people qualified for this discussion should join us on the thread on github 17:29
harry-wood yes good point :-) moving on 17:29
*** cypher_ has quit (Ping timeout: 480 seconds) 17:29
pierregiraud dodobas: l10n enhancements? 17:30
pierregiraud 17:30
dodobas yeah... so l10n ... 17:31
dodobas we should add proper support for plural forms 17:31
*** TOGA has quit (Quit: Leaving) 17:32
dodobas and... I'm not sure how to make JavaScript translatable... 17:32
dodobas there are different solutions... 17:32
dodobas but... not sure which are integratable with transifex 17:32
dodobas or... how to actually update/extract strings from JS files... 17:33
pierregiraud any example in mind for plural forms currently in the tasking manager? 17:33
pierregiraud for javascript I tried to the translated string in the mako templates 17:33
dodobas other thing... you mighthave noticed that we use Transifex 17:33
pierregiraud to put 17:33
dodobas there are two resources... master and current 17:33
dodobas master are strings from currently developed OSMTM 17:34
dodobas current is currently deployed version 17:34
dodobas Transifex is nice... integration and everything... but :) 17:35
dodobas it's really hard (or I don't know how to) latest updates 17:35
dodobas pierregiraud: not sure... maybe there is some reference in the Pyramid docs 17:35
wonderchook dodobas: have you talked to the InaSAFE guys? They've been through the transfex set-up recently and might have suggestions 17:36
dodobas pierregiraud: plural... like this one ... 17:37
dodobas ${locked} user(s) is (are) currently working on this project 17:37
pierregiraud OK 17:37
dodobas wonderchook: will do 17:37
ptressel Are the plural strings not separately translated? 17:38
ptressel (Not familiar with Transifex workflow -- we use Pootle for Sahana.) 17:38
pierregiraud this could be an option 17:38
*** almccon ( has joined #hot 17:38
dodobas wonderchook: i got everything nicely integrated... but... it's hard to follow updates... other then blindly update on a regular basis, or manually read 'latest changes on transifex' 17:38
dodobas ptressel: it depends on language 17:39
dodobas english has 2 plural forms... croatian 3... 17:39
wonderchook dodobas: you mean so you know which translations are at a good point to integrate? 17:39
dodobas wonderchook: we don't have a policy for that at the moment... 17:39
ptressel dodobas, So if you're using the English string as the translation key, that doesn't provide distinct keys for Croatian... 17:40
dodobas for example... less then 85% translated ... doesn't get in 17:40
wonderchook so is the issue lack of policy? 17:40
wonderchook or is there another problem? 17:40
dodobas ptressel: it's not like that... 17:40
dodobas the system will 'present' a user with a different input form for plural forms... 17:40
dodobas as long as the source files are prepared to 'render' plural forms 17:41
ptressel What is "system" here? 17:41
ptressel Transifex? 17:41
dodobas ptressel: any that can read PO files 17:41
dodobas so we need to prapare the sources... 'mark' strings which are plural.. and everything else should be 'automatic' 17:42
ptressel Not sure trying to pluralize on the fly is really doable. 17:42
dodobas wonderchook: well, at the moment I've been adding new languages to the osmtm... 17:42
dodobas and we should think about a policy to control when a language gets added to the osmtm... and when a language gets removed from osmtm 17:43
wonderchook have you looked at other project policies? how much needs to be translated to be added or removed? 17:44
wonderchook I think the translation needs to be enough to be useful 17:44
wonderchook but not sure what that means 17:44
dodobas ptressel: it was doable at the time PO format was introduced to Unix :) 17:44
ptressel Ha. 17:44
ptressel 70s 17:44
dodobas wonderchook: so enough is ... 45% ? 85% ? major strings translated ? :) 17:45
wonderchook I think something 50 or 60% 17:45
wonderchook but that is just me guessing 17:45
dodobas ptressel: i think it's just a change from 'gettext' to 'ngettext' or something similar ... 17:46
ptressel Yesbut...that just doesn't work. 17:46
ptressel In most languages, the change to plural affects many words in the sentence. 17:47
ptressel There are two common solutions: 17:47
dodobas wonderchook: qgis had removed all languages below 30%, for 2.0 relese 17:47
dodobas *release 17:47
dodobas ptressel: that's why translators need to update all strings... even plural :) 17:48
ptressel One is, give the translator an indefinite form "There is (are) %n user(s) on the system" and let them come up with an equivalent indefinite form 17:48
ptressel Second is, give the translator both a singular and plural form 17:48
pierregiraud shouldn’t we first find an example of implementations in pyramid 17:49
ptressel That is, the translator gets the entire phrase, not word by word. 17:49
pierregiraud and/or check if transifex will help us 17:49
dodobas pierregiraud: they use ngettext... 17:49
dodobas it's the only way... 17:49
ptressel  ? 17:49
ptressel Well, we don't do it for Sahana, so I'd contend it's not the only way... 17:50
dodobas indefinite forms sound really 'idiotic' in Croatian language 17:50
ptressel See above -- two options. 17:50
dodobas ptressel: yes yes 17:51
dodobas i don't like the first one... and think we should do the second one... 17:51
ptressel We do the second -- generate translation requests for both a singular sentence and a plural sentence 17:51
dodobas let ths system handle everythin, just be smart when to use gettext and when ngettext 17:51
ptressel ngettext isn't either of the two options I mentioned, so that's a third. 17:52
ptressel I don't think it works linguistically though. 17:52
dodobas it's up to the translator to do the work 17:53
clara maybe list the different options and discuss it in the issue queue for a bit - if you need to ask somebody else anyway? 17:53
*** marc1909 has quit (Quit: marc1909) 17:53
dodobas clara: we are only discussing... we might move this to the mailing list 17:54
ptressel  ;-) 17:54
dodobas present different options... and then decide... 17:54
dodobas or not decide 17:54
pierregiraud would it possible to do POC? 17:55
dodobas pierregiraud: yeah... 17:55
dodobas I'll prepare something on master OK ? frst thing tommorrow morning 17:56
dodobas anything else regarding osmtmv2 ? 17:56
dodobas or can we move on ? 17:56
pierregiraud we can move on 17:57
pierregiraud I thin 17:57
pierregiraud think 17:57
clara website? 17:57
dodobas clara: sure... 17:57
clara I started with the bi-monthly report about what has been done and what the priorities, and linked it from a new section on the wiki 17:57
dodobas and then wonderchook ... OAM 17:57
clara Mikel ask for displaying profiles and work groups on the website, so I had a go at it. For now just with the obvious info OSM account, twitter, countries, short description, picture, projects and work groups they are involved it. Plus his/her latest updates. 17:57
clara It looks like this for now: 17:58
clara Same for work groups: that's currently only description, links and members: 17:58
clara Next step is to ask each work group if they require anything else. Any ideas can be added to the issue: 17:58
clara ... end 17:58
clara any questions? 17:58
wonderchook great, with the profiles will they link from projects and blog posts as well? 17:58
clara yes 17:58
dodobas so... we should all write blog posts ?!?! 17:58
dodobas  :) 17:59
clara they are linked to the user name - even for users that are not active 17:59
clara and they can be added as references to projects and working groups 17:59
wonderchook great, I think one thing which is more a communications thing versus the tech side will be to think about the overall organization of the headers on the website 18:00
clara my next step would be to ask the other work groups whether there are any specific fields they require 18:00
wonderchook meaning would the working groups really be a top level 18:01
wonderchook but that isn't so much tech as figuring out how to arrange things 18:01
*** cquest_ has quit (Remote host closed the connection) 18:01
clara yes - and that can be easily changed 18:01
dodobas clara: it's good that you have don't a project manager report... now we have a template to work with :) 18:01
clara  :-) 18:02
dodobas anything else for the website? 18:02
clara not from me 18:02
pierregiraud I should be able to work on mine soon 18:02
clara pierregiraud: what do you mean? 18:03
pierregiraud project manager report 18:03
clara k 18:03
clara next point then? :) 18:04
wonderchook OAM? 18:05
dodobas wonderchook: yes 18:05
wonderchook okay so there is this website: with the documentation that dodobas put together 18:05
wonderchook which will continue to be filled out. 18:05
wonderchook the idea is to have a HOT Tech Challenge for the catalogue part of OAM soon 18:05
wonderchook we created a Google group specific for OAM:!forum/openaerialmap 18:05
wonderchook it seemed the HOT mailing list is probably too noisy for people just interested in OAM 18:06
wonderchook one discussion topic is at what point is the minimum viable product for delivery of OAM at the end of the grant 18:06
wonderchook or rather "what features are needed for it to be useful" 18:06
wonderchook did I miss anything dodobas? 18:07
dodobas WE NEED HELP ! 18:07
dodobas  :) 18:07
wonderchook ha, yes we want to recruit more tech people to help 18:07
wonderchook part of that is the HOT tech challenge 18:07
ptressel Well, we have a bunch of interested folks in Seattle... 18:08
wonderchook but also having the mailing list to point interested parties at, so if you know anyone that would be interested in helping please refer them to myself or the mailing list 18:08
ptressel Have been trying to entice in Michael Patrick. 18:08
*** harry-wood has quit (Quit: Leaving.) 18:08
dodobas if you notice there is a new way to contribute... 18:09
wonderchook oh awesome I missed that part 18:09
dodobas once you create an account on 18:10
dodobas you can select any text on the page and add a comment 18:10
wonderchook I will be do a BOF at FOSS4G and we'll have some community online meetings again soon 18:10
wonderchook but yes at the moment it is mostly dodobas with some help from me 18:10
clara do you wont to put something about OAM on the website as well? 18:10
dodobas it's supported by ... 18:11
wonderchook hi clara, I should make it a project page 18:11
wonderchook so yes! thanks for the reminder 18:11
dodobas so... we can even create a HOT instance.. and use it for 'something' 18:11
*** jgc ( has joined #hot 18:12
dodobas anything else... 18:13
ptressel Say we have a bunch of info on UAV imagery georeferencing...where would we put that? 18:13
dodobas ptressel: on a OAM Node... :) 18:13
ptressel Meaning? 18:14
dodobas at the moment, we don't exactly know what is an OAM Node... so... 18:14
ptressel This is just info 18:14
dodobas OAM will also host documentation and best-practice guidelines for processing imagery... and whatnot 18:15
ptressel Appears we can't look at the sub-pages without an annotateit account? 18:15
dodobas but we don't know how will it manifest it's self in the OAM 18:15
ptressel What I mean is, is there a place to put it now? 18:15
ptressel This is info about how to do it. 18:16
ptressel It's not a functioning system. 18:16
dodobas now... I don't know 18:16
wonderchook there is no specific place to put imagery right now because OAM doesn't exist 18:17
wonderchook there are all the other ad hoc existing ways to host imagery 18:17
wonderchook if the issue is there is a current need with specific imagery 18:17
ptressel Not imagery 18:17
dodobas or any info on how to georeference UAV... 18:17
ptressel Info about how to georeference UAV imagery. 18:17
wonderchook UAV pictures are imagery 18:17
dodobas ptressel: what subpages ? 18:18
wonderchook there are ways to georeference UAV imagery, but at the moment this is out of scope of the conversation of where we are with OAM 18:18
ptressel "What we have here is a failure to communicate." 18:18
wonderchook if you have a specific UAV imagery needs how about we discuss after the meeting 18:18
ptressel Poke any of the nav links on the left of 18:18
ptressel [11:17] ptressel "What we have here is a failure to communicate." 18:18
ptressel What we have is *info* about *how to* georeference UAV imagery. 18:19
* clara needs to be off now - bye! 18:19
dodobas bye clara... 18:19
dodobas ptressel: so let me summarize 18:19
pierregiraud ptressel: Guns 18:19
dodobas there no specific place to put info about processing UAV... at the moment 18:19
dodobas however it should be a part of OAM 18:20
dodobas we are trying to figure out, and propose technical solutions for OAM 18:20
dodobas and if you have any comments, please share 18:21
* pierregiraud has to go too 18:21
ptressel Yes, and we have folks who work with lidar data professionally. 18:21
ptressel The SFM work was done at the U of Washington. 18:21
dodobas I have a short sysadmin report... 18:21
ptressel But going back to the page... 18:22
* pierregiraud stays a bit more 18:22
dodobas everything is more or less fine... 18:22
dodobas other then blackout on 3rd, August 18:22
pierregiraud hosting problems? 18:23
ptressel If I poke any of the nav links, like OAM Processing, I get a warning flash that says could not get auth from annotateit 18:23
ptressel And the page just contains Lorem ipsum -- is that what is actually there? 18:23
dodobas DC19 lost voltage, and crashed everything.... UPS did not kick in... 18:23
dodobas anyway... our server was quickly powered up... as some servers took a few hours to power on 18:24
*** hjart (~hjart@ has joined #hot 18:24
dodobas just a sec... brb 18:24
dodobas thats the second outage in 12months... 18:26
dodobas which is not to bad 18:26
wonderchook dodobas: at some point we want a back-up system, right? 18:26
dodobas server is not under any heavy load... there are more then enough resources 18:27
dodobas wonderchook: yes 18:27
dodobas that's something I NEED to do 18:27
ptressel What database is it using? and can that db do journalling? 18:28
dodobas the idea is to preapre a Amazon/DigitalOcean image which can be quickly powered on 18:28
wonderchook dodobas: I think we have a couple different options. If we wanted to buy a server we could probably get free hosting for it. Though there is a certain advantage of not doing it that way 18:28
wonderchook but yes the amazon option is also good :) 18:28
dodobas ptressel: well, mysql and postgresql and sqlite ... :) 18:29
ptressel Hmm... 18:29
dodobas ptressel: but HA fail over systems ... are not an option 18:29
ptressel Not what I meant... 18:29
wonderchook I think the other thing with the systems is there are some that are far more important than others. e.g. the tasking manager is really key 18:29
ptressel Journalling writes out the transaction before it performs it. 18:29
ptressel So if the system crashes during transaction processing, it can roll back and replay the transaction 18:30
dodobas wonderchook: yeah, that is critical 18:30
dodobas wonderchook: I don't think we need a full failover setup... just something that can be 'started' up with the latest daily backup 18:31
wonderchook yeah, that is sufficent 18:32
dodobas so yeah, I'll focus on that... 18:32
*** jgc has quit (Ping timeout: 480 seconds) 18:32
ptressel 18:32
ptressel Looks like postgresql can do journalling. 18:33
dodobas ptressel: yeah, but if the server crahes... and it can't be recoverd... for some reason... journaling will not help 18:33
ptressel Depends... 18:34
ptressel Do you mean a disk crash? 18:34
ptressel Or power fail? 18:34
dodobas a meteor burns half of a datacenter 18:35
dodobas or a mouse eats through a network cable 18:35
ptressel Those are different cases... 18:36
ptressel The distinction is, did you lose the local storage. 18:36
ptressel To protect against losing local storage, can remote mirror / sync. 18:36
*** skorasaurus2 (~skors@ has joined #hot 18:36
dodobas it's an outage, and the server is not reachable, we need to have a working tasking maanger instance 18:37
*** TOGA ( has joined #hot 18:37
dodobas you don't know when the issue is going to be fixed 18:37
ptressel Yes, and you can bring it up using the mirrored db. 18:37
ptressel The server config is relatively static -- the data isn't 18:38
dodobas that's why I said... HA Failover are not an option 18:39
dodobas we only want to mitigate the issue 18:39
ptressel 18:39
ptressel We're failing to communicate. ;-) 18:40
dodobas and we don't are about 12h of lost data 18:40
dodobas *care 18:40
ptressel The standard way to deal with this is database replication. 18:40
dodobas ptressel: i know all of that... 18:40
ptressel Ok... 18:40
dodobas i've done log shipping and multi slave setup... 18:41
ptressel So, the problem is...? 18:41
dodobas there is no problem 18:41
ptressel Ok, nm. 18:42
ptressel Sounded like there was something you wanted to do re outages. 18:42
dodobas we are ... 18:42
dodobas there is going to be a 'stand-in' system that can be quickly powered up... using latest daily database backup :) 18:43
ptressel Ah, so you don't want any *suggestions* about what to do about outages... 18:43
ptressel Ok, got it. 18:43
*** skorasaurus has quit (Ping timeout: 480 seconds) 18:43
dodobas if there is nothing else to add... 18:44
dodobas i think we can end this TWG meeting 18:45
dodobas ill do the usual summary ... 18:45
dodobas oh... I forgot... 18:45
dodobas we finally have IRC chat logging... 18:45
dodobas ptressel: tnx for that 18:45
dodobas 18:45
wonderchook should we tell people we are logging when we join the channel? is there a protocol about that? 18:45
dodobas it in the topic 18:46
wonderchook ok 18:46
bdon why not use amazon RDS for the postgis database? 18:46
dodobas bdon: it too expensive 18:46
ptressel How much does it cost? 18:46
dodobas *it's 18:46
dodobas 18:47
bdon maybe they'd be willing to donate? HOT is a registered nonprofit right? 18:47
dodobas so the TWG meeting has ended... 18:47
wonderchook bdon: yes HOT is a non-profit 18:48
wonderchook and tax exempt 18:48
dodobas bdon: there is no point to use RDS, unless you host everything else on Amazon 18:49
bdon fair enough, out of the box high availability seems like a big win though 18:50
dodobas to host everything we would need about 5Gb of memory... let me work that out... 18:50
dodobas so.. we need an RDS pg + RDS mysql 18:52
dodobas + 3 small instances.. 18:52
dodobas + 1 micro for testing 18:52
dodobas so.. without any data trafic... and stored instances... it's about 115$ per month 18:54
dodobas but.. im not sure if that's wnough 18:54
dodobas at the moment we pay less than 100$ for a nice server that has ~10% utilization 18:55
dodobas it's a SINGLE serverthat had 2 (couple of hours) outages in 12 months 18:55
dodobas i do understand that it's not all about money... 18:56
wonderchook we used to host everything on Amazon, we are saving money now though :) 18:56
dodobas ptressel: ahh, yes some pages are just Lorem Ipsum pages... sorry for the confusion 19:01
ptressel kk 19:02