Humanitarian OSM Team/Working groups/Technical/meeting 2014-09-22

Meeting to discuss Tech topics on Monday 22nd September 2014

IRC log:

19:01:38< dodobas> hi everyone.. Tech Working Group meeting is about to start...
19:01:59< dodobas> we have prepared a tentative agenda https://hackpad.com/TWG-Meeting-X15-Tentative-Agenda-CccX3ujK0p7
19:01:59< mkl> hi dodobas, i'm here for the meeting
19:02:09< dodobas> mkl: gr8
19:02:48-!- Paul_Mallet [~oftc-webi@alf94-h05-87-88-155-160.dsl.sta.abo.bbox.fr] has joined #hot
19:03:20< dodobas> I guess I can start... if no one else wants to ?
19:04:58< dodobas> IFAIK, pgiraud ig going to release a new OSMTM2 version in the next couple of days
19:05:09< bgirardot> I'm here to listen in as I'd like to contribute eventually.
19:05:30-!- pierregiraud [~pierregir@ver73-2-88-160-44-40.fbx.proxad.net] has joined #hot
19:05:34< dodobas> if there are no major issues
19:06:07< dodobas> we've just updated the translations... so if you want to, there is still time to contirbute to the master resource on transife
19:06:29< dodobas> https://www.transifex.com/projects/p/osm-tasking-manager2/resource/master/
19:06:41< mkl> that's great. i need to review all that work, to confirm camp2camp's.
* pierregiraud not supposed to be here, waiting for someone who’s late		19:06
19:07:07< mkl> actually if pierregiraud has time, would be good to just go through together / demo
19:07:10< mkl> not now i guess ;)
19:07:43< dodobas> i guess there are more details on hackpad what pierregiraud wantedg to discuss
19:08:02< dodobas> that's for tasking manager...
19:08:22< pierregiraud> nothing more to say
19:08:30< pierregiraud> thanks dodobas
19:08:32< dodobas> sysadmin... therewere no major issues orthe server
19:09:26< dodobas> I was asked to setup a geonode instance for HOT
19:09:28-!- harry-wood [~Adium@95.129.187.173] has quit [Quit: Leaving.]
19:09:28< dodobas> http://geonode.org/
19:10:03< dodobas> but... it's very simple to get it running,but not as simple to configure for production
19:11:09< dodobas> as we use an external database, I did not have best of time setting everything up
19:11:45< dodobas> geonode is built on three components, python service, postgresql database and geoserver
19:12:44< dodobas> and I still haven't got Python service ->Geoserver working
19:13:07< dodobas> so to sum up... I'm working on getting it deployed
19:13:35< sev_hotosm> @dodobas there is still time=?
19:14:12< dodobas> does anyone else has something to report (that is not on the agenda) or we can continue with hot-exports ?
19:14:47< sev_hotosm> what for the geonode instance? What data will be populated?
19:15:31< sev_hotosm> TMv2.1= there is a description of the new functionalities?
19:15:43< sev_hotosm> user oriented or more for backstage?
19:16:02< dodobas> sev_hotosm: i think that it will be used primarly for HOT data exchange... for example you want to share some data... and you don't want to upload it to the dropbox or send as an attachement
19:16:28< dodobas> basically a place to share geodata (used internally by HOT)
19:16:36-!- FredB [~makina@mir31-1-82-224-13-158.fbx.proxad.net] has quit [Quit: FredB]
19:17:03< dodobas> sev_hotosm: there is this...for the version 2.5 https://github.com/hotosm/osm-tasking-manager2/releases/tag/untagged-6965e95fec15c7ba7d14
19:17:19< dodobas> or a full changelog... https://github.com/hotosm/osm-tasking-manager2/releases
19:17:57< sev_hotosm> ah a request from Ebola Activation I guess
19:18:59-!- spatialfrog_ [~spatialfr@CPE000e84807d02-CMbcc8100f7698.cpe.net.cable.rogers.com] has joined #hot
19:19:09< dodobas> sev_hotosm: does that answer your quesstions?
19:20:01< sev_hotosm> first link does ot work for me, trying the second one
19:20:34< sev_hotosm> second works and yes it answers
19:20:39< dodobas> hmm, maybe you need to be logged in to github...
19:21:05< dodobas> so... next thing on the agenda is.. hot-export
19:21:06< bgirardot> It didn't work for me either and I am logged in, returned 404
19:21:12< sev_hotosm> Am logged in
19:21:54< dodobas> hot-export ... we recently activated job deprecation policy
19:22:11< dodobas> https://github.com/hotosm/hot-exports/issues/70
19:22:23-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has joined #hot
19:22:28< dodobas> as there are constant problems with disk space on the server...
19:22:29-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has quit []
19:22:49< dodobas> the exports use ~320G of disk space...
19:23:40-!- pierregiraud [~pierregir@ver73-2-88-160-44-40.fbx.proxad.net] has quit [Quit: pierregiraud]
19:24:00< dodobas> AFAICT, exports service was never designed to perserve history od exports
19:24:34-!- spatialfrog [~spatialfr@CPE000e84807d02-CMbcc8100f7698.cpe.net.cable.rogers.com] has quit [Ping timeout: 480 seconds]
19:24:34-!- spatialfrog_ is now known as spatialfrog
19:24:54< dodobas> so sev_hotosm raised an issue on the mailing list, to keep some old inactive jobs/runs as long as possible
19:26:07< dodobas> also, there is a pile of issues on the GH... mostly missing functionality
19:26:31< dodobas> so those would require a 'large amount of work' to get done
19:26:58< dodobas> any comments, before i continue?
19:29:29-!- mkl [~0c04df02@shenron.openstreetmap.org] has quit [Quit: CGI:IRC]
19:29:33-!- CGI570 [~0c04df02@shenron.openstreetmap.org] has joined #hot
19:30:23< sev_hotosm> identify the jobs to be kept would help?
19:30:38< dodobas> the issue we need to disscus about is how to keep those jobs...
19:31:29< dodobas> my proposal is to simply add a prefix to the jobs .. something along the lines of "[History] Actual name of the job"
19:31:29< sev_hotosm> for the moment, everything would be flushed?
19:31:49-!- MM4M731 [~49028ada@shenron.openstreetmap.org] has quit [Quit: CGI:IRC (Ping timeout)]
19:32:01< dodobas> everything that is marked to be deleted 3400 jobs...
19:32:03< sev_hotosm> why not, but the name can be modified?
19:32:17< dodobas> is should be... hmm
19:32:59-!- cypher_ [~fernando@200.16.132.146] has quit [Ping timeout: 480 seconds]
19:33:56< sev_hotosm> as a user, seems I cannot when logged in, with my own jobs
19:34:18< dodobas> yeah... let me look at the code...maybe there is a hiddenlink
19:36:08< dodobas> sev_hotosm: maybe no...
19:36:18< dodobas> so we need job_ids
19:36:32< dodobas> I can 'undelete' all of the currently marked jobs
19:37:04< sev_hotosm> marked as?
19:37:08< dodobas> so we'll have more time to find which jobs need to be perserved
19:37:19< dodobas> marked as 'delete'
19:37:58< dodobas> http://export.hotosm.org/en/jobs?deleted=y&notowner=y&page=12
19:38:02< dodobas> for example
19:40:24-!- marc1909 [~marc1909@p5DC5F780.dip0.t-ipconnect.de] has quit [Ping timeout: 480 seconds]
19:40:25< sev_hotosm> there are a few of them I would like to mahe historical
19:40:47< dodobas> sev_hotosm: how fast can you find thoes
19:41:14< dodobas> I can prolong the deadline... so you'll have time to send me job_ids
19:41:43< sev_hotosm> I just need to send you job ids? But for the next flush?
19:41:58< sev_hotosm> not possible to put this [historical] once for all?
19:42:47< dodobas> once I mark those jobs as historical.. those will never get flushed/purged
19:43:29< dodobas> we don't want to keep every job... the system was never designed to do that
19:45:01< dodobas> for example... we could add a feature... 'archive job' that would copy all of the job runs to the amazon S3 or some other cloud provider
19:46:05< sev_hotosm> so you mark the existing ones as [historical] as only the super admin can do it, and for the new ones everyone needs to add this prefix when creating ?
19:46:07-!- marc1909 [~marc1909@p5DC5F780.dip0.t-ipconnect.de] has joined #hot
19:46:16< dodobas> as at the moment... there are no backups... and if for example .. something happens to the geofabrik's server... we will lose everything
19:47:00< dodobas> sev_hotosm: yes.. i know it's not perfect... but that requires almost no developer time to implement
19:47:38< dodobas> on the other hand... wecan turn off deprecation policy... and only remove jobs which are marked as deleted by users
19:48:00-!- mm731 [~49028ada@shenron.openstreetmap.org] has joined #hot
19:48:08< dodobas> until we agree on a better solution
19:48:18< dodobas> how does that sound ? :)
19:49:05< mm731> I'm here to listen in as I'd like to contribute eventually.
19:49:23< dodobas> mm731: hi
19:49:39< dodobas> can you introduce yourself ? :)
19:50:23< mm731> sure. I am a graduate student in GIS at Eastern Michigan University. I am interested in learning how to use the HotOSM layers in ArcGIS Online
19:50:30-!- harry-wood [~Adium@146.200.105.22] has joined #hot
19:51:02-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has joined #hot
19:51:06< dodobas> mm731: nice to meet you
19:51:15-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has quit []
19:51:40< dodobas> sev_hotosm: so what dp you think?or anyone else ?:)
19:52:05< dodobas> shoud we just disable deprecation policy at the moment
19:53:00< sev_hotosm> @dodobas sorry I was not criticizing, just wanted to be sure about the process
19:53:21< sev_hotosm> Improving this would require coding
19:54:03< sev_hotosm> what would you suggest if this is possible? I suggested tagging. Possibility for users to edit their jobs would be good anyway
19:54:23-!- mkl [~mikel@117.sub-70-208-145.myvzw.com] has joined #hot
19:55:12< sev_hotosm> dodobas what proportion of marked as deleted? I fear this is not enough to give a breath to the disk space
19:55:18-!- CGI570 [~0c04df02@shenron.openstreetmap.org] has quit [Quit: CGI:IRC (Ping timeout)]
19:55:46< dodobas> sev_hotosm: hmm,I can check that, but it will require some time...
19:56:19< dodobas> at the moment ... /dev/vdb1 387G 339G 30G 93% /home/hot/var/runs
19:56:53< dodobas> so thats... maybe 20 days before we run out of disk space again...
19:57:54< dodobas> should i stop the deprecation policy or you will send me job_id which need to be archived ?
19:58:18< sev_hotosm> yes so we definitely need to find a solution.
19:58:26< sev_hotosm> I can send these ids
19:58:57< sev_hotosm> what format?
19:59:08< sev_hotosm> they are basically in wikipages
19:59:15< sev_hotosm> but you need a table?
19:59:33< sev_hotosm> 1 column, 1 ID per row?
19:59:40< dodobas> just a list...
20:00:00< dodobas> comma separated
20:01:50-!- CGI718 [~0c04df02@shenron.openstreetmap.org] has joined #hot
20:03:51< dodobas> can we continue with the agenda ?
20:04:08< sev_hotosm> sure
20:05:04< dodobas> so still on hot-export... a lot of isses pilled up.. most of those require proper development
20:06:16< dodobas> so we would need to prioritize issues, try to figure out how much time is required and then see if some funding is possible
20:06:39< CGI718> dodobas, i remember seeing there was a pull request for the redesign work, and an offer of help
20:06:39-!- mkl [~mikel@117.sub-70-208-145.myvzw.com] has quit [Ping timeout: 480 seconds]
20:06:47-!- CGI718 is now known as mkl
20:06:59< mkl> whatever became of that pull request?
20:07:46-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has joined #hot
20:08:38< dodobas> mkl: I've create a test instance... but UI was half done... not really working
20:09:26< dodobas> they switced from OL to Leaflet...
20:09:49< dodobas> we could take a look at it again if you want ?
20:11:19< dodobas> priorities list -> specification document -> funding ? :)
20:11:35-!- marc1909 [~marc1909@p5DC5F780.dip0.t-ipconnect.de] has quit [Quit: marc1909]
20:11:53< mkl> sounds like a plan
20:12:04< mkl> i guess this one is lacking a project owner
20:12:39< dodobas> mkl: well I'm in the manger 'role' but... :)
20:13:29< mkl> gotta clear up the last bits of the OSMTM oversight, but then
20:14:01< mkl> i'm willing to help. would be good to have 2-3 of us to co-work the digestion of github tickets -> priorities -> specification
20:14:23< mkl> i also want to wrap up some of the website updates with clara
20:14:31-!- JulienME [~Julien@2a01:e35:2f4f:eaa0:21f:d0ff:fe27:7e0e] has joined #hot
20:14:50< dodobas> i'll draft a 'RFC' for hot-export issues priority...maybe we get the ball rolling
20:15:28< dodobas> mkl: we need to schedule that kinda issue-thon ? :)
20:16:03< dodobas> discuss on mumble... and ork through the issues
20:17:38< dodobas> mkl: do you have a free spot in your calendar ?:)
20:18:19< mkl> let me see
20:19:23< mkl> thursday same time
20:19:47< mkl> thursday 5 gmt
20:20:04< dodobas> ok... I also free
20:20:07< mkl> cool
20:20:33< dodobas> i'll also draft an announcement mail on the ML
20:20:36< sev_hotosm> happy to see this going ahead
20:21:04< sev_hotosm> where can we put feedback regarding priorities?
20:21:16< dodobas> sev_hotosm: it will be in the email...
20:22:48< dodobas> next thing on the agenda is HIRC... mkl reported a lot of issues, some are simple...some require a bit more time...
20:23:11< dodobas> again we need a priority list of some kind... :)
20:23:50< dodobas> there are also some open questions... about TWG TOR document
20:24:49< dodobas> sev_hotosm: did we cover those, or you maybe have some additional comments?
20:24:57< sev_hotosm> where is the HIRC issues list?
20:25:12< dodobas> https://github.com/candela-it/hirc/issues
20:25:29< dodobas> this is the demo instance... http://hirc.dev.hotosm.org/
20:25:49< sev_hotosm> thx, added the link to the agenda
20:26:25< sev_hotosm> regarding my first open question it would be great to explain how and where the "prioritize issues and feature requests" is organized
20:26:53< dodobas> sev_hotosm: the idea was to let project managers do that...
20:27:10< sev_hotosm> and the next step if coding is necessary
20:27:11< dodobas> and create project reports... but we kinda did not do that...
20:27:54< dodobas> for the second one... I guess someone would be put in charge by the board
20:28:14< sev_hotosm> OK but if this is about user cases, how project managers can state what is to be prioritized first. They may not be end users
20:28:48< dodobas> sev_hotosm: project managers are a 'gateway' for the project
20:29:20< dodobas> they are responsible to communicate with users... at their leisure
20:31:42< sev_hotosm> mkl would it not be possible to make a list of potential dev and make the community vote to prioritze them when it deals with new features for end users (not for backstage)?
20:32:03-!- alexbarth [~alexbarth@50-198-132-58-static.hfc.comcastbusiness.net] has quit [Remote host closed the connection]
20:32:13< sev_hotosm> would also be a good way to raise interest from contributros
20:32:47< mkl> i think there's a lot of ways to manage that. discussing directly with users and having a feedback process is a very good idea.
20:33:27< dodobas> maybe introduce a 'bounty budget'
20:33:31< mkl> i'd be inclined to start with a quick review and prioritization, just to get context straight
20:33:55< mkl> then to reach out and arrange interviews with users
20:34:11< mkl> from there, draw up priorities, and solicit feedback
20:34:34< mkl> ultimately, decisions need to be made, but i think this process grounds it in what's actually needed
20:34:45< mkl> btw, this might be something i find time to work on during the fellowship
20:35:34< sev_hotosm> and organize like a poll? Would be even possible to mention how many days required for each new feature requests (if possible to know that by advance) vs the # of funded days, so that people need to make choice
20:36:17< sev_hotosm> and do not make an open letter to Santa Klaus (I want all the toys catalog!)
20:36:57< mkl> i really prefer the idea of discussion among key users, with opportunity for comment
20:37:17< mkl> yes, we'd want to make some time estimates on features
20:38:59< sev_hotosm> on the hot list?
20:39:09< sev_hotosm> I mean for the discussion
20:39:56-!- alexbarth [~alexbarth@50-198-132-58-static.hfc.comcastbusiness.net] has joined #hot
20:39:56-!- sev_hotosm [~severin@105.234.15.132] has quit [Read error: Connection reset by peer]
20:40:12< mkl> yea, HOT list is the best broad discussion poitn
20:41:09< dodobas> sev_hotosm: it's difficult to estimate, at best we can try in big units... half-a-week, week, two weeks ..
20:41:34-!- le_loup_qui_rit [~Thunderbi@2a02:2788:74:c70:1089:92f7:f5e:f616] has quit [Ping timeout: 480 seconds]
20:41:51-!- alexbarth [~alexbarth@50-198-132-58-static.hfc.comcastbusiness.net] has quit [Remote host closed the connection]
20:43:08-!- SpikeUK [~Thunderbi@host86-156-203-219.range86-156.btcentralplus.com] has joined #hot
20:43:24< dodobas> does anyone else have anything else to report ?
20:43:48< dodobas> if not... we are going to close the official part of the meeting...
20:44:22< dodobas> I'll do the usual summary first thing tomorrow morning... and post it to the ML
20:44:26-!- marc1909 [~marc1909@p5DC5F780.dip0.t-ipconnect.de] has joined #hot
20:44:27< dodobas> thank you all