OpenAerialMap/Meeting Jan 22, 2015

From OpenStreetMap Wiki
Jump to navigation Jump to search

#hot: OpenAerialMap Brainstorming Session Jan 22 2015

Meeting started at 17:56:50 UTC. Meeting ended at 20:38:42 UTC.


Meeting Links

People present (lines said)

  • Cristiano (155)
  • wildintellect (119)
  • BlakeGirardot (36)
  • dodobas (28)
  • smathermather (27)
  • balrog-k2n (16)
  • nhv (14)
  • alexbarth (1)


IRC Log

Jan 22 18:57:51 Cristiano: Hi dodobas, thanks for joining
Jan 22 18:59:00 Cristiano: Hello smathermather and winditellect, thanks for coming
Jan 22 18:59:15 Cristiano: *wildintellect
Jan 22 18:59:34 smathermather: Hi Cristiano, wildintellect!
Jan 22 18:59:45 Cristiano: Are you guys OK with using gdocs for keeping notes?
Jan 22 18:59:53 Cristiano: or prefer hackpad?
Jan 22 19:00:05 wildintellect: whatever works for you
Jan 22 19:00:17 smathermather: Yup whichever.
Jan 22 19:00:46 Cristiano: OK, let's go with gdocs then since it's already there :)
Jan 22 19:01:17 Cristiano: shall we list the particicpants there, so we know who is here?
Jan 22 19:01:56 BlakeGirardot: sounds godo
Jan 22 19:01:58 dodobas: #topic rollcall
Jan 22 19:01:58 BlakeGirardot: good
Jan 22 19:02:28 Cristiano: Cool - let's get started then
Jan 22 19:02:58 dodobas: BlakeGirardot: do commands work ?
Jan 22 19:02:59 Cristiano: I believe most of you are familiar with OAM and went through the links I posted, right?
Jan 22 19:04:38 wildintellect: I meant to but never got around to installing
Jan 22 19:05:02 BlakeGirardot: I read through just about all of the materials so far
Jan 22 19:05:16 Cristiano: I went though some of the previous discussions about OAM and want to ask again what do you think OAM should be and what should NOT be
Jan 22 19:05:50 Cristiano: I started listing some of these answers in the gdoc
Jan 22 19:06:26 Cristiano: wildintellect: we don't have an installer yet :)
Jan 22 19:06:50 wildintellect: Cristiano, well there is the code on github that crschmidt wrote
Jan 22 19:07:39 Cristiano: yes, that's right, I'm not sure that it's still working though
Jan 22 19:08:36 wildintellect: well it was never quite operational in the sense that it was deployed
Jan 22 19:09:29 Cristiano: but we should definitely re-use code that was already written for sure. Especially if it was slightly operational at some point
Jan 22 19:09:40 BlakeGirardot: Is there a link for that code, I don't find it in a quick github search?
Jan 22 19:09:50 wildintellect: https://github.com/oam/oam
Jan 22 19:10:16 dodobas: Cristiano: i have done a simple code review as a preparation for the initial ideas...
Jan 22 19:10:37 dodobas: i can add it to the gdocs, what's the link ? :)
Jan 22 19:10:40 BlakeGirardot: Oh I see it is in the gdoc, thank you wildintellect
Jan 22 19:10:53 Cristiano: Oh, yes thanks dodobas, that would be great
Jan 22 19:11:18 Cristiano: I saw your review forwarded from Kate, forgot to add the link
Jan 22 19:12:06 dodobas: Cristiano: was that publicized ? :)
Jan 22 19:12:41 Cristiano: I only saw it in an email, but we should post it
Jan 22 19:12:58 dodobas: can you share gdocs link ? :)
Jan 22 19:13:12 BlakeGirardot: https://docs.google.com/document/d/1XYbZVozbuwLrI-8Rs6Wx75PxALI92s7M1eInZ-LlJsw/edit#
Jan 22 19:14:04 Cristiano: Sorry, before we go into discussing existing code, do you all agree on the definition of OAM?
Jan 22 19:14:20 Cristiano: (as defined in my list of be/not be?)
Jan 22 19:14:26 BlakeGirardot: I have a little disagreement with one of the NOT's
Jan 22 19:14:37 smathermather: Yup, I think that's a nice succinct list.
Jan 22 19:15:09 Cristiano: OK. I want to hear Blake's disagreement :)
Jan 22 19:15:09 BlakeGirardot: I think while OpenDroneMap might be operationally responsible for coding or outline workflow, OAM would definately have to be concerned with helping end users process UAV imagery
Jan 22 19:15:43 wildintellect: minor nuances about central v distributed that aren't clear in the Should be
Jan 22 19:15:43 BlakeGirardot: And I think processing UAV imagery is in a few of the existing OAM documents so we would be using their code/workflow as well
Jan 22 19:15:59 Cristiano: OK. You are saying like OSM is helping users understand how to colect data with GPS?
Jan 22 19:16:06 Cristiano: *collect
Jan 22 19:16:20 smathermather: I see UAV processing as a few different problems...
Jan 22 19:16:30 smathermather: one is storage and distribution
Jan 22 19:16:34 BlakeGirardot: I think it would be closer to helping them do something with the gps traces they collected.
Jan 22 19:16:57 smathermather: and the other is georeferencing...
Jan 22 19:17:11 smathermather: which has two parts
Jan 22 19:17:26 balrog-k2n: BlakeGirardot: that's sort of my thinking also, I would like (possibly in the future) some sort of integration of the processed imagery with raw unprocessed images
Jan 22 19:17:37 smathermather: the quick and dirty affine transformation work that's oh so valuable
Jan 22 19:17:50 wildintellect: publiclab has that covered in some ways
Jan 22 19:18:00 smathermather: exactly, wildintellect
Jan 22 19:18:20 smathermather: OpenDroneMap is about the problem of having a video stream or a bunch of images
Jan 22 19:18:25 wildintellect: the part that bugs me about public lab is that you can only do it by hand
Jan 22 19:18:32 Cristiano: From experience working with aerial imagery, quick and dirty transofrmation of non-ortho data may introduce very significnat errrors
Jan 22 19:18:33 smathermather: and you want to process to a single cohesive dataset automatically
Jan 22 19:18:40 wildintellect: which doesn't work in places without great landmarks
Jan 22 19:19:17 BlakeGirardot: I just don't think we can say we are not about helping our end users figure out what to do with raw imagery. For most of our end users, I bet that will be their #1 challenge
Jan 22 19:19:19 Cristiano: Right. You may not be able to find enough features to do a good georegistration
Jan 22 19:20:04 Cristiano: Blake: IMHO that would be the role of ODM
Jan 22 19:20:07 BlakeGirardot: It works to our advantage to to make sure they are collecting it well and georeferencing it well so we have less to worry about :)
Jan 22 19:20:08 wildintellect: I see a pipeline of OpenDroneMap -> Georegistration -> PublicLab/OAM
Jan 22 19:20:26 smathermather: Exactly.
Jan 22 19:20:58 Cristiano: Agree. But I would definetely open to post how-tos and other material on a OAM wiki
Jan 22 19:21:03 BlakeGirardot: So we would say something like: "hey work with ODM to collect and georeference your images and then contact us." ?
Jan 22 19:21:28 Cristiano: Yes, work with ODM, and then use the ODM direct upload to OAM :-)
Jan 22 19:21:43 smathermather: :)
Jan 22 19:22:11 Cristiano: ... and you data will be available tiled or for full download to anyone
Jan 22 19:22:17 BlakeGirardot: Ok, I don't disagree, but several of the docs reference that we would accept raw ungeoreferened images, or do raw images mean georeferenced?
Jan 22 19:22:36 smathermather: Yes -- licensed appropriately.
Jan 22 19:22:46 wildintellect: well raw could meant 2 different things
Jan 22 19:22:56 Cristiano: Yes, raw in this context is more like GeoTIFFs instead of tiles...
Jan 22 19:23:03 BlakeGirardot: I understood that to be the OAM-P part of the this diagram:
Jan 22 19:23:32 Cristiano: OAM-P would only deal with re-projecting (if necessary) and tiling
Jan 22 19:23:34 BlakeGirardot: http://dodobas.github.io/OpenAerialMap/oam-components/
Jan 22 19:23:44 BlakeGirardot: Oh I see. Ok, great
Jan 22 19:23:50 BlakeGirardot: My misunderstanding
Jan 22 19:24:16 Cristiano: Ideally all data coming into OAM would already be (ortho)rectified
Jan 22 19:25:12 Cristiano: That is why I believe engaging other communities around software that does mosaicking would be important
Jan 22 19:25:24 smathermather: So, any metadata flags for (quick and dirty) georeferencing vs. full orthorectification?
Jan 22 19:25:47 Cristiano: data from those software is geo-ready and then it just needs a place to make it accessible
Jan 22 19:26:13 Cristiano: smathermather: yes, that would nice also to help with the QA aspects
Jan 22 19:27:23 BlakeGirardot: So we are eliminating the "DIY imagery collector" who doesn't know about processing from one of dodobas's description docs, they would go work with ODM to get started, not us.
Jan 22 19:27:59 wildintellect: well we can provide guidance, but we would not be providing that type of user software
Jan 22 19:28:12 Cristiano: I would say so, but that is mainly to focus on the scope of OAM
Jan 22 19:28:47 BlakeGirardot: Ok, cool, just mentioning it as that if we use that doc we should revise it a bit.
Jan 22 19:28:47 wildintellect: that doesn't mean we can't make a list of software we know people can use that will output results good enough for upload
Jan 22 19:29:27 BlakeGirardot: Oh, yes, I understand, but hopefully ODM will make that one of their priorities as they probably will deal with a lot of DIY collectors
Jan 22 19:30:18 Cristiano: From what I have seen ODM is on the right track and a really cool stack of software already
Jan 22 19:30:34 wildintellect: I suspect there will be many overlapping contributors to both
Jan 22 19:30:38 BlakeGirardot: That ends my questions about oam should and should not be
Jan 22 19:30:41 smathermather: Yes, we'll have to build out docs for ODM to make onboarding easier.
Jan 22 19:31:08 smathermather: Also, hosted version, just focusing on core functionality first.
Jan 22 19:31:22 balrog-k2n: I think there's also a middle ground that can be considered. I'd like to see a service that is a catalogue of free possibly not-yet-rectified imagery but with metadata such as may be produced by OpenDroneMap, i.e. inclination, angles, field of view, etc.
Jan 22 19:32:17 Cristiano: That's a good point
Jan 22 19:32:17 smathermather: balrog-k2n -- very interesting proposition.
Jan 22 19:32:57 Cristiano: I remember seeing a lot of data from the national guard (I believe) during Sandy, that was not ortho, but still was very useful
Jan 22 19:32:58 balrog-k2n: so that a user could ask in a WMS's URL parameters for say a mosaic where objects are seen slightly from the north
Jan 22 19:33:10 balrog-k2n: obviously if data is available
Jan 22 19:33:52 dodobas: OAM should also be an imagery data archive ... raw, unprocessed, processed, ...
Jan 22 19:34:06 Cristiano: WMS and non-orth data don't go along well, maybe I don't understand your question (balrog)
Jan 22 19:34:14 smathermather: Oh, so this is not just about raw imagery but non-plan-view imagery?
Jan 22 19:34:32 wildintellect: we should probably clarify what we mean by raw
Jan 22 19:34:46 dodobas: it's distributable design should prevent data loss
Jan 22 19:34:56 Cristiano: I would say we start with ortho nadir data, but we should def consider other type of data (oblique, multiband, etc)
Jan 22 19:35:06 balrog-k2n: Cristiano: I'm not an expert but I think the task of producing a mosaic once you have all the parameters from photographs is relatively easy can could be done live, not really sure about this
Jan 22 19:35:26 Cristiano: Let's add these in the section of the gdocs for later features, so we keep track
Jan 22 19:36:04 balrog-k2n: obviously the set of parameters should be such that it would allow a ready to use geotiff to also be submitted
Jan 22 19:36:40 wildintellect: taking in raw un-mosaic could be a lot more bandwidth and disk space
Jan 22 19:36:58 smathermather: Especially from video feeds or similar...
Jan 22 19:37:00 wildintellect: and cpu time to build
Jan 22 19:37:00 Cristiano: balrog: exactly, that would be the preferred (and only, initially) option for submission)
Jan 22 19:38:04 wildintellect: here's an example of the flipside
Jan 22 19:38:04 Cristiano: right, I would see that as a task for ODM, then send then then sending the mosaic or individual frames (nadir or oblique) up to OAM
Jan 22 19:38:04 wildintellect: http://openaddresses.io/
Jan 22 19:38:28 wildintellect: they take in metadata files that reference where the real data is
Jan 22 19:38:40 wildintellect: and then occasionally rebuild a definitive dataset
Jan 22 19:39:59 Cristiano: So, the process would be similar. Data is submitted to OAM in raw (georef geotiff) format, along with a metadata file
Jan 22 19:39:59 balrog-k2n: Cristiano: my only worry about accepting only this form is that the final user is constrained by the choices of the person producing the mosaic in which photographs were used
Jan 22 19:40:04 Cristiano: then OAM (reprojects if necessary and) builds the tiles
Jan 22 19:40:38 balrog-k2n: and can't receive a WMS map that merge photographs from different sessions
Jan 22 19:40:39 wildintellect: this seems to get back to if it's a central or distributed resources
Jan 22 19:40:47 Cristiano: balrog; please elaborate
Jan 22 19:41:41 balrog-k2n: Cristiano: the workflow for me normally is to get a ton of images, choose ones for a specific purpose (e.g. based on angle, but not ground resolution), produce a geotiff from them and then use it
Jan 22 19:41:46 Cristiano: wildintellect: it will be central to start, but built to be able to have redundancy through other nodes
Jan 22 19:42:01 balrog-k2n: I would optimally make that choice later
Jan 22 19:42:51 smathermather: balrog-- on the fly mosaicing based on a set of criteria delivering geotiff, WMS, or tiles?
Jan 22 19:42:56 wildintellect: Cristiano, thats a big difference from the last iteration
Jan 22 19:43:00 Cristiano: OK. So is your concern about those other frames that you don't use?
Jan 22 19:44:08 wildintellect: I get part of the issue, is the initial user only chooses what to include
Jan 22 19:44:08 balrog-k2n: Cristiano: yes, exactly, and also the impossibility of getting a blend of my UAV / kite session with someone else's session in a neighbouring area
Jan 22 19:44:08 wildintellect: and that processing may destroy some part of the data
Jan 22 19:44:46 wildintellect: balrog-k2n, ideally the whole point of the mosaic, referencing is so that neighboring data can work together
Jan 22 19:45:04 Cristiano: OK. Well, that is part of the process of making aerial mosaics :) but we should def include features in future versions where real raw data is available
Jan 22 19:46:40 balrog-k2n: Cristiano: I would basically like the mission statement to not exclude an implementation of that at some point if possible
Jan 22 19:47:03 Cristiano: wildintellect: to go back to your point about distributed vs central
Jan 22 19:47:16 Cristiano: balrog: OK, let's add it
Jan 22 19:47:37 wildintellect: Well the original history of OAM, the big problem was that a centralized server for all imagery was simply impossible under limited budget
Jan 22 19:47:51 wildintellect: so the original model was nodes, with a central catalog
Jan 22 19:48:07 Cristiano: I think we should start simple with a central release, then make it portable and working together as nodes
Jan 22 19:48:09 wildintellect: nodes contained subsets of data, the Catalog made it all usable
Jan 22 19:48:59 wildintellect: so in that sense upload of imagery from ordinary users was never in the design, unless a node owner allowed it
Jan 22 19:49:09 Cristiano: Yes, we should def build the catalog architecure with that in mind
Jan 22 19:50:08 Cristiano: but for sake of simplicity to start, I would try building something that is self-contained and works as a single instance
Jan 22 19:50:42 wildintellect: if its a central catalog and node instances, it shouldn't matter if it's on one machine or several
Jan 22 19:50:55 wildintellect: but yes you can put both on the same hardware or rack to start
Jan 22 19:51:12 Cristiano: Exactly
Jan 22 19:51:47 balrog-k2n: wildintellect: in the 2010 atempt at OAM it could be seen that there are organisations willing to donate machines with what seemed like crazy specifications, maybe hardware is not as big a problem
Jan 22 19:51:47 wildintellect: I would build it with the correct model to start since master<-> node will be working from the beginning
Jan 22 19:51:56 Cristiano: I'm not sure how much time everyone still has, but I would like to go through some of the initial requirements
Jan 22 19:52:59 wildintellect: balrog-k2n, yes telascience has crazy machines, even I have access to some largish stuff - but neither could be counted on for long term reliability without $
Jan 22 19:54:47 wildintellect: sure we could hide that complication with a set of distributed/redundant nodes
Jan 22 19:54:59 wildintellect: so end users would get a place to upload
Jan 22 19:55:38 Cristiano: Yes, each client/node would also be set to upload to other nodes with more capacity
Jan 22 19:55:48 wildintellect: this might be one of those cases where if the data gets large enough, we have look at S3 Public domain storage and pay to get stuff out when it's needed
Jan 22 19:56:23 wildintellect: S3 might actually be a good place for older data to get archived
Jan 22 19:56:54 Cristiano: Well, part of this OAM restart effort is also to design sustainability and capacity strategies
Jan 22 19:57:15 Cristiano: so, we'll def work on these issue and find what is sustainable to start
Jan 22 19:57:49 Cristiano: Sorry, to go back to the agenda - can you guys take a look at the list of user requirements I started
Jan 22 19:58:45 wildintellect: which section of the gdoc?
Jan 22 19:58:46 Cristiano: I would like to get that squared and get some feedback from end-users to make sure we start right
Jan 22 19:59:05 Cristiano: "Initial list of tasks in order of priority"
Jan 22 19:59:22 Cristiano: first item, the link that says "here"
Jan 22 20:00:44 Cristiano: It's probably linked to our previous discussion of what OAM is/not
Jan 22 20:01:21 Cristiano: but it's also aimed at finding potential beta testers and initial supporters
Jan 22 20:02:53 BlakeGirardot: Do you have a list of people to send it to Cristiano ?
Jan 22 20:02:58 Cristiano: Please add your comments or additional questions there directly, so we can move on :)
Jan 22 20:04:15 Cristiano: I do have an initial list, but also wanted to ask the community to identify those users
Jan 22 20:04:31 BlakeGirardot: I am just curious if that is something we need to do, complile a list of organizations that we should consider users or potential contributors. I see
Jan 22 20:05:00 BlakeGirardot: So both, we have some starts and can work on identifying more.
Jan 22 20:05:42 Cristiano: I think it would be good. We could make this a survey form and automatically feed a list of those who answer plus add more manually
Jan 22 20:05:50 BlakeGirardot: I am so reluctant to add to what you have created, but aside from license issues, there are also legal issues that we need to be aware of, and I'd add a question about that.
Jan 22 20:06:18 BlakeGirardot: Maybe it doens't fit in that document now that I think about it.
Jan 22 20:06:50 Cristiano: I'd rather add and discuss it, then we can remove it if it does not fit :)
Jan 22 20:07:09 BlakeGirardot: Ok, I'll add it if you don't mind, yank it as you see fit :)
Jan 22 20:07:40 Cristiano: The next thing I would like your feedback on it the other "here" link
Jan 22 20:07:50 Cristiano: (the basic list of metadata)
Jan 22 20:08:09 Cristiano: that was adapted from Drazen's work on GitHub
Jan 22 20:08:13 wildintellect: I think we still need to clarify definitions of raw
Jan 22 20:09:11 Cristiano: should we use "raw frames" and "raw geotiffs"
Jan 22 20:09:14 Cristiano: ?
Jan 22 20:09:22 wildintellect: raw in my mind is what came off the camera directly
Jan 22 20:09:40 Cristiano: Yes, that's raw too :)
Jan 22 20:09:52 wildintellect: I don't consider WCS raw
Jan 22 20:09:59 wildintellect: but when compared to WMS
Jan 22 20:10:03 wildintellect: some might call it that
Jan 22 20:10:35 Cristiano: OK, so how about "unprocessed frames" and "processed ortho mosaic"?
Jan 22 20:10:51 wildintellect: that might be more clear
Jan 22 20:10:53 Cristiano: WCS and WMS are only services
Jan 22 20:11:05 Cristiano: WMS is a representation
Jan 22 20:11:18 wildintellect: sure but WCS serves the data as is, where WMS can be resampled
Jan 22 20:11:25 Cristiano: WCS could be "raw" as in that you can get GeoTIFFs out of it
Jan 22 20:11:32 Cristiano: right
Jan 22 20:12:09 wildintellect: and one step back is what was mentioned earlier of even more original obliques
Jan 22 20:12:52 BlakeGirardot: As someone not as well versed as everyone else here :) Raw to me means right off the camera
Jan 22 20:13:09 wildintellect: and I think we should stick to that definition
Jan 22 20:13:09 Cristiano: OK. wildintellect: Can you add notes about these definitions in the gdoc?
Jan 22 20:13:19 balrog-k2n: in the Metadata Schema some more of the entries can perhaps be obtained automatically from the image, I don't see anything missing in the list
Jan 22 20:13:26 Cristiano: agree, RAW (capital) is what comes off the camera
Jan 22 20:13:39 BlakeGirardot: Aye, RAW
Jan 22 20:13:58 Cristiano: OK. thanks balrog. Please add notes about it
Jan 22 20:14:31 wildintellect: remember many people also take jpg (raw)
Jan 22 20:14:52 balrog-k2n: also a URI could be a assigned instead of the UUID (not opinion on what is better)
Jan 22 20:14:53 smathermather: (or video frames... or or
Jan 22 20:15:01 Cristiano: I am thinking that list of matadata would be compiled together with the geotiff as an XML/text file and uploaded with the submission to the OAM catalog
Jan 22 20:15:51 Cristiano: Yes, I'm thinking what URI vs UUID may involve in a disconnected environemnt
Jan 22 20:17:13 Cristiano: So, once we have these basic metadata and the geotiff, what shall we do? :))
Jan 22 20:17:21 smathermather: A UUID is easy to generate without user input, which has advantages.
Jan 22 20:17:36 Cristiano: ... let's talk about the catalog idea?
Jan 22 20:17:50 BlakeGirardot: Ok, again, newbi here: Is there a need to distinguish between meta data on the indivdiual images or files and one the dataset as a whole?
Jan 22 20:18:40 wildintellect: ideally there should be metadata per image (most values in a set can be replicated)
Jan 22 20:18:49 wildintellect: so that images can be used outside their original set
Jan 22 20:19:05 wildintellect: at least thats the impression I got from the existing docs
Jan 22 20:19:25 Cristiano: Yes, there's EXIF-like matadata which is automaticall created by the sensor, but also dataset metadata, which is more what I'm interested to start
Jan 22 20:19:48 BlakeGirardot: Ok, thank you both, I see.
Jan 22 20:20:35 Cristiano: dataset (AKA ortho geotiff mosaic) metadata needs some user input
Jan 22 20:22:21 Cristiano: So, and this basic metadata set together with spatial ineherited metadata is the information used to populate the catalog
Jan 22 20:23:10 Cristiano: so, a user can search both geographically and by dataset properties
Jan 22 20:23:19 smathermather: One small thing -- should "EPSG format" be more specific, e.g. "proj4 string" or similar?
Jan 22 20:23:57 wildintellect: EPSG format is EPSG:####
Jan 22 20:24:01 wildintellect: not a proj string
Jan 22 20:24:13 Cristiano: yes, same as used in WMS
Jan 22 20:24:34 smathermather: Ah, so no use of non EPSG coordinate systems... :)
Jan 22 20:24:38 smathermather: ?
Jan 22 20:24:53 wildintellect: that helps to avoid getting strange proj we can't reproject into standard services
Jan 22 20:25:28 Cristiano: agree, but we can look into other CRSs later :-)
Jan 22 20:26:30 balrog-k2n: I'm afraid there are already systems in the EPSG set that require an external grid file ;)
Jan 22 20:26:52 Cristiano: The catalog development will be the first tech challenge. Do you guys know of code that we can start with or adapt?
Jan 22 20:28:34 Cristiano: Anyone familiar with Geonode?
Jan 22 20:28:46 wildintellect: Yes
Jan 22 20:29:05 wildintellect: I have installed it several times over the years
Jan 22 20:29:23 wildintellect: I would start with pycsw
Jan 22 20:29:40 wildintellect: I think that's something that didn't exist back when previous code was written
Jan 22 20:29:59 wildintellect: there are several issues with using geonode in this context
Jan 22 20:30:08 Cristiano: Good, thanks for pointing it out
Jan 22 20:30:18 Cristiano: let's add a link to the gdoc
Jan 22 20:31:37 Cristiano: I think we should def start with something lightweight
Jan 22 20:31:45 smathermather: start with pycsw for simplicity?
Jan 22 20:31:57 wildintellect: that would be the oam code on github perhaps?
Jan 22 20:32:00 Cristiano: not tot familiar with pycsw, but it looks like it is :-)
Jan 22 20:32:16 wildintellect: yes pycsw just serves a catalog of your data
Jan 22 20:32:55 wildintellect: over the csw standard
Jan 22 20:33:19 wildintellect: it's used on data.gov in conjunction with ckan
Jan 22 20:33:38 wildintellect: the authors are quite accessible
Jan 22 20:33:47 Cristiano: perfect, and I guess we can easily extend to schema to include OAM specific namespaces
Jan 22 20:33:49 wildintellect: though I know one is in another meeting right now
Jan 22 20:34:23 Cristiano: great, let's try to connect with them
Jan 22 20:34:52 wildintellect: eoxserver would be another tool to look at
Jan 22 20:35:29 Cristiano: Can you list them on the gdoc wildintellect?
Jan 22 20:35:41 Cristiano: (the devs that you mentioned)
Jan 22 20:35:42 Cristiano: thanks
Jan 22 20:37:00 Cristiano: eoxserver looks interesting and useful too
Jan 22 20:37:54 Cristiano: That probably leads us to the next Tech Challenge, the development of the actual first OAM node
Jan 22 20:38:06 Cristiano: OAM-S + OAM-P
Jan 22 20:38:23 wildintellect: I think most of the software is on http://live.osgeo.org - if you want a testbed
Jan 22 20:38:46 Cristiano: I'll def check it out
Jan 22 20:39:29 Cristiano: anyone familiar with tilestache or lightweight tile serving software?
Jan 22 20:39:38 dodobas: yes...
Jan 22 20:39:46 dodobas: you name it i got it :)
Jan 22 20:39:56 wildintellect: yes
Jan 22 20:39:59 Cristiano: Bravo! :-)
Jan 22 20:40:15 Cristiano: how much can they scale do you think?
Jan 22 20:40:29 wildintellect: tilestache has a lot of potential with uqsgi
Jan 22 20:40:32 wildintellect: uwsgi
Jan 22 20:40:45 wildintellect: my instances have never been hit that hard though
Jan 22 20:40:52 wildintellect: but I'm pretty sure it's what Stamen uses
Jan 22 20:40:54 dodobas: you are alway bound by IO... as you are reading files of a hard drvive
Jan 22 20:41:09 wildintellect: sure, but thats what SSDs are for
Jan 22 20:41:20 dodobas: and then network IO
Jan 22 20:41:36 wildintellect: thats what universities and google/amazon are for
Jan 22 20:41:50 dodobas: most providers will cap network traffic/bandwidth
Jan 22 20:42:31 dodobas: or just charge more, but in essence, tile 'proxy' middleware has almost no overhead
Jan 22 20:42:59 dodobas: unless, you are doing something fancy, like with maproxy, and combining map layers on the fly
Jan 22 20:42:59 nhv: caching tile servers scale very well
Jan 22 20:43:15 dodobas: or reprojecting/resampling
Jan 22 20:43:53 Cristiano: well, there will be need for on-demand tiling
Jan 22 20:44:09 Cristiano: especially when there's fresh data that needs quick turnaround
Jan 22 20:44:28 wildintellect: that seems like a good place to have discussions with people like mapbox
Jan 22 20:44:53 wildintellect: or where paying for amazon is useful
Jan 22 20:44:54 Cristiano: and, yes, CPU (GPU?) power for repro and tiling in general
Jan 22 20:45:13 dodobas: there are no GPU tiles AFAIK...
Jan 22 20:45:16 dodobas: *tilers
Jan 22 20:45:28 Cristiano: there should be :)
Jan 22 20:45:36 dodobas: GPUs always have the context switching problem...
Jan 22 20:45:37 nhv: NASA open sourced some interesting tile server code recently https://wiki.earthdata.nasa.gov/display/GIBS/2014/02/04/OnEarth+and+MRF+Now+Available+on+GitHub
Jan 22 20:46:17 dodobas: you need to read something of a 'slow' disk, pump i through bus to GPU memory..
Jan 22 20:46:29 dodobas: then pump it out again ...
Jan 22 20:47:48 Cristiano: I would interested to know how much compression would be a factor in these processes
Jan 22 20:48:31 dodobas: it's probably doable, but at the same time, it's much eaiser to just spin up more instances (server) for 'burst' rendering
Jan 22 20:48:35 Cristiano: anyway, do you guys know who should we talk to at Mapbox about the processing/tiling stuff?
Jan 22 20:48:57 wildintellect: Cristiano, absolutely compression matters, but higher compression also has longer read times
Jan 22 20:49:01 Cristiano: or maybe someone is already in this room :-)
Jan 22 20:49:54 Cristiano: right, but if we can offset that to the GPU..... I'm not expert here, just thinking in abstract terms
Jan 22 20:50:13 wildintellect: I'm not sure GPU will really deliver on what you want here
Jan 22 20:50:24 wildintellect: GPU itself is difficult to scale
Jan 22 20:50:30 wildintellect: due to heat issues
Jan 22 20:50:40 wildintellect: the big GPU farms do liquid immersion
Jan 22 20:50:55 wildintellect: it also means you have to rewrite your code
Jan 22 20:50:57 nhv has never used GPU when building tile caches
Jan 22 20:51:16 wildintellect: I'd say existing compute clusters might be better suited
Jan 22 20:51:23 wildintellect: if you are trying to preseed
Jan 22 20:51:53 wildintellect: so Mapbox missing maps site says talk to https://twitter.com/brunosan
Jan 22 20:52:05 nhv: I have found pre seeding is usually io bound
Jan 22 20:52:24 wildintellect: though that page makes it sound like HOT already has a contact with mapbox
Jan 22 20:52:35 wildintellect: nhv, yes, but that I have some ways to solve
Jan 22 20:52:52 Cristiano: Yes, I should talk to Kate about a contact
Jan 22 20:53:08 wildintellect: ie we could do tile stuff with hadoop
Jan 22 20:53:25 wildintellect: map/reduce style - though primarly that would just be a map function
Jan 22 20:53:32 wildintellect: and could be written in python
Jan 22 20:54:02 nhv: python map async works well for tile seeding  :-)
Jan 22 20:54:11 Cristiano: Good, I'd like to hear implementation ideas
Jan 22 20:54:35 Cristiano: please add any relevant link or idea directly to the gdoc
Jan 22 20:55:18 dodobas: there is no need for anything fancy like hadoop/map reduce...
Jan 22 20:55:43 wildintellect: dodobas, I agree we are unlikely to get there
Jan 22 20:55:48 Cristiano: Now, getting into practical scheduling, we're going to need to get people coding/piecing code together soon
Jan 22 20:55:55 wildintellect: I just happen to already have a hadoop cluster
Jan 22 20:56:31 Cristiano: who's available to help me draft these two tech challenges?
Jan 22 20:56:49 wildintellect: alexbarth, did someone tell you we mentioned mapbox?
Jan 22 20:57:04 Cristiano: Just to make sure we're going in the right direction with code and implementation
Jan 22 20:57:17 alexbarth: wildintellect: hey there - mentioned where?
Jan 22 20:57:25 wildintellect: before you logged in
Jan 22 20:57:32 wildintellect: here
Jan 22 20:57:53 wildintellect: we were discussing how to handle spikes in demand from OAM when a crisis comes up
Jan 22 20:58:25 wildintellect: mapbox came up as a group that would know how to handle that or possible be an avenue to serve a layer through
Jan 22 20:59:31 wildintellect: or at least provide some technical advise on the topic
Jan 22 21:00:29 Cristiano: We were looking into existing OS software that can handle tiling and serving in a lightwieght scalble way
Jan 22 21:00:43 Cristiano: mentioned tilestache
Jan 22 21:01:19 Cristiano: and that that's probably something MB uses
Jan 22 21:01:31 wildintellect: well I said Stamen
Jan 22 21:01:46 Cristiano: oops
Jan 22 21:02:35 wildintellect: MB definitely has the scaling part down
Jan 22 21:03:01 Cristiano: OK, well let's talk to MB and see if they're keen on advising
Jan 22 21:03:46 Cristiano: No, for the logistics: who's managing the HOT OAM GitHub?
Jan 22 21:04:01 Cristiano: *Now
Jan 22 21:04:08 wildintellect: which oam
Jan 22 21:04:17 wildintellect: oam wasn't always under HOT
Jan 22 21:04:20 Cristiano: That should probably be the place to start collecting issues and feature requests, right?
Jan 22 21:04:33 dodobas: Cristiano: anyone who is admin on HOTOSM GH organization
Jan 22 21:04:37 wildintellect: so you mean this one https://github.com/oam/oam?
Jan 22 21:05:03 Cristiano: no, the one under HOT
Jan 22 21:05:05 wildintellect: https://github.com/hotosm/OpenAerialMap
Jan 22 21:07:23 Cristiano: I would go with the HOT one, unless we have access to the other account
Jan 22 21:07:58 Cristiano: But anyway, it's more for keeping track of development, issues, and feature proposals
Jan 22 21:08:01 wildintellect: I can get us access with an email to Chris
Jan 22 21:08:07 wildintellect: but seems fine to do it under HOT
Jan 22 21:08:36 wildintellect: Cristiano, note: I'm the chair of OSGeo sys admin committee
Jan 22 21:08:48 Cristiano: OK. And maybe it makes sense, since that's where the funding comes from
Jan 22 21:09:02 Cristiano: oh, cool
Jan 22 21:09:49 Cristiano: well, it'd be great to get OSGeo on board with this effort as well
Jan 22 21:11:02 Cristiano: I'm also thinking we could get someone through GSoC with OSGEO if that's possible
Jan 22 21:11:13 nhv: the 'right way' to get OSGeo on board is to become an OSGeo project
Jan 22 21:11:38 nhv: probably a good fit for osgeo labs once you get some code going
Jan 22 21:11:49 Cristiano: OK, I'd be happy to look into it
Jan 22 21:12:00 nhv: http://wiki.osgeo.org/wiki/OSGeo_Labs
Jan 22 21:12:35 nhv: wildintellect: are you going to contact Chris  ?
Jan 22 21:13:07 wildintellect: Cristiano, yes we could try to put it under OSGeo for GSoC
Jan 22 21:13:17 wildintellect: you should get on the specific mailing list for that
Jan 22 21:13:26 wildintellect: nhv, if we need to
Jan 22 21:13:56 nhv: it would be good to at least fyi him
Jan 22 21:14:24 Cristiano: OK, I will. What's the best list to post to?
Jan 22 21:14:33 wildintellect: nhv, I think he's on the Talk-OAM list
Jan 22 21:14:45 nhv: :-) probably
Jan 22 21:15:07 Cristiano: Yes please, if you guys have direct contact with Chris, let's making him aware of this new effort
Jan 22 21:15:14 wildintellect: Cristiano, for GSoC http://lists.osgeo.org/mailman/listinfo/soc
Jan 22 21:16:03 Cristiano: Cool, I will post there
Jan 22 21:16:37 wildintellect: nhv, you want to email Chris
Jan 22 21:17:08 Cristiano: dodobas - can you start to clean up the HOT OAM GitHub with this updated discussion?
Jan 22 21:17:38 Cristiano: and give access to some of us to help?
Jan 22 21:18:00 dodobas: Cristiano: well just create a pull request
Jan 22 21:18:22 dodobas: anyway, you should probably be an hotosm GH admin
Jan 22 21:18:29 Cristiano: OK
Jan 22 21:18:40 dodobas: what's your GH account?
Jan 22 21:18:55 Cristiano: I think it's time to wrap up. Are you guys OK to meeting again in a week?
Jan 22 21:19:23 Cristiano: And we'll use the ML to continue some of the discussion in the meantime
Jan 22 21:20:43 Cristiano: Feel free to add to the gdoc anything that we forgot to note
Jan 22 21:21:06 Cristiano: and I think Blake will take care of posting the log from this meeting to the wiki
Jan 22 21:22:17 wildintellect: can probably make it again next week, would be good to shoot for shorter meetings down the road
Jan 22 21:22:51 Cristiano: Anything else? I think we did a lot of good brainstorming. Yes, agree on keeping it shorter :)
Jan 22 21:24:07 Cristiano: nhv and wildintellect, I'm not sure I have your contact info, but as long as you are on the OAM list
Jan 22 21:24:34 nhv: I am not on OAM list nhv at cape dot com
Jan 22 21:24:43 Cristiano: Thanks everyone for joining!
Jan 22 21:24:54 Cristiano: nhv: thanks
Jan 22 21:25:26 wildintellect: its not hard to find me, but I am on the list
Jan 22 21:25:47 nhv easy to find also
Jan 22 21:26:09 wildintellect: nhv, well if they know what #IRC channel to find you in
Jan 22 21:26:29 wildintellect: <- is my nick everywhere but twitter
Jan 22 21:26:55 nhv: there aren't too many 'nhv's on the web  :-)
Jan 22 21:31:19 wildintellect: nhv you're not even on the first page of google results for NHV
Jan 22 21:38:20 BlakeGirardot: Thank you Cristiano and everyone else !

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=OpenAerialMap/Meeting_Jan_22,_2015&oldid=1586424"