Essen Developers Workshop/Data Replication
Data replication and distribution
The Idea is that write requests go to a central database and read requests are satisfied from a seperate hierarchical set of local cache servers
central server | +----+---+ | | | +------+-+ | +----+-------+ | | | | | ics ics germany uk | | | cache cache | +---+--+ | other | | karlsruhe customized local local cache clients server server
- central server: the OSM server we are using now
- ics: intermediary cache server, there can be any amount of these. I expect them to be complete mirrors
- local server: a server that is run by any interested person who needs fast read access and is willing to set up a caching server (e.g., Osmarender developers, Editor developers, ...)
- possible other clients: Dirty Tile marker, RSS feed, mapnik converter, ...
- The central server increases a counter on each change of the data.
- This a counter is referred as "sync point" from now on. The sync point can be a timestamp, a steadily increasing counter, or whatever.
- The clients (JOSM, tiles@home, mapnik, whatever) makes a read request to his caching server.
- The read request should be something like the If-Modified-Since header of HTTP.
- The caching server sends its last known sync point to its upstream server.
- The upstream server sends a block of data (quite possibly in XML)
- The local server persists those changes to its own, local database and stores the successful "sync point" in its local database
- The local server notifies the upstream of the successful synchronization so that the upstream can purge the data stored up to that point for that client
- The local caches can be initially seeded with the latest planet file. The seeding would not be needed later on.
- The client can notify its upstream of the kind of data it is interested in (bounding boxes, tag combinations like "complete railroad map", "complete water borders").
- The server can stop preparing update data if a local cache server has not asked for its updates for a configurable amount of time (one day? one week?) and restart only if the local cache reappears again.
- Multiple bounding boxes (my local cache is interested in the data of Munich, Bagdad and Turkey)
- Chunking of update data? What my local server wants to re-sync just after the complete TIGER data was imported?
- The features / bounding boxes a client / cache is interested in should not be re-sent with each request. Create a seperate call for that.
- What if a client changes the features it is interested in later? A cache that collected railroad data might ask for airports later. How to we force a re-sync? Re-seed with the current planet data?
- How does a client discover the cache it is supposed to read from? GeoIP?
- Push / or Pull: Steve prefers Pull
- If the socket is already open, send additional data without without being asked. New request type for this?
- the client can request a maximum amount of data it wants to receive
- The server / client has to be set up by an administrator so Joe Random User can't swamp the main server