User:Mmd/Overpass API/Performance Project 2016/Full Attic DB Setup

From OpenStreetMap Wiki
Jump to: navigation, search

Summary

Prerequisites:

  • Initial setup of database with first ODbL compliant planet is already done, dispatcher is up and running.

Multi Diff based update

  • Update based on daily diffs
  • Preparation: load daily diffs (instead of minutely diffs) from planet OSM server: nohup ./fetch_osc.sh 0 http://planet.openstreetmap.org/replication/day [osc_target_directory]
  • Copy & adjust script apply_osc_to_db.sh to cater for up to 4 GB of .osc data: ( `du -m $TEMP_DIR | awk '{ print $1; }'` -le 4000 ) ]];.
  • Run db update: nohup ./apply_osc_to_db.sh [osc_target_directory] 0 --meta=attic &
  • One update_from_dir call will now process between 6-12 days of OSM data. Due to a peak memory consumption of > 15GB, at least 32GB main memory (!) is highly recommended.


  • Total runtime for 1341 days OSM data: 4 days
  • Parts of the db was moved to hard disk during the update (space constraints)
  • Total db size (excl. areas): 173GB (after clone db!)


Comparison Single Diff based (left) vs. Multi diff based update (right)

Overpass API Perf2016 Attic DB replication lag 2.png

Overpass API Perf2016 Attic DB memory multi.png

Overpass API Perf2016 Attic DB cpu multi.png

Overpass API Perf2016 Attic DB disk util multi.png

Overpass API Perf2016 Attic DB disk throughput multi.png

Single Diff based update

Summary

  • based on script fetch_osc_and_apply.sh
  • update_database --flush-size=0 is essential to avoid database corruption -> adjust script accordingly!
  • Frequent DB reorganization: update process was stopped several times to clone database and continue DB update on cloned DB -> massive reduction in DB size, up to 50 GB(!) during first run
  • Total runtime for 1300+ days (starting September 2012 thru present) about 8-8,5 days.
  • Total runtime without parallel processing / db reorganization: 13 days
  • Initial full area creation: 4,5 hours (all on SSD) - (see comment)
  • (v0.7.52 needs 13,5 hours for full area creation)
  • Resulting DB size: ~180GB

Munin stats

Overpass API Perf2016 Attic DB replication lag.png


Overpass API Perf2016 Attic DB disk util.png


Overpass API Perf2016 Attic DB cpu.png


Overpass API Perf2016 Attic DB df.png


Overpass API Perf2016 Attic DB memory.png