User:Mmd/Planet import challenge 22

From OpenStreetMap Wiki
Jump to navigation Jump to search

Runtime comparison

Using lz4 map file compression

Version Overpass API 0.7.58.5 Overpass API 0.7.59.120 (mmd)
Commit 766266ec136f2e49ec5243f83d8741f3f6748bbb 3b1e7b34b4c5b69918b9f61a3c5d642477db8c83 a4479f84bb767813eacf6b3443e167af75e32eab 465a3fef2f53ba559366fb62ace07aaa9cca8df4 9ef1e5b23a0f7ffcc1393a5498ccdb09318f1392 9ef1e5b23a0f7ffcc1393a5498ccdb09318f1392 9ef1e5b23a0f7ffcc1393a5498ccdb09318f1392
flush size = 16 (default) flush size = 64 flush size = 16 (default) flush size = 64 flush size = 64 flush size = 64 flush size = 48 flush size = 32 flush size = 64
User time (seconds) 99982.26 87259.37 55720.12 44965.89 34340.63 33374.36 34697.80 37377.67 33270.08
System time (seconds) 3753.62 3176.61 4088.99 3553.10 3382.84 3424.39 3366.12 3512.75 3375.83
Percent of CPU this job got 95% 94% 158% 154% 137% 136% 139% 141% 137%
Elapsed (wall clock) time (h:mm:ss or m:ss): 30:12:17 26:32:17 10:29:47 8:42:09 7:36:42 7:28:18 7:35:50 8:03:07 7:22:40
Maximum resident set size (kbytes) 7958192 27284816 7849728 23197216 24712892 24766280 20856304 12998596 22568672
  • Overpass API 0.7.58.5: osmium cat ~/planet/planet-220829.osm.pbf -f xml | /usr/bin/time -v ./update_database --db-dir=/db --meta --compression-method=lz4 --map-compression-method=lz4
  • Overpass API 0.7.59.120 (mmd): /usr/bin/time -v src/update_database --db-dir=/db --use-osmium --meta --parallel=10 --initial-load < ~/planet/planet-220829.osm.pbf
  • Overpass API 0.7.59.120 (mmd) flush size 64: /usr/bin/time -v src/update_database --db-dir=/db --use-osmium --meta --flush-size=64 --parallel=10 --initial-load < ~/planet/planet-220829.osm.pbf

Resulting database size: about 230G


With no map file compression

Version Overpass API 0.7.58.5 Overpass API 0.7.59.120 (mmd)
Commit 465a3fef2f53ba559366fb62ace07aaa9cca8df4
flush size = 16 (default) flush size = 64
User time (seconds) 95595.48 31566.81
System time (seconds) 9238.92 8212.37
Percent of CPU this job got 86% 107%
Elapsed (wall clock) time (h:mm:ss or m:ss): 33:32:58 10:14:22
Maximum resident set size (kbytes) 7490992 26060352


  • Overpass API 0.7.58.5: osmium cat ~/planet/planet-220829.osm.pbf -f xml | /usr/bin/time -v ./update_database --db-dir=/db --meta --compression-method=lz4 --map-compression-method=no
  • Overpass API 0.7.59.120 (mmd) flush size 64: /usr/bin/time -v src/update_database --db-dir=/db --use-osmium --meta --flush-size=64 --parallel=10 --initial-load --map-compression-method=no < ~/planet/planet-220829.osm.pbf


Resulting database size: 258G - 263G

CMake options

  • CMAKE_CXX_FLAGS_RELWITHDEBINFO: -O2 -g -DNDEBUG -ftree-vectorize -march=native -flto
  • CMAKE_CXX_COMPILER: /usr/bin/g++-12
  • CMAKE_EXE_LINKER_FLAGS: -flto -fwhole-program
  • USE_UNITY_BUILD: ON


  • Compiler: g++-12 (Ubuntu 12.1.0-2ubuntu1~22.04) 12.1.0
  • Running on Ubuntu 22.04

Hardware

Planet file stats

Name: planet-220829.osm.pbf
Format: PBF
Compression: none
Size: 69869926964
Timestamps:
  First: 2005-05-21T21:03:22Z
  Last: 2022-08-28T23:59:56Z
Number of changesets: 0
Number of nodes: 7882127434
Number of ways: 883613025
Number of relations: 10176596
Smallest changeset ID: 0
Smallest node ID: 1
Smallest way ID: 37
Smallest relation ID: 11
Largest changeset ID: 0
Largest node ID: 9984322083
Largest way ID: 1089962152
Largest relation ID: 14514635
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:Mmd/Planet_import_challenge_22&oldid=2420999"