Osm2pgsql/benchmarks/Dell R610 import revodrive

Hardware: FR:Servers/Hardware#Dell_R610Dell R610 with dual Xeon 6 cores: total 12 cores @ 2.93 GHz + 64GB RAM

SSD on PCIe: Revodrive 3 480GB (P/N: RVD3-FHPX4-480G this is not the 'X2' model)

OS: Ubuntu 12.04LTS (kernel 3.2 needed to get the Revodrive detected by the kernel).

Planet 2013-01-30 import with hstore and flatnodes (will be updated by diffs)

osm2pgsql --create --number-processes 10 -C 16000 -k --flat-nodes /ssd/osm2pgsql/flat-nodes.raw   -s -S ./default.style -G -m --unlogged 


osm2pgsql SVN version 0.81.0 (64bit id space)

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE:  table "planet_osm_point" does not exist, skipping
NOTICE:  table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE:  table "planet_osm_line" does not exist, skipping
NOTICE:  table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE:  table "planet_osm_polygon" does not exist, skipping
NOTICE:  table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE:  table "planet_osm_roads" does not exist, skipping
NOTICE:  table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache   
Sharing dense sparse
Node-cache: cache=16000MB, maxblocks=2048001*8192, allocation method=11
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0  
Mid: pgsql, scale=100 cache=16000
Setting up table: planet_osm_nodes
NOTICE:  table "planet_osm_nodes" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "pla
net_osm_nodes"
Setting up table: planet_osm_ways
NOTICE:  table "planet_osm_ways" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE:  table "planet_osm_rels" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Reading in file: /home/jocelyn/planet-130130.osm.pbf
Processing: Node(1755043k 1513.0k/s) Way(167918k 27.66k/s) Relation(1782000 334.71/s)  parse time: 12554s

Node stats: total(1755043102), max(2134381692) in 1160s
Way stats: total(167918844), max(203427217) in 6070s
Relation stats: total(1782000), max(2730466) in 5324s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending ways...
Maximum node in persistent node cache: 2134900735
        99602563 ways are pending
Using 10 helper-processes
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Helper process 0 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 9 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 2 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 6 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 3 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 7 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 4 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 8 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 1 out of 10 initialised
processing way (0k) at 0.00k/s (done 0 of 10)Helper process 5 out of 10 initialised
Process 0 finished processing 9960257 ways in 3729 sec
Process 1 finished processing 9960257 ways in 3741 sec
processing way (99184k) at 26.52k/s (done 2 of 10)Maximum node in persistent node cache: 2134900735
Process 7 finished processing 9960256 ways in 3747 sec
Maximum node in persistent node cache: 2134900735
Process 3 finished processing 9960256 ways in 3747 sec
Maximum node in persistent node cache: 2134900735
Process 5 finished processing 9960256 ways in 3749 sec
Maximum node in persistent node cache: 2134900735
Process 9 finished processing 9960256 ways in 3754 sec
Maximum node in persistent node cache: 2134900735
Process 2 finished processing 9960257 ways in 3755 sec
Maximum node in persistent node cache: 2134900735
Process 8 finished processing 9960256 ways in 3755 sec
Maximum node in persistent node cache: 2134900735
Process 4 finished processing 9960256 ways in 3766 sec
Maximum node in persistent node cache: 2134900735
Process 6 finished processing 9960256 ways in 3788 sec
Maximum node in persistent node cache: 2134900735

All child processes exited

99602563 Pending ways took 3788s at a rate of 26294.24/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
Maximum node in persistent node cache: 2134900735
        0 relations are pending

Using 10 helper-processes
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Mid: loading persistent node cache from /ssd/osm2pgsql/flat-nodes.raw
Maximum node in persistent node cache: 2134900735
Process 0 finished processing 0 relations in 3 sec
Process 9 finished processing 0 relations in 3 sec
Maximum node in persistent node cache: 2134900735
Process 5 finished processing 0 relations in 3 sec
Maximum node in persistent node cache: 2134900735
Process 1 finished processing 0 relations in 3 sec
Maximum node in persistent node cache: 2134900735
Process 6 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735
Process 2 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735
Process 7 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735
Process 3 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735
Process 8 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735
Process 4 finished processing 0 relations in 4 sec
Maximum node in persistent node cache: 2134900735

All child processes exited
0 Pending relations took 4s at a rate of 0.00/s

node cache: stored: 1755043102(100.00%), storage efficiency: 86.55% (dense blocks: 1857756, sparse nodes: 62738887), hit rate: 100.00%
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_point
Maximum node in persistent node cache: 2134900735
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_ways (fastupdate=off)
Building index on table: planet_osm_rels (fastupdate=off)
Stopped table: planet_osm_nodes in 0s
Analyzing planet_osm_polygon finished
Analyzing planet_osm_point finished
Analyzing planet_osm_roads finished
Analyzing planet_osm_line finished
Stopped table: planet_osm_rels in 42s
Copying planet_osm_point to cluster by geometry finished
Copying planet_osm_roads to cluster by geometry finished
Creating indexes on  planet_osm_roads finished
All indexes on  planet_osm_roads created  in 1190s
Completed planet_osm_roads
Creating indexes on  planet_osm_point finished
All indexes on  planet_osm_point created  in 1629s
Completed planet_osm_point
Copying planet_osm_polygon to cluster by geometry finished
Copying planet_osm_line to cluster by geometry finished
Creating indexes on  planet_osm_polygon finished
All indexes on  planet_osm_polygon created  in 6547s
Completed planet_osm_polygon
Creating indexes on  planet_osm_line finished
All indexes on  planet_osm_line created  in 6986s
Completed planet_osm_line
Stopped table: planet_osm_ways in 8255s   

Osm2pgsql took 24604s overall