Osm2pgsql/benchmarks/HP Z800 import 1
< Osm2pgsql | benchmarks
Jump to navigation
Jump to search
PauLLA - www.paulla.asso.fr
Planet import
mapnik@iemt003:~$ date; time osm2pgsql -d mapnik --tablespace-main-data data --tablespace-main-index index --tablespace-slim-data data --tablespace-slim-index index -p planet_osm -s -C 16000 --hstore planet-120613.osm.pbf; date
jeudi 21 juin 2012, 18:25:57 (UTC+0200)
osm2pgsql SVN version 0.80.0 (32bit id space)
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=16000MB, maxblocks=2048001*8192, allocation method=11
Mid: pgsql, scale=100 cache=16000
Setting up table: planet_osm_nodes
NOTICE: table "planet_osm_nodes" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: table "planet_osm_ways" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: table "planet_osm_rels" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"
Reading in file: planet-120613.osm.pbf
Processing: Node(1485458k 206.6k/s) Way(139049k 25.08k/s) Relation(560810 211.95/s)
Standard exception processing way_id 1082366: TopologyException: side location conflict at 822832 8.30585e+06
Processing: Node(1485458k 206.6k/s) Way(139049k 25.08k/s) Relation(1097950 238.43/s)
Standard exception processing way_id 1792031: TopologyException: side location conflict at -734926 5.24355e+06
Processing: Node(1485458k 206.6k/s) Way(139049k 25.08k/s) Relation(1443610 247.07/s) parse time: 18576s
Node stats: total(1485458339), max(1785763556) in 7189s
Way stats: total(139049263), max(167173297) in 5544s
Relation stats: total(1443613), max(2228587) in 5843s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending ways
Using 1 helper-processes
processing way (80017k) at 3.29k/s
Process 0 finished processing 80017402 ways in 24292 sec
All child processes exited
80017402 Pending ways took 24292s at a rate of 3293.98/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending relations
Using 1 helper-processes
Process 0 finished processing 0 relations in 0 sec
All child processes exited
node cache: stored: 1485458339(100.00%), storage efficiency: 89.27% (dense blocks: 1437032, sparse nodes: 128328180), hit rate: 100.00%
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_rels
Building index on table: planet_osm_ways (fastupdate=off)
Building index on table: planet_osm_rels (fastupdate=off)
Analyzing planet_osm_polygon finished
Analyzing planet_osm_point finished
Analyzing planet_osm_line finished
Analyzing planet_osm_roads finished
Stopped table: planet_osm_rels in 52s
Copying planet_osm_roads to cluster by geometry finished
Copying planet_osm_point to cluster by geometry finished
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 1170s
Completed planet_osm_roads
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 2873s
Completed planet_osm_point
Copying planet_osm_line to cluster by geometry finished
Copying planet_osm_polygon to cluster by geometry finished
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 6201s
Completed planet_osm_line
Creating indexes on planet_osm_polygon finished
All indexes on planet_osm_polygon created in 6672s
Completed planet_osm_polygon
Stopped table: planet_osm_ways in 10539s
Osm2pgsql took 53410s overall
real 890m10.352s
user 316m1.641s
sys 28m11.178s
vendredi 22 juin 2012, 09:16:07 (UTC+0200)
mapnik@iemt003:~$
Daily diff
Imported in less than one hour
mapnik@iemt003:~$ osmosis --read-replication-interval workingDirectory=${WORKOSM_DIR} --simplify-change --write-xml-change ${CHANGE_FILE}
Updating tile server at vendredi 22 juin 2012, 14:00:01 (UTC+0200)
22 juin 2012 14:00:01 org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.40.1
22 juin 2012 14:00:02 org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
22 juin 2012 14:00:02 org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
22 juin 2012 14:00:02 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
22 juin 2012 14:01:25 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline complete.
22 juin 2012 14:01:25 org.openstreetmap.osmosis.core.Osmosis run
INFO: Total execution time: 83798 milliseconds.
mapnik@iemt003:~$ osm2pgsql -d mapnik --tablespace-main-data data --tablespace-main-index index --tablespace-slim-data data --tablespace-slim-index index -p planet_osm -a --slim --hstore -e 15:15 -o ${EXPIRY_FILE} ${CHANGE_FILE}
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102401*8192, allocation method=11
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /home/mapnik/otue/changes.osc.gz
Large polygon (22772 x 60366 metres, OSM ID 4741633) - only expiring perimeter
[snip]
Large polygon (20756 x 18765 metres, OSM ID -2234917) - only expiring perimeter
Processing: Node(1592k 2.1k/s) Way(188k 0.50k/s) Relation(3768 11.02/s) parse time: 1460s
Node stats: total(1592184), max(1789230743) in 742s
Way stats: total(188760), max(167536031) in 376s
Relation stats: total(3768), max(2234973) in 342s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending ways
Using 1 helper-processes
Large polygon (22772 x 60366 metres, OSM ID 4741633) - only expiring perimeter
[snip]
Large polygon (24821 x 26848 metres, OSM ID 167531912) - only expiring perimeter
processing way (123k) at 0.43k/s
Process 0 finished processing 123213 ways in 289 sec
All child processes exited
123213 Pending ways took 289s at a rate of 426.34/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending relations
Using 1 helper-processes
Large polygon (28347 x 29319 metres, OSM ID -5077) - only expiring perimeter
[snip]
Large polygon (16876 x 22611 metres, OSM ID -2234254) - only expiring perimeter
processing relation (8331) at 7.41/s
Process 0 finished processing 8339 relations in 1124 sec
All child processes exited
8339 Pending relations took 1124s at a rate of 7.42/s
node cache: stored: 1413978(100.00%), storage efficiency: 87.98% (dense blocks: 1095, sparse nodes: 323872), hit rate: 9.53%
Completed planet_osm_roads
Completed planet_osm_point
Completed planet_osm_line
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Completed planet_osm_polygon
Stopped table: planet_osm_nodes in 0s
Stopped table: planet_osm_ways in 0s
Stopped table: planet_osm_rels in 0s
Writing dirty tile list (1547K)
Osm2pgsql took 2874s overall
osm2pgsql SVN version 0.80.0 (32bit id space)
Hourly diff
Imported in less than 2 minutes
mapnik@iemt003:~$ osmosis --read-replication-interval workingDirectory=${WORKOSM_DIR} --simplify-change --write-xml-change ${CHANGE_FILE}
Updating tile server at samedi 23 juin 2012, 15:10:01 (UTC+0200)
23 juin 2012 15:10:01 org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.40.1
23 juin 2012 15:10:01 org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
23 juin 2012 15:10:01 org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
23 juin 2012 15:10:01 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
23 juin 2012 15:10:07 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline complete.
23 juin 2012 15:10:07 org.openstreetmap.osmosis.core.Osmosis run
INFO: Total execution time: 6054 milliseconds.
mapnik@iemt003:~$ osm2pgsql -d mapnik --tablespace-main-data data --tablespace-main-index index --tablespace-slim-data data --tablespace-slim-index index -p planet_osm -a --slim --hstore -e 15:15 -o ${EXPIRY_FILE} ${CHANGE_FILE}
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102401*8192, allocation method=11
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /home/mapnik/otue/changes.osc.gz
Large polygon (43824 x 29766 metres, OSM ID -14166) - only expiring perimeter
[snip]
Large polygon (14720 x 22466 metres, OSM ID -2246894) - only expiring perimeter
Processing: Node(49k 2.1k/s) Way(8k 0.61k/s) Relation(103 8.58/s) parse time: 50s
Node stats: total(49354), max(1799079776) in 24s
Way stats: total(8571), max(168678994) in 14s
Relation stats: total(103), max(2246901) in 12s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending ways
Using 1 helper-processes
Large polygon (21192 x 11788 metres, OSM ID 11796929) - only expiring perimeter
Large polygon (22204 x 14235 metres, OSM ID 33079083) - only expiring perimeter
processing way (6k) at 0.60k/s
Process 0 finished processing 6353 ways in 11 sec
All child processes exited
6353 Pending ways took 11s at a rate of 577.55/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending relations
Using 1 helper-processes
Large polygon (74678 x 82629 metres, OSM ID -7401) - only expiring perimeter
[snip]
Large polygon (21192 x 11788 metres, OSM ID 11796929) - only expiring perimeter
processing relation (281) at 8.52/s
Process 0 finished processing 281 relations in 33 sec
All child processes exited
281 Pending relations took 33s at a rate of 8.52/s
Completed planet_osm_point
node cache: stored: 43935(100.00%), storage efficiency: 73.02% (dense blocks: 30, sparse nodes: 19631), hit rate: 6.99%
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Stopped table: planet_osm_ways in 0s
Stopping table: planet_osm_rels
Stopped table: planet_osm_rels in 0s
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Writing dirty tile list (118K)
Osm2pgsql took 94s overall
osm2pgsql SVN version 0.80.0 (32bit id space)
5 minutes diff
Imported in less than 15 seconds (13,5 in average)
Tiles generation
All tiles for planet from zoom 0 to zoom 9 with default mapnik style.
mapnik@iemt003:~$ time render_list -a -s /var/run/renderd/renderd.sock -n 6 -z 0 -Z 9
Requesting xml(default) ......
[snip]
Requesting xml(default) x(504) y(504) z(9)
Total for all tiles rendered
Meta tiles rendered: Rendered 5464 tiles in 2845.56 seconds (1.92 tiles/s)
Total tiles rendered: Rendered 349696 tiles in 2845.56 seconds (122.89 tiles/s)
Total tiles handled: Rendered 5464 tiles in 2845.56 seconds (1.92 tiles/s)
real 47m25.571s
user 0m0.304s
sys 0m0.560s
mapnik@iemt003:~$