User:Kms

I live in the West End of Glasgow and I am mapping north of the River Clyde from Crow Road/Balshagray Avenue east towards the M8 and City Centre. I am currently concentrating on Broomhill. See the progress on the map, you need to switch to Osmarender rather than Mapnik for the full effect!

Getting Started with OSM

This section will contain the notes I've made on getting started with GPS, gpsbabel, GPX, and using JOSM to create nodes, segments, and ways.

Building an OSM home server

It is possible to take the XML dump from Planet.osm and load this into a PostgreSQL database for further processing such as rendering tiles with Mapnik. I based my process on the notes by User:Mac and the Slippy Map. My home system is running Fedora Core 6 other distributions maybe different.

Getting and unpacking the XML

Download the file from one of the sites listed. The planet.osm file is regenerated once a week, on a Wednesday. The downloaded file is compressed using bzip2, to uncompress this:

/usr/bin/bunzip2  planet-070307.osm.bz2

You should now have a very large XML file called planet-070307.osm.

Building and running UTF8Sanitize

The source code for UTF8Sanitize is available from the OSM subversion repository. To check out the source code use the following command:

svn co http://svn.openstreetmap.org/utils/planet.osm/C/ UTF8Sanitize

and then to build the program:

cd UTF8Sanitize
make

You should now be able to process the planet.osm file you created above a command like:

cat /path/to/planet.osm | UTF8sanitize | /path/to/planet.san.osm

You should get output like:

Error at line 72505269
Error at line 72583739
Summary:
chars1: 3301140876
chars2: 139176
chars3: 1589
chars4: 0
chars5: 0
chars6: 1
lines : 73468080

If you get errors like I have, don't worry, the advice from the mailing list is that you can continue.

Creating an SQL file

The next stage is create an SQL file suitable for importing into PostgreSQL. This is done using the osm2pgsql command. You can get this command from subversion in the same way as the UTF8sanitize command:

svn co http://svn.openstreetmap.org/utils/osm2pgsql osm2pgsql

To build this command you need the geos library, install this using YUM:

# yum install geos geos-devel

Now you can build osm2pgsql:

cd osm2pgsql
make

To do the actual processing:

oms2pgsql /path/to/planet.san.osm > /path/to/planet.sql

You should get output like:

Processing: Node(8420k)
Processing: Segment(8830k)
Processing: Way(439k)
Node stats: out(92707), total(8423077), max(26390114)
Segment stats: out(8325691), total(8831307), max(22376860)
Way stats: out(471541), total(439146), max(4335934)
Way stats: duplicate segments in ways 123

Prepare the database

Before you load the data into the database you need to install PostGIS and get PostgreSQL running and initialised:

yum install postgis postgis-jdbc postgis-utils
service postgresql start
chkconfig postgresql on

To initialise the database you need to be user postgres, to switch to this user:

$ su -
Password: 
# su - postgres
$

You need to become root first because the postgres user probably has no password. You then need to create the database, a user, and set some privileges:

createdb -EUNICODE osm
createuser -S -D -R osm
echo "GRANT ALL ON SCHEMA PUBLIC TO osm;" | psql osm 
echo "create language plpgsql;" | psql osm
psql osm < /usr/share/pgsql/contrib/lwpostgis.sql
echo "grant all on geometry_columns to osm;" | psql osm
echo "grant all on spatial_ref_sys to osm;" | psql osm

You should only need to carry put these steps the very first time you setup the system.

Loading into the database

To load the data into the database you should be user postgres, to switch to this user:

$ su -
Password: 
# su - postgres
$

You need to become root first because the postgres user probably has no password. To load the data:

cat /path/to/planet.sql | psql osm

You'll get an endlessly scrolling display of INSERT 0 1 statments then finally:

INSERT 0 1
COMMIT
VACUUM
CREATE INDEX
ALTER TABLE
CLUSTER
VACUUM
WARNING:  there is no transaction in progress
COMMIT

And that's it! You now have the data in PostGIS. The next step is to build a system for rendering the data to create an actual map. You may also want to repeat these steps each Wednesday or thursday when the new planet.osm is created.

How long does it take?

How long does it take to do the processing necessary to build a OSM server at home? These are timings from my home PC, a Hyperthreaded 3GHz Intel machine with 1Gb RAM and a single SATA hard disk.

Running UTF8Sanitize

$ time cat planet-070307.osm | ./svn/utils/planet.osm/C/UTF8sanitize > planet.osm
Error at line 72505269
Error at line 72583739
Summary:
chars1: 3301140876
chars2: 139176
chars3: 1589
chars4: 0
chars5: 0
chars6: 1
lines : 73468080

real    5m23.187s
user    3m42.466s
sys     0m37.270s

Running osm2pgsql

$ time ./osm2pgsql /export/osm/planet.osm > /export/osm/planet.sql
Processing: Node(8420k)
Processing: Segment(8830k)
Processing: Way(439k)
Node stats: out(92707), total(8423077), max(26390114)
Segment stats: out(8325691), total(8831307), max(22376860)
Way stats: out(471541), total(439146), max(4335934)
Way stats: duplicate segments in ways 123

real    17m32.223s
user    16m4.528s
sys     0m19.521s

Populating the database

$ time cat planet.sql | psql osm

<< snip all the INSERT 0 1 statements >>

INSERT 0 1
COMMIT 
VACUUM
CREATE INDEX
ALTER TABLE
CLUSTER
VACUUM
WARNING:  there is no transaction in progress
COMMIT

real    7m31.268s
user    0m32.942s
sys     0m31.310s