User:SomeoneElse/Ubuntu 1404 tileserver updating

From OpenStreetMap Wiki
Jump to: navigation, search

(note that storing this text in the wiki destroys the formatting - to see it as it was written, edit it, select the text, and paste into a proper text editor)

(very rough notes)

Also there's some stuff in the packaged instructions.

That says: sudo apt-get install osmosis

Is that the same osmosis that I have, or a different one (i.e. is there an osmosis in apmon's PPA that contains osm2pgsql etc.?)

It then suggests:

sudo /usr/bin/ gis www-data

I do actually have that:

myuseraccount@myservername:~$ locate /home/myuseraccount/src/osm2pgsql/

And it then suggests:

sudo -u www-data /usr/bin/openstreetmap-tiles-update-expire 2012-04-21

which I have too:

/home/myuseraccount/src/osm2pgsql/ myuseraccount@myservername:~$ locate openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/docs/openstreetmap-tiles-update-expire.1

but before running anything read through what I did last time that I did this (in 2012)

myuseraccount@myservername:~$ type osmosis osmosis is /usr/local/bin/osmosis (good)

myuseraccount@myservername:~$ type java java is /usr/bin/java

myuseraccount@myservername:~$ emacs src/mod_tile/openstreetmap-tiles-update-expire myuseraccount@myservername:~$ type osm2pgsql osm2pgsql is /usr/local/bin/osm2pgsql

myuseraccount@myservername:~$ type renderd_expired -bash: type: renderd_expired: not found so I may get that problem again

12/05/2014 10:12: I didn't notice any problem.

Although that doesn't seem to exist:

myuseraccount@myservername:~$ locate rendered_expired (nothing)

11/05/2014 19:05


I've imported some recent data (last night's Geofabrik file) Now set up updates

touch /var/lib/mod_tile/planet-import-complete

The date of the file is 658807300 May 11 05:33 great-britain_20140510.pbf

sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10

I needed to: sudo mkdir /var/log/tiles/

myuseraccount@myservername:~/data$ sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10 /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 34: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: cannot create /var/log/tiles//run.log: Directory nonexistent

myuseraccount@myservername:~/data$ sudo mkdir /var/log/tiles/ myuseraccount@myservername:~/data$ sudo chown myuseraccount /var/log/tiles

myuseraccount@myservername:~/data$ sudo -u myuseraccount /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10 --2014-05-11 21:07:03-- Resolving ( Connecting to (||:80... connected. HTTP request sent, awaiting response... 200 OK Length: 330 [text/plain] Saving to: â/var/lib/mod_tile/.osmosis/state.txtâ

100%[======================================>] 330 --.-K/s in 0.001s

2014-05-11 21:07:03 (297 KB/s) - â/var/lib/mod_tile/.osmosis/state.txtâ saved [330/330]

sudo emacs /var/lib/mod_tile/.osmosis/configuration.txt

In /var/lib/mod_tile/.osmosis/configuration.txt I currently have baseUrl= (not found)

I'm told I need to change it to (which does exist)

edit the script to set path up, cd to correct dir.

I'm logged in as myuseraccount, so:

myuseraccount@myservername:~/data$ /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 1: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: ./replag: not found

myuseraccount@myservername:~$ locate replag /home/myuseraccount/src/mod_tile/replag

so I need to edit the script to fix that by cding to the correct directory.

state.txt is now:

  1. Sun May 11 21:12:09 BST 2014

sequenceNumber=864897 timestamp=2014-05-10T00\:59\:02Z

replag still fails (not executable) So fix that. (OK)

Took 86 seconds.

Look for my next edit is in already is 20140410 00:29 That's in too. Was it before? Not sure...

The current state.txt is 5:58 on 10th is in already, which is AFTER my state.txt (so I need to keep going for a bit)

Next, how will I change it to get more than an hour? maxInterval = 3600 in configuration.txt

Try 7200

I can't see yet That's 23:15 on 10th May. (13 hours to go yet)

11/05/2014 22:13


Also look at rnr54847.txt

qqqmap some notes on cron here too

Next try and run from cron /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire Wait until 22:20

myuseraccount@myservername:~/data$ cat ~/t.txt rm: cannot remove â/var/lib/mod_tile/changes.osc.gzâ: No such file or directory rm: cannot remove â/var/lib/mod_tile/dirty_tiles.23381â: No such file or directory

[2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s)\

and 11 hour(s)

[2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state

myuseraccount@myservername:~/data$ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

So stick (most of) that in the script

java's now running.

We're now at: [2014-05-11 21:50:06] 20496 expiring tiles [2014-05-11 21:50:07] 20496 Done with import [2014-05-11 21:50:25] 21328 start import from seq-nr 865316, replag is 1 day(s) and 12 hour(s) [2014-05-11 21:50:25] 21328 downloading diff [2014-05-11 21:51:08] 21328 importing diff [2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state [2014-05-11 22:27:01] 24231 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:27:01] 24231 downloading diff [2014-05-11 22:27:37] 24231 importing diff

It takes about 10 minutes per hour of import, so schedule 1 hour every 10 minutes. or every 5 minutes, as if it's already running, it doesn't run twice

12/05/2014 00:30


Currently "replag is 1 day(s) and 6 hour(s)".

12/05/2014 8:56


It's nearly caught up (5 hours behind)

12/05/2014 10:27


It had caught up within an hour

But the osm2pgsql used by osmosis does not use my lua style

12/05/2014 18:15


So where in the script does it invoke osmspgsql? Comment out in crontab

I'm running it from my src area

In the script it does:

-o "$EXPIRY_FILE.$$" $CHANGE_FILE 1>&2 2> "$PGSQLLOG"; then
       m_error "osm2pgsql error"

So bung it in to: OSM2PGSQL_OPTIONS= I guess?

My import line was:

osm2pgsql --create --slim -C 1500 --number-processes 1 -S ../openstreetmap-carto/ --multi-geometry --tag-transform-script /home/myuseraccount/data/style.lua --hstore-all cut.pbf

Note I haven't been referencing hstore in updates, which may be an error.

14/05/2014 11:17: Not really. It would have been an error if I had been using hstore for anything.

For now just try: OSM2PGSQL_OPTIONS="--tag-transform-script /home/myuseraccount/data/style.lua"

And run: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire

(no errors so far - the script says that it is importing).

Meanwhile let's edit something so that a PF east of Tibshelf changes: created_at="2014-05-12T17:29:49Z" closed_at="2014-05-12T17:29:53Z"

That manual run seemed to behave OK

/var/log/tiles/osm2pgsql.log contains Using lua based tag processing pipeline with script /home/myuseraccount/data/style.lua

Reenable crontab. The next one runs. state.txt is now 17:37:02