User:SomeoneElse/Ubuntu 1404 tileserver updating
(note that storing this text in the wiki destroys the formatting - to see it as it was written, edit it, select the text, and paste into a proper text editor)
(very rough notes)
Also there's some stuff in the packaged instructions. http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/
That says: sudo apt-get install osmosis
Is that the same osmosis that I have, or a different one (i.e. is there an osmosis in apmon's PPA that contains osm2pgsql etc.?)
It then suggests:
sudo /usr/bin/install-postgis-osm-user.sh gis www-data
I do actually have that:
myuseraccount@myservername:~$ locate install-postgis-osm-user.sh /home/myuseraccount/src/osm2pgsql/install-postgis-osm-user.sh
And it then suggests:
sudo -u www-data /usr/bin/openstreetmap-tiles-update-expire 2012-04-21
which I have too:
/home/myuseraccount/src/osm2pgsql/install-postgis-osm-user.sh myuseraccount@myservername:~$ locate openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/docs/openstreetmap-tiles-update-expire.1
but before running anything read through what I did last time that I did this (in 2012)
myuseraccount@myservername:~$ type osmosis osmosis is /usr/local/bin/osmosis (good)
myuseraccount@myservername:~$ type java java is /usr/bin/java
myuseraccount@myservername:~$ emacs src/mod_tile/openstreetmap-tiles-update-expire myuseraccount@myservername:~$ type osm2pgsql osm2pgsql is /usr/local/bin/osm2pgsql
myuseraccount@myservername:~$ type renderd_expired -bash: type: renderd_expired: not found so I may get that problem again
12/05/2014 10:12: I didn't notice any problem.
Although that doesn't seem to exist:
myuseraccount@myservername:~$ locate rendered_expired (nothing)
11/05/2014 19:05
====
I've imported some recent data (last night's Geofabrik file) Now set up updates
touch /var/lib/mod_tile/planet-import-complete
The date of the file is 658807300 May 11 05:33 great-britain_20140510.pbf
sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10
I needed to: sudo mkdir /var/log/tiles/
myuseraccount@myservername:~/data$ sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10 /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 34: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: cannot create /var/log/tiles//run.log: Directory nonexistent
myuseraccount@myservername:~/data$ sudo mkdir /var/log/tiles/ myuseraccount@myservername:~/data$ sudo chown myuseraccount /var/log/tiles
myuseraccount@myservername:~/data$ sudo -u myuseraccount /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10
--2014-05-11 21:07:03-- http://osm.personalwerk.de/replicate-sequences/?2014-05-10T00:00:00Z
Resolving osm.personalwerk.de (osm.personalwerk.de)... 88.198.11.23
Connecting to osm.personalwerk.de (osm.personalwerk.de)|88.198.11.23|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 330 [text/plain]
Saving to: â/var/lib/mod_tile/.osmosis/state.txtâ
100%[======================================>] 330 --.-K/s in 0.001s
2014-05-11 21:07:03 (297 KB/s) - â/var/lib/mod_tile/.osmosis/state.txtâ saved [330/330]
sudo emacs /var/lib/mod_tile/.osmosis/configuration.txt
In /var/lib/mod_tile/.osmosis/configuration.txt I currently have baseUrl=https://planet.openstreetmap.org/minute-replicate (not found)
I'm told I need to change it to https://planet.openstreetmap.org/replication/minute/ (which does exist)
edit the script to set path up, cd to correct dir.
I'm logged in as myuseraccount, so:
myuseraccount@myservername:~/data$ /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 1: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: ./replag: not found
myuseraccount@myservername:~$ locate replag
/home/myuseraccount/src/mod_tile/replag
so I need to edit the script to fix that by cding to the correct directory.
state.txt is now:
- Sun May 11 21:12:09 BST 2014
sequenceNumber=864897 timestamp=2014-05-10T00\:59\:02Z
replag still fails (not executable) So fix that. (OK)
Took 86 seconds.
Look for my next edit http://www.openstreetmap.org/changeset/22227798 http://www.openstreetmap.org/changeset/22235257 http://www.openstreetmap.org/changeset/22235915 http://www.openstreetmap.org/changeset/22241982 is in already
http://www.openstreetmap.org/changeset/22242271 is 20140410 00:29 That's in too. Was it before? Not sure...
The current state.txt is 5:58 on 10th
http://www.openstreetmap.org/changeset/22249325 is in already, which is AFTER my state.txt (so I need to keep going for a bit)
Next, how will I change it to get more than an hour?
maxInterval = 3600
in configuration.txt
Try 7200
I can't see http://www.openstreetmap.org/changeset/22260728 yet That's 23:15 on 10th May. (13 hours to go yet)
11/05/2014 22:13
====
Also look at rnr54847.txt
qqqmap some notes on cron here too
Next try and run from cron /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire Wait until 22:20
myuseraccount@myservername:~/data$ cat ~/t.txt rm: cannot remove â/var/lib/mod_tile/changes.osc.gzâ: No such file or directory rm: cannot remove â/var/lib/mod_tile/dirty_tiles.23381â: No such file or directory
[2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s)\
and 11 hour(s)
[2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state
myuseraccount@myservername:~/data$ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
So stick (most of) that in the script
java's now running.
We're now at: [2014-05-11 21:50:06] 20496 expiring tiles [2014-05-11 21:50:07] 20496 Done with import [2014-05-11 21:50:25] 21328 start import from seq-nr 865316, replag is 1 day(s) and 12 hour(s) [2014-05-11 21:50:25] 21328 downloading diff [2014-05-11 21:51:08] 21328 importing diff [2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state [2014-05-11 22:27:01] 24231 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:27:01] 24231 downloading diff [2014-05-11 22:27:37] 24231 importing diff
It takes about 10 minutes per hour of import, so schedule 1 hour every 10 minutes.
or every 5 minutes, as if it's already running, it doesn't run twice
12/05/2014 00:30
====
Currently "replag is 1 day(s) and 6 hour(s)".
12/05/2014 8:56
=
It's nearly caught up (5 hours behind)
12/05/2014 10:27
====
It had caught up within an hour
But the osm2pgsql used by osmosis does not use my lua style
12/05/2014 18:15
==
So where in the script does it invoke osmspgsql? Comment out in crontab
I'm running it from my src area
In the script it does:
if ! $OSM2PGSQL_BIN -a --slim -e$EXPIRY_METAZOOM:$EXPIRY_METAZOOM $OSM2PGSQL_OPTIONS\ -o "$EXPIRY_FILE.$$" $CHANGE_FILE 1>&2 2> "$PGSQLLOG"; then m_error "osm2pgsql error" fi
So bung it in to: OSM2PGSQL_OPTIONS= I guess?
My import line was:
osm2pgsql --create --slim -C 1500 --number-processes 1 -S ../openstreetmap-carto/openstreetmap-carto.style --multi-geometry --tag-transform-script /home/myuseraccount/data/style.lua --hstore-all cut.pbf
Note I haven't been referencing hstore in updates, which may be an error.
14/05/2014 11:17: Not really. It would have been an error if I had been using hstore for anything.
For now just try:
OSM2PGSQL_OPTIONS="--tag-transform-script /home/myuseraccount/data/style.lua"
And run: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire
(no errors so far - the script says that it is importing).
Meanwhile let's edit something so that a PF east of Tibshelf changes:
http://www.openstreetmap.org/changeset/22296454 created_at="2014-05-12T17:29:49Z" closed_at="2014-05-12T17:29:53Z"
That manual run seemed to behave OK
/var/log/tiles/osm2pgsql.log contains Using lua based tag processing pipeline with script /home/myuseraccount/data/style.lua
Reenable crontab. The next one runs. state.txt is now 17:37:02