User:SomeoneElse/Ubuntu 1404 tileserver updating

From OpenStreetMap Wiki
Jump to navigation Jump to search

(note that storing this text in the wiki destroys the formatting - to see it as it was written, edit it, select the text, and paste into a proper text editor)

(very rough notes)

Also there's some stuff in the packaged instructions. http://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/

That says: sudo apt-get install osmosis

Is that the same osmosis that I have, or a different one (i.e. is there an osmosis in apmon's PPA that contains osm2pgsql etc.?)

It then suggests:

sudo /usr/bin/install-postgis-osm-user.sh gis www-data

I do actually have that:

myuseraccount@myservername:~$ locate install-postgis-osm-user.sh /home/myuseraccount/src/osm2pgsql/install-postgis-osm-user.sh


And it then suggests:

sudo -u www-data /usr/bin/openstreetmap-tiles-update-expire 2012-04-21

which I have too:

/home/myuseraccount/src/osm2pgsql/install-postgis-osm-user.sh myuseraccount@myservername:~$ locate openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/docs/openstreetmap-tiles-update-expire.1


but before running anything read through what I did last time that I did this (in 2012)

myuseraccount@myservername:~$ type osmosis osmosis is /usr/local/bin/osmosis (good)

myuseraccount@myservername:~$ type java java is /usr/bin/java

myuseraccount@myservername:~$ emacs src/mod_tile/openstreetmap-tiles-update-expire myuseraccount@myservername:~$ type osm2pgsql osm2pgsql is /usr/local/bin/osm2pgsql

myuseraccount@myservername:~$ type renderd_expired -bash: type: renderd_expired: not found so I may get that problem again

12/05/2014 10:12: I didn't notice any problem.


Although that doesn't seem to exist:

myuseraccount@myservername:~$ locate rendered_expired (nothing)

11/05/2014 19:05

====

I've imported some recent data (last night's Geofabrik file) Now set up updates

touch /var/lib/mod_tile/planet-import-complete

The date of the file is 658807300 May 11 05:33 great-britain_20140510.pbf

sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10

I needed to: sudo mkdir /var/log/tiles/

myuseraccount@myservername:~/data$ sudo -u www-data /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10 /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 34: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: cannot create /var/log/tiles//run.log: Directory nonexistent

myuseraccount@myservername:~/data$ sudo mkdir /var/log/tiles/ myuseraccount@myservername:~/data$ sudo chown myuseraccount /var/log/tiles


myuseraccount@myservername:~/data$ sudo -u myuseraccount /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire 2014-05-10 --2014-05-11 21:07:03-- http://osm.personalwerk.de/replicate-sequences/?2014-05-10T00:00:00Z Resolving osm.personalwerk.de (osm.personalwerk.de)... 88.198.11.23 Connecting to osm.personalwerk.de (osm.personalwerk.de)|88.198.11.23|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 330 [text/plain] Saving to: â/var/lib/mod_tile/.osmosis/state.txtâ

100%[======================================>] 330 --.-K/s in 0.001s

2014-05-11 21:07:03 (297 KB/s) - â/var/lib/mod_tile/.osmosis/state.txtâ saved [330/330]


sudo emacs /var/lib/mod_tile/.osmosis/configuration.txt

In /var/lib/mod_tile/.osmosis/configuration.txt I currently have baseUrl=https://planet.openstreetmap.org/minute-replicate (not found)

I'm told I need to change it to https://planet.openstreetmap.org/replication/minute/ (which does exist)


edit the script to set path up, cd to correct dir.

I'm logged in as myuseraccount, so:

myuseraccount@myservername:~/data$ /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: 1: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire: ./replag: not found


myuseraccount@myservername:~$ locate replag /home/myuseraccount/src/mod_tile/replag

so I need to edit the script to fix that by cding to the correct directory.

state.txt is now:

  1. Sun May 11 21:12:09 BST 2014

sequenceNumber=864897 timestamp=2014-05-10T00\:59\:02Z

replag still fails (not executable) So fix that. (OK)

Took 86 seconds.

Look for my next edit http://www.openstreetmap.org/changeset/22227798 http://www.openstreetmap.org/changeset/22235257 http://www.openstreetmap.org/changeset/22235915 http://www.openstreetmap.org/changeset/22241982 is in already

http://www.openstreetmap.org/changeset/22242271 is 20140410 00:29 That's in too. Was it before? Not sure...

The current state.txt is 5:58 on 10th

http://www.openstreetmap.org/changeset/22249325 is in already, which is AFTER my state.txt (so I need to keep going for a bit)


Next, how will I change it to get more than an hour? maxInterval = 3600 in configuration.txt

Try 7200

I can't see http://www.openstreetmap.org/changeset/22260728 yet That's 23:15 on 10th May. (13 hours to go yet)


11/05/2014 22:13

====

Also look at rnr54847.txt

qqqmap some notes on cron here too

Next try and run from cron /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire Wait until 22:20

myuseraccount@myservername:~/data$ cat ~/t.txt rm: cannot remove â/var/lib/mod_tile/changes.osc.gzâ: No such file or directory rm: cannot remove â/var/lib/mod_tile/dirty_tiles.23381â: No such file or directory

[2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s)\ 

and 11 hour(s)

[2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state

myuseraccount@myservername:~/data$ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

So stick (most of) that in the script

java's now running.

We're now at: [2014-05-11 21:50:06] 20496 expiring tiles [2014-05-11 21:50:07] 20496 Done with import [2014-05-11 21:50:25] 21328 start import from seq-nr 865316, replag is 1 day(s) and 12 hour(s) [2014-05-11 21:50:25] 21328 downloading diff [2014-05-11 21:51:08] 21328 importing diff [2014-05-11 22:09:18] 21328 expiring tiles [2014-05-11 22:09:19] 21328 Done with import [2014-05-11 22:20:01] 23381 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:20:01] 23381 downloading diff [2014-05-11 22:20:01] 23381 [error] Osmosis error [2014-05-11 22:20:01] 23381 resetting state [2014-05-11 22:27:01] 24231 start import from seq-nr 865436, replag is 1 day(s) and 11 hour(s) [2014-05-11 22:27:01] 24231 downloading diff [2014-05-11 22:27:37] 24231 importing diff


It takes about 10 minutes per hour of import, so schedule 1 hour every 10 minutes. or every 5 minutes, as if it's already running, it doesn't run twice


12/05/2014 00:30

====

Currently "replag is 1 day(s) and 6 hour(s)".

12/05/2014 8:56

=

It's nearly caught up (5 hours behind)

12/05/2014 10:27

====

It had caught up within an hour

But the osm2pgsql used by osmosis does not use my lua style


12/05/2014 18:15

==

So where in the script does it invoke osmspgsql? Comment out in crontab

I'm running it from my src area

In the script it does: 

   if ! $OSM2PGSQL_BIN -a --slim -e$EXPIRY_METAZOOM:$EXPIRY_METAZOOM $OSM2PGSQL_OPTIONS\
-o "$EXPIRY_FILE.$$" $CHANGE_FILE 1>&2 2> "$PGSQLLOG"; then
       m_error "osm2pgsql error"
   fi

So bung it in to: OSM2PGSQL_OPTIONS= I guess?

My import line was:

osm2pgsql --create --slim -C 1500 --number-processes 1 -S ../openstreetmap-carto/openstreetmap-carto.style --multi-geometry --tag-transform-script /home/myuseraccount/data/style.lua --hstore-all cut.pbf

Note I haven't been referencing hstore in updates, which may be an error.

14/05/2014 11:17: Not really. It would have been an error if I had been using hstore for anything.


For now just try: OSM2PGSQL_OPTIONS="--tag-transform-script /home/myuseraccount/data/style.lua"

And run: /home/myuseraccount/src/mod_tile/openstreetmap-tiles-update-expire

(no errors so far - the script says that it is importing).


Meanwhile let's edit something so that a PF east of Tibshelf changes:

http://www.openstreetmap.org/changeset/22296454 created_at="2014-05-12T17:29:49Z" closed_at="2014-05-12T17:29:53Z"

That manual run seemed to behave OK

/var/log/tiles/osm2pgsql.log contains Using lua based tag processing pipeline with script /home/myuseraccount/data/style.lua

Reenable crontab. The next one runs. state.txt is now 17:37:02

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:SomeoneElse/Ubuntu_1404_tileserver_updating&oldid=1550417"