User:EdLoach

From OpenStreetMap Wiki
Jump to: navigation, search
EdLoach
osm traces map
Country (admin_level_2) England


County Essex
Town (admin_level_8) Clacton
I use or have experience with a Locosys GPS device
I edit with both Potlatch and JOSM

I support the license
upgrade to ODbL v 1.0

ODbL-Supporter.png


Contents

About me

Public-images-osm logo.png

I support OSM by being a
member of the
OpenStreetMap Foundation

Are you?

Based in Clacton, I am using a Locosys Genie NaviGPS GT-31 and JOSM to try and add a lot more detail to the Tendring Peninsula (and Clacton in particular). My first edit (on 7th August 2008) was before getting a GPS device and involved correcting the spelling of the name of a road I knew well. The second was when I realised each section of the way needed the spelling correcting.

In my first year of mapping I helped map almost all of the roads in the Tendring district of Essex and made a start on walking all of the public footpaths, which may take many many months as they take that much longer than driving the same distance.

Having got a new phone which runs Windows Mobile towards the end of my first mapping year I downloaded OSMTracker and having mastered Custom Waypoints for adding custom text as a waypoint, have now started using that to note house numbers while out walking, although I still try and track with the GT-31 at the same time as the reception seems better. Also makes noting bus stops easy (though I often also add a custom text waypoint for any name or route references, or just may take a photo).

In addition to the Tendring area, I also try and map a bit more of Wolverhampton each time I visit relatives there.

I also mapped quite a bit of east Playa Blanca on foot and added detail to the town centre when on holiday there in July 2009. There is a bit of the town centre I realised I missed after processing all my gpx files on the return to the UK - zoom to the town centre and just west of the beach there are steps going up to nowhere and a couple of ways that are marked heading towards the beach which just end. There is another pedestrian way or two in the gap between them.

Mapping Parties

I have attended:

I organised:

Brewery Mapping

Not something I've done yet, but the question pops up occasionally on how to map them. So thoughts on how at User:EdLoach/Brewery.

Armchair mapping

Firstly, I have to say I am not a great fan of armchair mapping in many cases. There is no real substitute for a survey. So many details are still lacking when only say Bing, OS OpenData StreetView and OS OpenData Locator are available. There are also arguments for and against mapping an area currently bereft of mappers; will they be drawn more by a blank canvas, or will they perhaps be drawn by improving what is already there, perhaps initially via reporting issues in MapDust/Skobbler for example. Who knows? It is probably different for everyone. Having said all that I have done a bit of armchair mapping, usually of areas I have at least visited at some point in the past, and details are below.

County of Herefordshire, July 2011

After working on North Shropshire (below), Herefordshire was looking like it needed some work (and wasn't getting any as at 18/07/2011 - details from OSM Analysis): 

Rank                        404
Area                      County Of Herefordshire	
Roads in OS Locator       2,685
Road missing from OSM     1,497
Percentage Complete          43.76
Missing Change Last Day       0
Missing Change Last Week      0	
Missing Change Last 30 Days -60

The same ground rules were used as for North Shropshire, with the following added during the task.

It was noted that a number of ways looked to be phonetically similar to that in Locator, but without a survey it isn't clear whether the mapped spelling or the Locator spelling is correct, so no changes were made. It is possible a mapper was using voice annotation of their trace, so had no note of the actual spellings, but it is equally possible that any transcription error was at the OS.

Once the area had been covered that leaves us with the following to survey to "complete" the county:

There are a lot of ways in Herefordshire that have no names in OS Locator. Many were added while adding nearby missing named roads. Some do have names in StreetView funnily enough. A further pass of the county was made using MapDust which (amongst other items) located an entire unnamed business park which was still missing after all the OS Locator boxes on the ITO layer had been reviewed. This could lead to a category of:

OSM Analysis Summary after the above exercise: 

Rank                           132
Area                         County Of Herefordshire	
Roads in OS Locator          2,685
Road missing from OSM          141
Percentage Complete             94.26
Missing Change Last Day        -37
Missing Change Last Week      -384	
Missing Change Last 30 Days -1,395

Having now armchair mapped two areas it seems that named road coverage can generally be taken to about 90% using this method, and road coverage itself will be even higher. Clearing the existing MapDust reports (by checking, resolving and closing them) will hopefully over time reveal reports of where names are incorrect or missing (if they don't get surveyed by an OSMer sooner). MapDust reports I couldn't close include things like missing speed limits (a future major mapping task) and missing postcodes (when someone presumably was searching for a postcode as destination).

North Shropshire, July 2011

North Shropshire was languishing low down the tables on ITO's OSM Analysis, and as a regular visitor to the county in my youth I thought I'd see what I could do, and ended up adding over 500 road names from ITO's OSM Locator tiles. I set myself some ground rules:

This leaves about 100 ways which need a survey; to find the course of any way that isn't in Bing/StreetView, or ways that need checking as they are already named with something different from the OS Locator name. It also allows names added from OS Locator to be easily highlighted using the source:name tag (key) in much the same way as noname ways have been in the past, allowing surveys once there are keen mappers in the area. It also leaves North Shropshire with many missing ways now added, and a number of nearly joined ways correctly linked so available for routing. OSM Analysis name completion has gone up by over 50% (although survey's are still really required to confirm whether OS Locator names are accurate).

Tendring District and Bing Imagery, November 2010-

As I type this, the news has only just been announced. Below are what I think I might be able to use the imagery for best locally.

Buildings

If resolution is sufficiently good, improve building outlines over those available in OS Opendata StreetView

Landuse

This area currently has very poor Yahoo imagery coverage. It seems likely that Bing imagery should be good enough to determine farmland and possibly even distinguish field boundaries hedge vs fence vs ditch vs whatever. Field boundaries, combined with the existing 90%+ complete footpath coverage should bring the level of detail up to something ramblers can use (along with existing complete road data and other existing data).

Power Lines

Again, dependent on resolution available, it looks like power lines and pylons can be confirmed. Some of the minor lines not available in the OS opendata might also be visible (at least in combination with road/footpath surveys and extrapolation).

Tendring District and OS OpenData, April 2010-

Buildings

I have started tracing building outlines in the district, but try and restrict the tracing to areas where I hope to be able to go and survey the house numbers. Occasionally I find house numbers not visible from the road (due to parked vehicles, failing light and small numbers, or whatever) and use addr:housenumber=? to allow a later XAPI query to find where needs resurveying.

To date most of the surveying has been done in north Clacton, Little Clacton and Aingers Green, with some tracing in Bradfield and Brightlingsea for visits where I didn't get the hoped for mapping time.

Coastlines

I have converted the vector map district information for coastlines to WGS84 projection using a corrected .prj file and an import/export via Merkaartor. I have used this to enhance the coastline of the district. All coastlines for the Tendring district have now been updated in this way. In combination with the StreetView slippymap background the Mean High Water is being used as the coastline, and the Mean Low Water ways are used to help define tidal wetland multipolygons.

Power lines

These were extracted from the Settlement_Line VectorMapDistrict information. A pylon was assumed at each gap in the cables. No assumptions were made at the ends so these should be surveyed to check. The ones at the nearest substation to me go from the last pylons to the substation.

Water features

Lakes, reservoirs and the like were added from the information in the NaturalFeature_Line data. Where there was an existing stream at lower resolution, usually NPE traced, already in OSM I realigned that along the middle of the areas of water that were added.

Woodland

The NPE woods I added ages ago I removed and added the woods that are available in the NaturalFeature_Area data.

Administrative Boundaries

I added these for Tendring to parish level some time ago now, based on NPE parish boundaries and estimates of changes based on road signs. This was pretty close to the boundaries in the AdministrativeBoundary VectorMapDistrict stuff, but sufficiently wrong and lacking in detail for me to systematically replace all the boundaries with the more accurate information. The VectorMap stuff doesn't have a coast side boundary for parishes/counties but having discussed this with other mappers it is usually the case that the boundary follows the low water mark, so I have now used this rather than have a separate boundary way. Changes have been made from where the East of England boundary leaves Mersea to where the Suffolk border rounds the tip of land at the end of the Stour estuary. My biggest surprise was that Elmstead parish has a coastline; I thought Alresford shared a boundary with Wivenhoe.

Duplicate node clearing February/March 2010

After help clearing the UK of duplicate nodes thanks to Matt's excellent tool here:

http://matt.dev.openstreetmap.org/dupe_nodes/

I decided to just have a quick look at where my sister used to live in Effingham County, Georgia, USA. The whole state was red with duplicates. Very red. At first I was examining and merging them individually, but after some discussion on the #osm irc channel I got into a more systematic approach at clearing the main cuplrit in the state, the USGS-LULC land use import.

Adjacent landuse areas were using duplicate nodes on top of each other. If there was a hole in an area then a multipolygon existed with no tags on the inner way, but then two or three (usually) landuses inside that hole.

So the solution? JOSM with the validator plugin. At first I was using the remote control plugin to download areas, but more efficiently it was suggested by SK53 (who has some tips on cleaning up areas that have been subject to an NHD upload here User:SK53/NHD_Upload) that the JOSM tile downloader would be more efficient. So I worked out roughly that at z12 the range of tiles I want to cover to cover the state is 1074-1129 x and 1622-1687 y. I download an area of 4x3 tiles (e.g. 1074-1077x1676-1678) and use select to select the nodes that are likely to need validating. That query is made up as follows:

That latter one, finds ways with no tags that are in a relation, and then selects the nodes in those ways.

Both can be joined into a single query using | (in build 3037 OR seems to be faulty):

Another useful search in some areas is for orphaned nodes:

Or nodes with no tags that aren't in ways

A variant on that one were *old* orphaned nodes:

I think you can work that one out...

Update

Things slowed down once I started hitting areas of Georgia which also had been subject to NHD or LandPro08 imports; this meant downloading and checking smaller areas, as well as using a couple more queries. The four I'm running are: 

(type:node tags:0 -child)|(type:node tags:1 created_by=* -child)
child(type:way tags:0 child(type:relation))|child(type:way (attribution=NHD|source=NHD) -man_made=* -waterway=canal)
child(type:way tags:0 child(type:relation))|child(source=USGS-LULC)
child(type:way tags:0 child(type:relation))|child(source=LandPro08)

Haiti Boundaries 2010

I got a bit involved with the Haiti mapping in early 2010. Initially I was tracing detail from the public domain CIA map tiles (rivers/streams/peaks/place names sort of thing) and later got involved with trying to tidy up the administrative boundaries and assigning the correct pcode to each relation.

The latest image was created with my Kosmos boundary rules (link lower down this page), and the Geofabrik Haiti extract. Previous images had used a XAPI request to get all relations with boundary=administrative (or perhaps type=boundary, I forget now) as a tag.

Osmosis to keep a local copy of a .osm file updated

First get tendring.osm by downloading a Geofabrik extract in non-pbf format, and extracting it using 

osmosis --rx essex.osm.gz2 --bp clipIncompleteEntities=yes file=tendring.txt --wx tendring.osm

The tendring.txt bounding polygon is available from the Tendring(Essex District) page.

Once only execute 

osmosis --rrii

to prepare for synchronisation.

Edit configuration.txt if you want to download more than 3600 seconds worth of updates at a time, or want to use hourly diffs rather than minutely.

Download a state.txt file from planet.openstreetmap.org - pick a nnn.state.txt earlier than the Geofabrik extract timestamp (I chose 23:59 of previous day) and save in same folder as configuration.txt as state.txt

Then run 

osmosis --rri --simc --rx tendring.osm --ac --bp clipIncompleteEntities=yes file=tendring.txt --wx new.osm && copy new.osm tendring.osm /y && del new.osm

as often as it takes to catch up. The above is for Windows 7. Linux seems to have a mv command which will replace the copy and del commands. See the planet.osm/FAQ which is what I based the above on.

GB boundaries validation

I downloaded the Great Britain pbf download from Geofabrik and used osmosis to convert to .osm as I couldn't get the bzip2 support working in Boundaries.pl. 

osmosis --read-pbf file=great_britain.osm.pbf --write-xml file=gb.osm

I then ran gb.osm through Boundaries.pl using the command 

perl boundaries.pl -in=gb.osm -html=gb.htm -csv=gb.csv

This failed as gb.osm seemed to exclude ways shared with an adjacent extract, in particular some that form part of the Scilly Isles relation, and some along the border of Great Britain (extract, technically United Kingdom boundary) and Ireland. So I started again with the British Isles extract. 

osmosis --read-pbf file=british_isles.osm.pbf --write-xml file=british_isles.osm
perl boundaries.pl -in=british_isles.osm -html=boundaries.htm -csv=boundaries.csv

Note: I have explicitly specified filenames for html and csv in the line above, even though these are the default values. They are not those I actually used (test.htm and test.csv if you were really interested...).

The resulting files are available here (html) and here (csv). They should be updated each morning between 6am and 7am UK local time if I've left the computer on. Update: Routine now starts at 12:25 and takes just under an hour. While the .pbf is timestamped before 3am at Geofabrik, it doesn't upload to their ftp site until something after 9am (UK local time).

New version

I still run the tweaked boundaries.pl script, but have also written my own analysis tool which generated data in a format which made filtering easier for me to handle when displaying it here.

Correcting the boundaries

In many cases the boundary ways exist already but are either in the wrong relation or just missing. These boundaries are fairly easy to fix. In some cases not all the boundary ways exist yet, in which case the best source (as at time of writing) to add the missing sections from is OS OpenData BoundaryLine (also available in VectorMapDistrict).

The following notes are based on parish_region.shp from the May 2011 OS OpenData BoundaryLine release. They should apply for other types of boundaries too, but haven't yet been tested. Thanks to User:SK53 and User:chillly for their previously documented hints.

Getting a development copy of Potlatch 2 onto Windows 7 (64-bit)

Requirements

This should compile a potlatch2.swf file in resources subfolder. See the "Running" section in the readme.txt for options to launch. I ran potlatch2.html, right-clicked on the Potlatch 2 area (which stopped at 100%) and set global security settings to allow access to the resources folder. There are probably better ways, but this way looked easiest...

This looks at the dev server for testing. If you haven't already signed up on dev, when you first click Save you will have to first signup and login, then authorize the application before you can save anything.

Mapnik for a Linux novice

Host system for these tests

Notes

I've included the two links above so I don't mislay them. These tutorials suggest I should be able to follow the instructions and it will work. We'll see. I looked at Richard's Mapnik instructions first having already installed Ubuntu on a VMware image. I didn't know how to retro-fit LAMP and SSH (or what they meant) so Googled how to install them. This led me to the first of the two links which pointed me at a minimal install page, so I created a new VMware machine following those instructions using Lucid Lynx 10.04 64-bit image. I created it with 2 CPUs, 6GB RAM and 150GB HDD. Instead of importing all of planet (due to limited disk space) I am using the essex.osm.bz2 extract from Geofabrik (at the wget line in the Mapnik instructions, and the osm2pgsql line further down http://download.geofabrik.de/osm/europe/great_britain/england/essex.osm.bz2 ).

First attempts

Issue one

I got to ./autogen.sh and had errors. A lucky guess from reading these errors and the lines above and 

sudo apt-get install libtool

fixed the problem

Issue two

edit the conf file. Does Ubuntu not come with a decent edit command? DOS had one years ago. Anyway, after edit complained about permissions and sudo edit complained about the contents, I did a quick Google search. And ended up editing it with VI which did the job but is a piece of garbage. Update: have since discovered pico which seems much easier...

nano is quite good. -- Firefishy 16:09, 9 October 2011 (BST)

Issue 2a

When I repeated the install from scratch in October I had to change 

psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d gis

to 

psql -f /usr/share/postgresql/8.4/contrib/postgis.sql -d gis

and haven't checked why.

Issue three

processed_p.tar.bz2 is a big file. wget has failed twice now, though the third attempt (as I type) seems to be an hour from completion, which is the best it has been so far. Fingers crossed. Update: it downloaded but with an extra .2 on the end; trying to untar the archive without this .2 failed due to unexpected EOF. So presumably each failed wget command left a partial download on my drive which wasn't used to resume on next attempt.

Issue four

unzip wasn't installed, but trying to use it told me the command to install it.

And it works

My first image

I uploaded this by using ftp at the command line. 

ftp <url>

enter username and password. In my case it told me default upload was binary mode, which is what I wanted. I think there is a bin command needed otherwise. 

hash

(optional - shows some sort of progress) 

put image.png

(I was already in the correct folder, straight after the test, otherwise something like chdir ~/bin/mapnik/ is probably needed).

So, now what.

Hmmmm. Seems I need mod_tile and renderd or tirex. And I've not yet found a dummy's guide. More on this at a later date. Update: below here are my original notes on what I tried. Since I tried this, someone has very kindly created HowTo_mod_tile.

Tirex

See this page here regarding installing. Below are the commands I guessed at to do what is shown there. 

sudo apt-get install libipc-sharelite-perl
sudo apt-get install libjson-perl
sudo apt-get install libgd-gd2-perl
sudo apt-get install libwww-perl
cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/tirex tirex
cd tirex
make
make clean
sudo make install-all
sudo apt-get install libtest-simple-perl
sudo apt-get install libtest-harness-perl
prove

This finished with some text saying "All tests successful", some details, and "Result:PASS"

Mod_tile

As I write this the instructions for installation are a bit limited, saying only "build and install the mod_tile apache module from source". There is a bit lower down the page on getting the source code though, which I amended a bit inline with I did for mapnik, then repeated for tirex 

cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/mod_tile/ mod_tile
cd mod_tile
more readme.txt

Which helped me little. I don't speak German, but suspect this might help me, though it seems geared towards Debian rather than Ubuntu (I ignored the debuild commands in the tirex section above as I didn't have debuild and Google suggested it is something to do with Debian). 

sudo apt-get install install apache2 apache2-mpm-worker apache2-threaded-dev

This step might have done something nasty as I got a warning about phpmyadmin being installed. Not understanding the question I chose "No". 

wget -O /var/www/index.html http://svn.toolserver.org/svnroot/mazder/startup/openlayers/index.html

I think this gives me a page that will display stuff when everything works (it doesn't currently - I just get a zoom control on a blank page with a title of "Your own OpenStreetMap rendering stack").

So. More Googling. It seems I might be able to play with debuild after all. 

sudo apt-get build-dep linux-image-$(uname -r)

And wait for that to install. 

sudo apt-get install devscripts

Tirex revisited

So, with debuild available and devscripts installed I thought I'd try and execute the commands skipped earlier, just in case they were important. The apt-get lines below are ones I've added to install missing packages. I couldn't get libboost-program-options1.34.1 to install directly (reported as missing by make deb) so as I type I'm hoping that the 1.40 versions in libboost-dev work instead. I suspect it won't, but we'll see. 

cd ~/src/tirex
sudo apt-get install libboost-program-options-dev
sudo apt-get install libboost-dev
sudo apt-get install libboost-doc

Which didn't help. I still don't have libboost-program-options1.34.1 and can't find out how to install it. make deb fails without it. I don't really want to start tracking through the code to see if I can update the requirement either, so will ignore the make deb related steps for now.

So, some Googling later and the following got closer 

wget http://ucmirror.canterbury.ac.nz/ubuntu//pool/universe/b/boost/libboost-program-options1.34.1_1.34.1-16ubuntu1_amd64.deb
sudo dpkg -i libboost-program-options1.34.1_1.34.1-16ubuntu1_amd64.deb
make deb

but there was a no dependency information found error at the dpkg-shlibdeps step.

Switch2OSM

With the launch of Switch2OSM.org I thought I would try these instructions. I made a new VMWare minimal install of Ubuntu 10.04 with the mini.iso mentioned in notes above, this time with 1 processor 4GB RAM and 20GB Hard Drive. I hope that's enough for just Essex. I also followed these instructions to install the VMware tools, though had to replace every occurrence of /media/cdrom by just /media to get it to run.

Note 1

At the bit where it says 

...
CREATE FUNCTION
COMMIT
...
DROP FUNCTION

I just got 

...
CREATE FUNCTION
COMMIT

Note 2

In Installing osm2pgsql I needed to add a mkdir ~/src before cd ~/src

Note 3

In Installing osm2pgsql I got errors creating a directory at the make install step, permission denied, so have tried sudo make install which seems better.

Note 4

After python replies with >>>> press Ctrl-D to exit.

Note 5

./configure for mod_tile fails when checking for ICU >= 4.4

Thanks to apmon on #osm I edited configure.ac to check for 4.2 not 4.4, then reran autogen and configure steps again

Note 6

get-coastline.sh should be get-coastlines.sh

Kosmos Rule pages

Kosmos rules for Clacton Beer Festival leaflet map

Kosmos rules for Clacton Beer Festival leaflet map (mono)

Kosmos rules for Boundaries map

Maperitive Rule pages

Maperitive rules to highlight maxspeeds (mph) - colour scheme 1

Maperitive rules to highlight maxspeeds (mph) - colour scheme 2

Maperitive rules to show nodes/areas with housenumbers but no postcodes - red is node or area with addr:housenumber or addr:housename but no addr:postcode, and green is the same but with addr:postcode. Highways as thin black lines.

Using Maperitive Tiles

Sample HTML for displaying Maperitive generated tiles locally using Windows 7

JOSM Map Paint Style pages

Map paint style to colour addressed properties (number or name) drawn as areas in different colours depending whether they have a postcode or not

Essex towns template

{{User:EdLoach/Essex}} (uses {{User:EdLoach/EssexTowns}})

Ceremonial county of Essex
County town: Chelmsford
Unitary authoritiesArms of Essex.svg
Boroughs and districts
Towns
Basildon • Billericay • Braintree • Brentwood • Brightlingsea • Buckhurst Hill • Burnham-on-Crouch • Canvey Island • Chadwell-St-Mary • Chafford Hundred • Chelmsford • Clacton-on-Sea • Colchester • Corringham • Dovercourt • Epping • Frinton-on-Sea • Grays • Great Dunmow • Hadleigh • Halstead • Harlow • Harwich • Heybridge • Leigh-on-Sea • Loughton • Maldon • Manningtree • Ongar • Rayleigh • Rochford • Saffron Walden • Shoeburyness • South Benfleet • South Ockendon • South Woodham Ferrers • Southend-on-Sea • Stanford-le-Hope • Stansted Mountfitchet • Thaxted • Thundersley • Tilbury • Waltham Abbey • Walton-on-the-Naze • West Mersea • Wickford • Witham • Wivenhoe

Wiki development/tests

First stab at Unit Types for Map Features

Test page 1

Test Page 2

Esperanto Lëtzebuergesch

Test of tabbed map features concept

2012

Other

EdLoach commits code to OpenStreetMap under the name EdLoach.


Using JOSM for QA

I wanted a way to find houses that I've added a house number to but haven't added to an associatedStreet relation to identify the related streetname. I was hoping to create a JOSM mapCSS mappaint style, but couldn't work out whether it was possible. Nor did it seem to be in Maperitive. However, sometimes the simplest solutions are the best, and a quick use of the JOSM search facility proffered the following method. 

"addr:housenumber" OR "addr:housename"

child(type=associatedStreet)

Contributions

http://hdyc.neis-one.org/?EdLoach

Using QGIS and OS OpenData to determine postcodes in the Tendring District of Essex

Click for fullsize image

In 2010 the Tendring and Colchester CAMRA branches decided to revise their shared border such that the Tendring CAMRA branch area became the same as that of the Tendring District of Essex. As part of this CAMRA HQ needed to know which postcodes would be affected. I was asked to assist in determining this and thought it easiest to try and use OS OpenData to determine which postcodes were within Tendring, and in the case of areas which straddle the boundaries which weren't. I am aware (and passed on) that midpoints of postcodes that fall near the boundary may apply to a number of properties, some of which are in Tendring district and some are in Colchester borough, but for those pubs nearest the boundary the process below had them all in the correct district.

The steps below may not be optimal, but they worked for me when I didn't really know how to use QGIS and just used likely looking menu options as I went along. 

Postcode,A,B,C,D,E,F,G,H,I,X,Y,J,K,L,M,N,O,P

If desired (and I wanted to be able to email the postcodes in Tendring to someone who does have Excel but probably wouldn't have much use for a shapefile)

User Templates

Baseball.svg This user joined in with
Big baseball project 2011!
Retrieved from "http://wiki.openstreetmap.org/wiki/User:EdLoach"
Personal tools
Namespaces
Variants
Actions
site
Toolbox