TIGER
From OpenStreetMap Wiki
 Part of WikiProject United States.
|
The Topologically Integrated Geographic Encoding and Referencing system (TIGER) data for US streets, produced by the US Census Bureau, is a public domain data source which has been almost entirely imported into OSM. It provides a great starting point for mapping streets of the United States.
|
|
Contents |
Tiger 2009
The Tiger 2009 data was released at the beginning of October. Go to the TIGER 2009 wiki page to coordinate coverage.
Status
It is unlikely that the TIGER data will ever be reimported. Enough editing has occurred since the original upload of the TIGER 2005 data (which was not uploaded until 2007) that it will be difficult to determine if differences between future TIGER and OSM are due to good corrections made by OSM editors or they are from bad TIGER data. With the US mapping community is growing strongly now, it isn't as much of a concern. Do not worry about getting your work overwritten by new TIGER data. Go map!
TIGER/Line import is done! (Well, mostly.) See http://www.linux.com/feature/125344. The 50 states have all been imported, and imports for other US territories (American Samoa, etc) are complete.
An initial run took place in 2005, but unfortunately had to be shut off, and data purged, in November 2006 due to data integrity problems. See Old TIGER Import 2005/2006. A second import followed a different philosophy, and began in Sept 2007 using a new Ruby TIGER Parser. Here is a post describing the newest import.
The TIGER fixup page lists things to look for when fixing TIGER data.
Editing Bad TIGER Data
The TIGER fixup page lists things to look for when fixing TIGER data. First, make sure it's bad. Either it's obviously bad, or you can see if it matches GPS tracks or USGS or Landsat photos. Be careful you don't "fix" someone else's corrections if you're working in some area you're unfamiliar with. If you've examined a way, and it matches your GPS tracks, or your knowledge of the area, remove the tiger:reviewed=no tag.
USGS photos and topo maps can be downloaded in JOSM with the WMS plugin via the following URLs, for Urban Areas, old B&W aerial photos (DOQ), or topos (DRG), respectively:
http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=urbanarea&Styles=&SRS=EPSG:4326&format=image/jpeg& http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=doq&Styles=&SRS=EPSG:4326&format=image/jpeg& http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=drg&Styles=&SRS=EPSG:4326&format=image/jpeg&
In Beej Jorgensen's experience, the USGS Urban Area photos match up quite well with GPS tracks and with the pre-corrected TIGER data. The DOQ photos are close. The topo maps do not line up as well, but are still useful when the road is obscured by trees. And there is evidence that the Landsat photos also are out of alignment.
The Terraserver-based USGS data is in the public domain[1][2].
Even if the TIGER data is the better corrected data, it still might not render correctly at freeway onramps and offramps, since there are a lot of shared nodes in the data and no layers. And there are still errors in it, albeit a much smaller number of them.
But assuming an area needs serious correction, here are a few tricks for improving it and things to watch for:
- Use the "Align Nodes In Line" tool in JOSM (ctl-shift-L). Go to the endpoints of the road that falls in a straight line, move them to the centers of their respective intersections, then select all the points along the road. (Oh, how do I do that easily?) They'll be put in line with the endpoints.
- Lots of TIGER ways don't have enough nodes to model the road. In JOSM, handling this became much easier since ticket 595 has been implemented, as this gives you "virtual vertices" that you can just grab and drag to get a new node.
- To disconnect a way from a node, put a new node in the middle of the way, select the new node, and split the way at the new node. Then delete the new way segment.
- There's often duplicate or mismatched data at country boundaries.
- For each TIGER import way, there's a "reviewed" tag. People often forget to update this. Use it, but don't trust it if it says "no".
And use the JOSM Validator plugin to verify the data and your work.
Merging Existing Linework with TIGER data
There is a chance that TIGER data will coincide with data created from other sources. This is a stub section to outline the process of merging them.
The TIGER data are specially tagged with a number of 'tiger:' thing, such as "tiger:county=Washington, OR". This makes it easy to distinguish TIGER data from normal users' data. You can use these tags in JOSM to search or to modify your selections. This makes it pretty easy to, for instance, take all of the non-TIGER roads in an area and delete them. User:DaveHansen has patches to JOSM to help manage these tags. For instance, if you have two nodes with "tiger:tlid=12:34:987" and "tiger:tlid=567" (tlids are the TIGER data's unique object identifiers) and you decide to merge those two nodes, it would be nice to retain those data. So, the patch will not prompt the user to manually merge the tags, it will just make the result node: "tiger:tlid=12:34:567:987".
User:DaveHansen generally uses the JOSM validator plugin. He's written additions to it to overcome some of the TIGER shortcomings, such as having motorways and normal roads share nodes. He uses the pretty stock "Crossing Ways" test in order to merge TIGER data. This is because virtually all existing roads in OSM will be overlapped at some point by the TIGER data. So, he follows through the list of crossing ways, and tries to find the original OSM way and the TIGER way. Since the TIGER ways are usually well connected to other TIGER data, and replacing it with the OSM way will mean a lot of work, we generally keep the TIGER way. But, the old way can be used like a GPS trace (and along with GPS traces) to get the TIGER way into a nice, accurate place.
If anybody can think of good algorithms to help in merging TIGER data, please post to the josm-dev@openstreetmap.org mailing list and give suggestions. Don't forget to put 'TIGER' in the subject.
One way to approach the problem of merging datasets is to pick the best/most complete version and delete the ways while retaining the nodes for reference. For example, say User1 has created a number of roads (but not nearly as complete as the TIGER data) and a number of footpaths and cycleways. The goal is to remove the roads (opting for tiger instead) but keep the foot/cycle paths. You could load the data into JOSM and do a search for "User1 -highway:footway -highway:cycleway" and hit delete. The nice part about this technique is that unwayed nodes stick around which allows you to quickly scan the dataset (just look for a bunch of lonely red dots in JOSM) then find and correct areas where User1's roads had deviated significantly from the TIGER data. The not-so-nice part about this technique is that you now have a ton of old nodes sitting around cluttering up your area.
New Ruby scripts - Completed Work
21 June 2007 - User:DaveHansen has provided a summary of the current status to the mailing list:
First of all, all of the actual TIGER parsing code here came from Brandon Martin-Anderson. All I did was spit out some OSM objects from the data he produced. He did all of the hard work.
This code takes a set of TIGER shapefiles, and turns them into an OSM .xml file. That file can then be opened in JOSM and reviewed before uploading. If anyone just wants to look at their county, I'll be happy to run this for them and I'll just send you the .osm file. (dave [AT] sr71 [DOT] net)
You can get the code here: http://sr71.net/~dave/osm/tiger/
As of version 0.7, the tiger-to-osm software generates OSM data in the 0.4 format; this data can be converted to API v0.5 using 04to05.pl.
You can get the tiger shapefiles see #Getting TIGER shapefiles
But, don't go uploading anything produced with this just yet. We need to make sure it's actually producing good, sane data (Note that our confidence in this is growing. See mailing list discussions) Feel free to go run it on a county that you know well, and closely examine the output. Report anything that looks strange or incorrect. Run it like this:
sh tiger-zip-to-osm.sh zips/IL/TGR17019.ZIP
If you use that shell script, it does proper locking, and you can run multiple instances of the converter on a machine without duplicating work (it's SMP safe, effectively).
I feel like it is pretty darn slow. I guess that's mostly ruby's fault, but I'd appreciate any tuning tips that people have for performance tuning ruby. You'll need quite a bit of RAM to run this for any large-size counties. At least a gig, probably two. I put a machine out of memory pretty badly that had 8GB in it because I ran several instances of this at once, and it didn't have and swap.
Fixme: We have a page TIGER to OSM Attribute Map. The script currently follows this? or is that page confusing things?
Debian/Ubuntu Prerequisites
If you're going to run Dave's importer under Debian or Ubuntu, you will need the following packages installed:
- procmail (provides 'lockfile', the locking program)
- ruby
- is it compatible with ruby1.9? if so it will run MUCH faster and be much beter on memory usage, ruby1.9 is WAY faster than ruby1.8 (current default)
- unzip
Issues with this code:
- memory consumption can be very large. It is worse on 64-bit machines, and can reach ~3GB for the largest counties in the country
- it is slower than it could be
- The shapefiles are very large and can overwhelm JOSM
- The renderer does not support "unseparated" ways
Who is working on it?
- Brandon Martin-Anderson - (badhill [AT] gmail [DOT] com) - Initial TIGER parsing code
- User:DaveHansen - Took a wee bit of ruby code from SVN to start what I have now, but I've replaced almost all of it.
- Who else?
Various people discussing. Join the mailing list discussions e.g recent mailing list discussion
Getting TIGER map source
Even more outdated: Tiger 2009 data is now avaliable in shape format, and has been cleaned up for much improved in accuracy. Available here.
Outdated: Tiger 2008 data is now avaliable in shape format, and has been cleaned up for much improved in accuracy. Available here.
The TIGER data is not currently available in "shapefile" format -- it's in its own "TIGER" format, so shapefile tools and libraries won't work. (However, the US Census department is phasing out the TIGER format. The new format is described here: http://www.census.gov/geo/www/tiger/tgrshp.html
You can get TIGER zipfiles from here: http://www2.census.gov/geo/tiger/tiger2006se/
To find your county, look here: http://www.census.gov/geo/www/fips/fips65/data/national.txt
Champaign County, IL has an entry like this in that file:
IL,17,019,Champaign,H1
You can find its zip file here: http://www2.census.gov/geo/tiger/tiger2006se/IL/TGR17019.ZIP
You could also do something like this if you wanted to get a bunch of counties with the name "Middle" in them"
wget -O - http://www.census.gov/geo/www/fips/fips65/data/national.txt\ | grep Middle | perl -pe 's#,#/TGR#' | perl -pe 's#,##' \ | perl -pe 's#,.*#.ZIP#' \ | awk '{print "http://www2.census.gov/geo/tiger/tiger2006se/" $1}' \ | xargs wget
Census 2000 Geographic Terms and Concepts:
United States
Region
Division
State
County
County subdivision
Place (or part)
http://proximityone.com/tgrcfcc.htm - TIGER/Line File Census Feature Class Codes (CFCC)
TIGER browser and renderings
You can browse the TIGER data through an online interface provided by the census bureau (public domain) at http://tiger.census.gov/cgi-bin/mapbrowse-tbl/
You can also request renderings by passing URL parameters e.g. http://tiger.census.gov/cgi-bin/mapgen/.gif?lat=40.739&lon=-73.99&wid=0.83&iht=300&iwd=300 (instructions)
The open source program RoadMap will render (and allow navigation) with the TIGER data. The data must be pre-compiled to a more compact form beforehand, but pre-compiled map data is available. (http://roadmap.sourceforge.net)
Place Boundaries
The TIGER 2007 data includes place boundaries; User:Lordsutch (as "Chris Lawrence" and "TIGERcnl") uploaded municipal and CDP boundaries to OpenStreetMap for U.S. states and territories.
The following states are incomplete:
- Florida has a few large CDPs that may be missing their relations.
- Illinois is 99.9% complete but some edits while the upload was in progress around O'Hare led to deleted ways that made up the boundaries of a few cities and CDPs (most notably, Chicago). I will attempt to repair these areas in the future.
- Texas' upload was interrupted by the update to API 0.6 in OSM. The way forward to fix this is, as of yet, undecided.
Also, several states were uploaded under API 0.5, which means that some boundaries may exceed current OSM editing limits: AL, AK, AR, AZ, CA, CO, CT, DE, MS.
The Python scripts used are available at http://www.lordsutch.com/osm/
Addressing Data
The TIGER 2007 data includes address relationship data. Various users are currently working on automated translation of this data into Karlsruhe Schema-style ways that will be associated with the imported TIGER data (and hopefully will figure out a way to cope with merging it with TIGER ways that have been edited by OSM users); the eventual goal here is to be able to geocode addresses within OSM. This will largely rely on the tiger:tlid keys in the existing OSM data.
Links
The main TIGER Website can be found here:
The TIGER data can be found here:
- http://www.census.gov/geo/cob/bdy
- TIGER data direct links, for spidering
The US Census Bureau Website is:
