TIGER

From OpenStreetMap

Jump to: navigation, search
U.S. progress animation during the TIGER import (frame images)
U.S. progress animation during the TIGER import (frame images)

The Topologically Integrated Geographic Encoding and Referencing system (TIGER) data for US streets, produced by the US Census Bureau, is a public domain data source which has been almost entirely imported into OSM). It provides a great starting point for mapping streets of the United States.

Contents

Status

Done! (Well, mostly.) See http://www.linux.com/feature/125344. The 50 states have all been imported, and imports for other US principalities (American Samoa, etc) are underway. You can check the status of the import script.

An initial run took place in 2005, but unfortunately had to be shut off, and data purged, in November 2006 due to data integrity problems. See Old TIGER Import 2005/2006. A second import followed a different philosophy, and began in Sept 2007 using a new Ruby TIGER Parser. Here is a post describing the newest import.

The TIGER fixup page lists things to look for when fixing TIGER data.

Editing Bad TIGER Data

The TIGER fixup page lists things to look for when fixing TIGER data. First, make sure it's bad. Either it's obviously bad, or you can see if it matches GPS tracks or USGS or Landsat photos. Be careful you don't "fix" someone else's corrections if you're working in some area you're unfamiliar with. If you've examined a way, and it matches your GPS tracks, or your knowledge of the area, remove the tiger:reviewed=no tag.

USGS photos and topo maps can be downloaded in JOSM with the WMS plugin via the following URLs, for Urban Areas, old B&W aerial photos (DOQ), or topos (DRG), respectively: 

http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=urbanarea&Styles=&SRS=EPSG:4326&format=image/jpeg&

http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=doq&Styles=&SRS=EPSG:4326&format=image/jpeg&

http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=drg&Styles=&SRS=EPSG:4326&format=image/jpeg&

In Beej Jorgensen's experience, the USGS Urban Area photos match up quite well with GPS tracks and with the pre-corrected TIGER data. The DOQ photos are close. The topo maps do not line up as well, but are still useful when the road is obscured by trees. And there is evidence that the Landsat photos also are out of alignment.

The Terraserver-based USGS data is in the public domain[1][2].

Even if the TIGER data is the better corrected data, it still might not render correctly at freeway onramps and offramps, since there are a lot of shared nodes in the data and no layers. And there are still errors in it, albeit a much smaller number of them.

But assuming an area needs serious correction, here are a few tricks for improving it and things to watch for:

  1. Use the "Align Nodes In Line" tool in JOSM (ctl-shift-L). Go to the endpoints of the road that falls in a straight line, move them to the centers of their respective intersections, then select all the points along the road. (Oh, how do I do that easily?) They'll be put in line with the endpoints.
  2. Lots of TIGER ways don't have enough nodes to model the road. In JOSM, I do a lot of "N" (new), "shift-click" on the way (make a new node in the way), "S" (select), then drag the node into position. And repeat. (If JOSM ticket 595 would be implemented, that would be much easier.)
  3. To disconnect a way from a node, put a new node in the middle of the way, select the new node, and split the way at the new node. Then delete the new way segment.
  4. There's often duplicate or mismatched data at country boundaries.
  5. For each TIGER import way, there's a "reviewed" tag. People often forget to update this. Use it, but don't trust it if it says "no".

And use the JOSM Validator plugin to verify the data and your work.

Merging Existing Linework with TIGER data

There is a chance that TIGER data will coincide with data created from other sources. This is a stub section to outline the process of merging them.

The TIGER data are specially tagged with a number of 'tiger:' thing, such as "tiger:county=Washington, OR". This makes it easy to distinguish TIGER data from normal users' data. You can use these tags in JOSM to search or to modify your selections. This makes it pretty easy to, for instance, take all of the non-TIGER roads in an area and delete them. User:DaveHansen has patches to JOSM to help manage these tags. For instance, if you have two nodes with "tiger:tlid=12:34:987" and "tiger:tlid=567" (tlids are the TIGER data's unique object identifiers) and you decide to merge those two nodes, it would be nice to retain those data. So, the patch will not prompt the user to manually merge the tags, it will just make the result node: "tiger:tlid=12:34:567:987".

User:DaveHansen generally uses the JOSM validator plugin. He's written additions to it to overcome some of the TIGER shortcomings, such as having motorways and normal roads share nodes. He uses the pretty stock "Crossing Ways" test in order to merge TIGER data. This is because virtually all existing roads in OSM will be overlapped at some point by the TIGER data. So, he follows through the list of crossing ways, and tries to find the original OSM way and the TIGER way. Since the TIGER ways are usually well connected to other TIGER data, and replacing it with the OSM way will mean a lot of work, we generally keep the TIGER way. But, the old way can be used like a GPS trace (and along with GPS traces) to get the TIGER way into a nice, accurate place.

If anybody can think of good algorithms to help in merging TIGER data, please post to the josm-dev@openstreetmap.org mailing list and give suggestions. Don't forget to put 'TIGER' in the subject.

One way to approach the problem of merging datasets is to pick the best/most complete version and delete the ways while retaining the nodes for reference. For example, say User1 has created a number of roads (but not nearly as complete as the TIGER data) and a number of footpaths and cycleways. The goal is to remove the roads (opting for tiger instead) but keep the foot/cycle paths. You could load the data into JOSM and do a search for "User1 -highway:footway -highway:cycleway" and hit delete. The nice part about this technique is that unwayed nodes stick around which allows you to quickly scan the dataset (just look for a bunch of lonely red dots in JOSM) then find and correct areas where User1's roads had deviated significantly from the TIGER data. The not-so-nice part about this technique is that you now have a ton of old nodes sitting around cluttering up your area.

New Ruby scripts - Completed Work

21 June 2007 - User:DaveHansen has provided a summary of the current status to the mailing list:

First of all, all of the actual TIGER parsing code here came from Brandon Martin-Anderson. All I did was spit out some OSM objects from the data he produced. He did all of the hard work.

This code takes a set of TIGER shapefiles, and turns them into an OSM .xml file. That file can then be opened in JOSM and reviewed before uploading. If anyone just wants to look at their county, I'll be happy to run this for them and I'll just send you the .osm file. (dave [AT] sr71 [DOT] net)

You can get the code here: http://sr71.net/~dave/osm/tiger/

As of version 0.7, the tiger-to-osm software generates OSM data in the 0.4 format; this data can be converted to 0.5 using 04to05.pl.

You can get the tiger shapefiles see #Getting TIGER shapefiles

But, don't go uploading anything produced with this just yet. We need to make sure it's actually producing good, sane data (Note that our confidence in this is growing. See mailing list discussions) Feel free to go run it on a county that you know well, and closely examine the output. Report anything that looks strange or incorrect. Run it like this: 

sh tiger-zip-to-osm.sh zips/IL/TGR17019.ZIP

If you use that shell script, it does proper locking, and you can run multiple instances of the converter on a machine without duplicating work (it's SMP safe, effectively).

I feel like it is pretty darn slow. I guess that's mostly ruby's fault, but I'd appreciate any tuning tips that people have for performance tuning ruby. You'll need quite a bit of RAM to run this for any large-size counties. At least a gig, probably two. I put a machine out of memory pretty badly that had 8GB in it because I ran several instances of this at once, and it didn't have and swap.


Fixme: We have a page TIGER to OSM Attribute Map. The script currently follows this? or is that page confusing things?


Debian/Ubuntu Prerequisites

If you're going to run Dave's importer under Debian or Ubuntu, you will need the following packages installed:

  • procmail (provides 'lockfile', the locking program)
  • ruby
  • unzip

Issues with this code:

  • memory consumption can be very large. It is worse on 64-bit machines, and can reach ~3GB for the largest counties in the country
  • it is slower than it could be
  • The shapefiles are very large and can overwhelm JOSM
  • The renderer does not support "unseparated" ways

Who is working on it?

  • Brandon Martin-Anderson - (badhill [AT] gmail [DOT] com) - Initial TIGER parsing code
  • User:DaveHansen - Took a wee bit of ruby code from SVN to start what I have now, but I've replaced almost all of it.
  • Who else?

Various people discussing. Join the mailing list discussions e.g recent mailing list discussion


Getting TIGER map source

The TIGER data is not currently available in "shapefile" format -- it's in its own "TIGER" format, so shapefile tools and libraries won't work. (However, the US Census department is phasing out the TIGER format. The new format is described here: http://www.census.gov/geo/www/tiger/tgrshp.html

You can get TIGER zipfiles from here: http://www2.census.gov/geo/tiger/tiger2006se/OR/

To find your county, look here: http://www.census.gov/geo/www/fips/fips65/data/national.txt

Champaign County, IL has an entry like this in that file:

IL,17,019,Champaign,H1

You can find its zip file here: http://www2.census.gov/geo/tiger/tiger2006se/IL/TGR17019.ZIP

You could also do something like this if you wanted to get a bunch of counties with the name "Middle" in them" 

wget -O - http://www.census.gov/geo/www/fips/fips65/data/national.txt\
      | grep Middle | perl -pe 's#,#/TGR#' | perl -pe 's#,##' \
      | perl -pe 's#,.*#.ZIP#' \
      | awk '{print "http://www2.census.gov/geo/tiger/tiger2006se/" $1}'      \
      | xargs wget


Census 2000 Geographic Terms and Concepts: 

United States
   Region
      Division
        State
          County
            County subdivision
               Place (or part)

http://proximityone.com/tgrcfcc.htm - TIGER/Line File Census Feature Class Codes (CFCC)

TIGER browser and renderings

You can browse the TIGER data through an online interface provided by the census bureau (public domain) at http://tiger.census.gov/cgi-bin/mapbrowse-tbl/

You can also request renderings by passing URL parameters e.g. http://tiger.census.gov/cgi-bin/mapgen/.gif?lat=40.739&lon=-73.99&wid=0.83&iht=300&iwd=300 (instructions)

The open source program RoadMap will render (and allow navigation) with the TIGER data. The data must be pre-compiled to a more compact form beforehand, but pre-compiled map data is available. (http://roadmap.sourceforge.net)

Links

The main TIGER Website can be found here:

The TIGER data can be found here:

The US Census Bureau Website is:

Retrieved from "http://wiki.openstreetmap.org/index.php/TIGER"
Personal tools
recent changes