Florida Land Cover Import

From OpenStreetMap Wiki
Jump to: navigation, search


About

This page documents the process for importing water features extracted from land cover data that was mapped by Florida's five Water Management Districts in 2008-2010.

Goals

The primary goal of this effort is to add missing water areas and replace parts of unedited or poorly edited coastline with this data.

Lake names were discussed as another input to the import, but were dropped in order to move the project forward. Another person is interested in conflating lake names for other parts of the country, so Florida can be rolled into that project.

Schedule

  • Planning: June / July 2013
  • Import: Starting September 2013, ending: October 2013?
  • QA: post-import

Import Data

Background

Data source site: http://www.swfwmd.state.fl.us/data/gis/layer_library/category/physical_dense
Data license: Public Domain - All data produced by state and local governments in Florida is public domain according to Chapter 119, Florida Statutes, Florida’s Public Records Act
Permissions: Our data are freely available under Florida’s public records law and we don’t place any restrictions on the use. Hopefully this statement will work:

“These records are being produced under Chapter 119, Florida Statutes, Florida’s Public Records Act. Section 119.91(1), F.S. provides: ‘It is the policy of this state that all state, county, and municipal records are open for personal inspection and copying by any person. Providing access to public records is a duty of each agency.’”

I would just ask that you include the original source of the data, either a copy of our metadata or at least a link to our download page, in your metadata. Please feel free to contact me if you have any other questions.

Data Files

  • Northwest Florida Water Management District (NWFWMD)
  • South Florida Water Management District (SFWMD)
  • Southwest Florida Water Management District (SWFMWD) - One large shapefile in Florida StatePlane West HARN
  • St Johns River Water Management District (SJRWMD)
  • Suwannee River Water Management District (SRWMD)

Import Type

After discussion with Serge and others in an import meeting, the consensus was that an automated import with subsequent manual review and clean-up would be best for the area features, e.g. lakes and ponds. Reasons for this include:

  • A snapshot of OSM water features in Florida will be intersected with the water features from the landcover data. This will eliminate potential duplicate features as long as the import is carried out within a day or so of the OSM download and analysis.
  • There's approx 100,000 features to import, so a manual review just looking for basic conflicts will be too time consuming. The automated process should handle it.
  • Manual review after the import to double-check the import.
  • Manual review and re-alignment to fix some features that may have not been correctly mapped in the landcover data set.
  • Coastline updates will be a separate manual process.

Five people so far have signed on to work on this, with potentially more help from OSM Tampa.

There are no plans for taking in or processing subsequent updates (e.g. feeds, diffs, etc.)

Data Preparation

Landcover Processing

-- one or more shapefiles per Water Management District
ogr2ogr -select "fluccscode, flucsdesc, lev1, lev2, lev3, lev4, gis_update, comments" -append -skipfailures -nlt GEOMETRY -f "PostgreSQL" PG:"user=xx dbname=x host=xx port=xx password=xx" -nln landcover_swfwmd_2011 LANDUSELANDCOVER2011_arcad_ne.shp
-- projection is EPSG:2882
select UpdateGeometrySRID('landcover_swfwmd_2011_water', 'wkb_geometry', 2882)

-- check for invalid polygons
select st_isvalidreason(wkb_geometry) from landcover_swfwmd_2011 where st_isvalid(wkb_geometry) is false
-- its all Ring Self-intersections

-- fix invalid polygons
UPDATE landcover_swfwmd_2011
            SET wkb_geometry = ST_Buffer(wkb_geometry, 0.0)
            WHERE ST_IsValid(wkb_geometry) = false AND ST_IsValidReason(wkb_geometry) LIKE '%Ring Self-intersection%';

-- check to make sure there are no more invalid polygons
SELECT ST_IsValidReason(wkb_geometry) FROM landcover_swfwmd_2011 WHERE ST_IsValid(wkb_geometry) is false

-- create water-only landcover
-- union all polygons together to get rid of tile borders (source data was 32 tiles)
-- also selects just water
-- also dumps the multi-polygon output from st_union to single polygons
-- DROP TABLE landcover_swfwmd_2011_water
CREATE TABLE landcover_swfwmd_2011_water AS SELECT (ST_Dump(ST_Union(wkb_geometry))).geom as wkb_geometry,fluccscode,flucsdesc,lev1 FROM landcover_swfwmd_2011 WHERE lev1 = 5 GROUP BY fluccscode,flucsdesc,lev1;

-- add primary key
ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN ogc_fid SERIAL;
  
-- add geometry index
CREATE INDEX landcover_swfwmd_2011_water_idx
  ON landcover_swfwmd_2011_water
  USING gist
  (wkb_geometry);
  
-- add fluccscode index
CREATE INDEX landcover_swfwmd_2011_water_idx2
  ON landcover_swfwmd_2011_water
  USING btree
  (fluccscode);
  
-- visually QA output

-- Inspect codes
SELECT fluccscode,flucsdesc FROM landcover_swfwmd_2011_water GROUP BY fluccscode,flucsdesc ORDER BY fluccscode
-- cnt;fluccscode;,flucsdesc
-- 490;5100;"STREAMS AND WATERWAYS"
-- 3032;5200;"LAKES"
-- 33847;5300;"RESERVOIRS"
-- 2114;5400;"BAYS AND ESTUARIES"
-- 16;5720;"GULF OF MEXICO"

-- how many vertices do we have? 1814802
SELECT SUM(ST_NPoints(wkb_geometry)) FROM landcover_swfwmd_2011_water

-- simplify vertices to 1-meter
UPDATE landcover_swfwmd_2011_water SET wkb_geometry = ST_SimplifyPreserveTopology(wkb_geometry,1)

-- now how many vertices do we have? 1566758
SELECT SUM(ST_NPoints(wkb_geometry)) FROM landcover_swfwmd_2011_water

-- ADD natural and water columns
-- natural is reserved postgres word
ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN "natural" text;
ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN water text;

UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'river' WHERE fluccscode = 5100
UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'lake' WHERE fluccscode = 5200
UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'pond' WHERE fluccscode = 5300
UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'lagoon' WHERE fluccscode = 5400


# output to SHP file
# project the polygons to 4326 to match OSM
# skip codes 5400 and 5720
# select features that do not overlap with existing OSM water features
C:\"Program Files (x86)"\PostgreSQL\9.1\bin\pgsql2shp -f landcover_swfwmd_2011_water -h xx -p xx -u xx xx "SELECT  w.wkb_geometry, w.natural, w.water FROM landcover_swfwmd_2011_water as w WHERE w.fluccscode IN (5100,5200,5300) AND not exists (select * from planet_osm_polygon as osm where (osm.natural = 'water' or osm.waterway !=  or landuse = 'reservoir') and ST_IsValid( osm.way ) and ST_Intersects(osm.way,ST_TRANSFORM(w.wkb_geometry,900913))) and not exists ( select * from planet_osm_line as osm where osm.waterway !=  and ST_IsValid( osm.way ) and ST_Intersects(osm.way,ST_TRANSFORM(w.wkb_geometry,900913)))"

Tagging Plans

Current plans for tagging:

  • Water outlines - ways - natural=water
  • Water outlines - ways - water=*


Changeset Tags

Florida Water Import from <year> <water_management_district> Landcover

e.g. Florida Water Import from 2011 SWFWMD Landcover

Data Transformation

The source files are .shp-based and will need to be converted to OSM XML.

C:\"Program Files (x86)"\Java\jre6\bin\java -cp shp-to-osm-0.8.7-jar-with-dependencies.jar com.yellowbkpk.geo.shp.Main --maxnodes 30000 --shapefile landcover_swfwmd_2011_water.shp --rulesfile rules_swfwmd.txt --osmfile swfwmd_water_wgs84_.osc


Data Transformation Results

These files are from 7/8/2013. The whole automated process will be repeated before the import is done.

All of SWFWMD (63MB):

http://www.mapwise.com/temp/swfwmd_water_BIG0.osc

A few sections (approx 2MB each):

http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc1.osm
http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc2.osm
http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc3.osm
http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc4.osm


Workflow

Automated Portion

  1. Extract a copy of OSM for Florida
  2. Load into postgres
  3. Import landcover into postgres
  4. Check and fix invalid polygons
  5. Select water features from landcover
  6. Simplify vertices to 1m tolerance
  7. Process landcover water fields to get tags needed
  8. Export from postgres to shapefile while selecting landcover water where not intersects OSM water and projecting to EPSG:4326
  9. Convert to SHP to multiple .OSC files in 10k chunks
  10. Upload .OSC files to OSM

Manual QA Portion

  1. Assign areas to people as cake slices?
  2. Visually verify against Bing image.
  3. Realign ways that are grossly off or have changed.
  4. Validate the data using tools in the editors of choice.


Dedicated Import Account

TBD - follow Seattle Import convention?


References

The Seattle Import page was heavily referenced in putting this page together.