Florida Land Cover Import
This page documents the process for importing water features extracted from land cover data that was mapped by Florida's five Water Management Districts in 2008-2010.
The primary goal of this effort is to add missing water areas and replace parts of unedited or poorly edited coastline with this data.
Lake names were discussed as another input to the import, but were dropped in order to move the project forward. Another person is interested in conflating lake names for other parts of the country, so Florida can be rolled into that project.
- Planning: June / July 2013
- Import: Starting September 2013, ending: October 2013?
- QA: post-import
Data source site: http://www.swfwmd.state.fl.us/data/gis/layer_library/category/physical_dense
Data license: Public Domain - All data produced by state and local governments in Florida is public domain according to Chapter 119, Florida Statutes, Florida’s Public Records Act
Permissions: Our data are freely available under Florida’s public records law and we don’t place any restrictions on the use. Hopefully this statement will work:
“These records are being produced under Chapter 119, Florida Statutes, Florida’s Public Records Act. Section 119.91(1), F.S. provides: ‘It is the policy of this state that all state, county, and municipal records are open for personal inspection and copying by any person. Providing access to public records is a duty of each agency.’”
I would just ask that you include the original source of the data, either a copy of our metadata or at least a link to our download page, in your metadata. Please feel free to contact me if you have any other questions.
- Northwest Florida Water Management District (NWFWMD)
- South Florida Water Management District (SFWMD)
- Southwest Florida Water Management District (SWFMWD) - One large shapefile in Florida StatePlane West HARN
- St Johns River Water Management District (SJRWMD)
- Suwannee River Water Management District (SRWMD)
After discussion with Serge and others in an import meeting, the consensus was that an automated import with subsequent manual review and clean-up would be best for the area features, e.g. lakes and ponds. Reasons for this include:
- A snapshot of OSM water features in Florida will be intersected with the water features from the landcover data. This will eliminate potential duplicate features as long as the import is carried out within a day or so of the OSM download and analysis.
- There's approx 100,000 features to import, so a manual review just looking for basic conflicts will be too time consuming. The automated process should handle it.
- Manual review after the import to double-check the import.
- Manual review and re-alignment to fix some features that may have not been correctly mapped in the landcover data set.
- Coastline updates will be a separate manual process.
Five people so far have signed on to work on this, with potentially more help from OSM Tampa.
There are no plans for taking in or processing subsequent updates (e.g. feeds, diffs, etc.)
-- one or more shapefiles per Water Management District ogr2ogr -select "fluccscode, flucsdesc, lev1, lev2, lev3, lev4, gis_update, comments" -append -skipfailures -nlt GEOMETRY -f "PostgreSQL" PG:"user=xx dbname=x host=xx port=xx password=xx" -nln landcover_swfwmd_2011 LANDUSELANDCOVER2011_arcad_ne.shp
-- projection is EPSG:2882 select UpdateGeometrySRID('landcover_swfwmd_2011_water', 'wkb_geometry', 2882) -- check for invalid polygons select st_isvalidreason(wkb_geometry) from landcover_swfwmd_2011 where st_isvalid(wkb_geometry) is false -- its all Ring Self-intersections -- fix invalid polygons UPDATE landcover_swfwmd_2011 SET wkb_geometry = ST_Buffer(wkb_geometry, 0.0) WHERE ST_IsValid(wkb_geometry) = false AND ST_IsValidReason(wkb_geometry) LIKE '%Ring Self-intersection%'; -- check to make sure there are no more invalid polygons SELECT ST_IsValidReason(wkb_geometry) FROM landcover_swfwmd_2011 WHERE ST_IsValid(wkb_geometry) is false -- create water-only landcover -- union all polygons together to get rid of tile borders (source data was 32 tiles) -- also selects just water -- also dumps the multi-polygon output from st_union to single polygons -- DROP TABLE landcover_swfwmd_2011_water CREATE TABLE landcover_swfwmd_2011_water AS SELECT (ST_Dump(ST_Union(wkb_geometry))).geom as wkb_geometry,fluccscode,flucsdesc,lev1 FROM landcover_swfwmd_2011 WHERE lev1 = 5 GROUP BY fluccscode,flucsdesc,lev1; -- add primary key ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN ogc_fid SERIAL; -- add geometry index CREATE INDEX landcover_swfwmd_2011_water_idx ON landcover_swfwmd_2011_water USING gist (wkb_geometry); -- add fluccscode index CREATE INDEX landcover_swfwmd_2011_water_idx2 ON landcover_swfwmd_2011_water USING btree (fluccscode); -- visually QA output -- Inspect codes SELECT fluccscode,flucsdesc FROM landcover_swfwmd_2011_water GROUP BY fluccscode,flucsdesc ORDER BY fluccscode -- cnt;fluccscode;,flucsdesc -- 490;5100;"STREAMS AND WATERWAYS" -- 3032;5200;"LAKES" -- 33847;5300;"RESERVOIRS" -- 2114;5400;"BAYS AND ESTUARIES" -- 16;5720;"GULF OF MEXICO" -- how many vertices do we have? 1814802 SELECT SUM(ST_NPoints(wkb_geometry)) FROM landcover_swfwmd_2011_water -- simplify vertices to 1-meter UPDATE landcover_swfwmd_2011_water SET wkb_geometry = ST_SimplifyPreserveTopology(wkb_geometry,1) -- now how many vertices do we have? 1566758 SELECT SUM(ST_NPoints(wkb_geometry)) FROM landcover_swfwmd_2011_water -- ADD natural and water columns -- natural is reserved postgres word ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN "natural" text; ALTER TABLE landcover_swfwmd_2011_water ADD COLUMN water text; UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'river' WHERE fluccscode = 5100 UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'lake' WHERE fluccscode = 5200 UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'pond' WHERE fluccscode = 5300 UPDATE landcover_swfwmd_2011_water SET "natural" = 'water', water = 'lagoon' WHERE fluccscode = 5400
# output to SHP file # project the polygons to 4326 to match OSM # skip codes 5400 and 5720 # select features that do not overlap with existing OSM water features C:\"Program Files (x86)"\PostgreSQL\9.1\bin\pgsql2shp -f landcover_swfwmd_2011_water -h xx -p xx -u xx xx "SELECT w.wkb_geometry, w.natural, w.water FROM landcover_swfwmd_2011_water as w WHERE w.fluccscode IN (5100,5200,5300) AND not exists (select * from planet_osm_polygon as osm where (osm.natural = 'water' or osm.waterway != or landuse = 'reservoir') and ST_IsValid( osm.way ) and ST_Intersects(osm.way,ST_TRANSFORM(w.wkb_geometry,900913))) and not exists ( select * from planet_osm_line as osm where osm.waterway != and ST_IsValid( osm.way ) and ST_Intersects(osm.way,ST_TRANSFORM(w.wkb_geometry,900913)))"
Current plans for tagging:
- Water outlines - ways -
- Water outlines - ways -
Florida Water Import from <year> <water_management_district> Landcover
e.g. Florida Water Import from 2011 SWFWMD Landcover
The source files are .shp-based and will need to be converted to OSM XML.
C:\"Program Files (x86)"\Java\jre6\bin\java -cp shp-to-osm-0.8.7-jar-with-dependencies.jar com.yellowbkpk.geo.shp.Main --maxnodes 30000 --shapefile landcover_swfwmd_2011_water.shp --rulesfile rules_swfwmd.txt --osmfile swfwmd_water_wgs84_.osc
Data Transformation Results
These files are from 7/8/2013. The whole automated process will be repeated before the import is done.
All of SWFWMD (63MB):
A few sections (approx 2MB each):
http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc1.osm http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc2.osm http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc3.osm http://www.mapwise.com/temp/swfwmd_water_wgs84_.osc4.osm
- Extract a copy of OSM for Florida
- Load into postgres
- Import landcover into postgres
- Check and fix invalid polygons
- Select water features from landcover
- Simplify vertices to 1m tolerance
- Process landcover water fields to get tags needed
- Export from postgres to shapefile while selecting landcover water where not intersects OSM water and projecting to EPSG:4326
- Convert to SHP to multiple .OSC files in 10k chunks
- Upload .OSC files to OSM
Manual QA Portion
- Assign areas to people as cake slices?
- Visually verify against Bing image.
- Realign ways that are grossly off or have changed.
- Validate the data using tools in the editors of choice.
Dedicated Import Account
TBD - follow Seattle Import convention?
The Seattle Import page was heavily referenced in putting this page together.