Google Summer of Code/2012/Video Based Speed Limit and Road Sign Detector plugin for JOSM
Project Title
Video Based Speed Limit and Road Sign Detector plugin for JOSM
Description
A GSoC12 project by me Nikhil Upadhye mentored by Kai Krueger. Project aims to develop a plugin for JOSM which will detect speedlimits and roadsigns from the GPS synchronized video and add a related tag into map data.
I will use this wiki page to post details and updates about project.
Summary of Project Goals
- Importing video and synchronizing time with GPS tracks
Time synchronization is to be done by using time in GPX file and time of capture of video.
- Image processing of video to detect for roadsigns and speed limit board
This part is to develope robust image processing code for roadsign detection.I propose to use opencv java bindings Javacv for image processing.Image processing will be done in two parts. First part will aim to detect roadsings as objects and identify them using shapes. Number detection algorithm will be developed for identification of numbers. Second part will identify the roadsign features by matching them with existing images from collection using harr classifier in opencv. For this part, collection of roadsign is to be done.
- User Interface
User Interface will be desined which will include video display screen, video timeline bar. After video is processed, detected features will be added in video timeline as well as displayed map. User will review detection and approve integration in map.
Timeline
May 21 to May 30: Build basic user interface for plugin for testing furthur development. Importing video in plugin and synchronizing with GPS tracks.
May 31 to June 20: Developing First part of Image processing as mentioned above. Road sign detection code based on object recognition for normal light condition video will be build during this period.
June 21 to July 1: Extending code for differnt light condition video.
July 2 to July 8: Reserved Week for unexpected delays and testing and review of code. Otherwise work will be carried furthur.
July 9 to July 13: Submition of Mid-term report
July 14 to July 31: Developing Second part of Image processing as mentioned in proposal. Road sign detection code by matching existing roadsign images.
August 1 to August 13: Reserved for unexpected delay, improving User Interface, testing, debugging and completing Documentation.
August 13 to August 24: Submission of Final report.
A Request to all Mappers
If you have videos recorded for video mapping, it would be really great if you could share it with me for testing purpose. You can email me at nikhil.spitfire@gmail.com
Project Updates
Github : http://github.com/nikhil9/
In initial period I studied existing videomapping and photomapping plugins for JOSM. I tried to fix videomapping plugin but couldn't get it work. Based on photo_geotagging plugin, I developed videoprocessor plugin removing unwanted features from photo_geotagging plugin. I will integrate image processing to this plugin.
In Image processing part I have implemented SURF for samplematching using javacv. Planning to implement knearest, haartraining algorithm for speedlimit and roadsign detection. I am preparing training dataset for haartraining. This could also improve surf based detector as more samples will be available for matching.
Swedish speed limit sign was haar-trained using opencv. Image below shows the result. Curretnly it only detects speed limit 70 sign board. Code can be found on github repo 'haarTraining'.
Plugins are created for processing Geotagged images and video which can be found at ImageProcessor and VideoPlayer.
- ImageProcessor
Plugin processes Images for speed limit/road sign detection and locate there position on map and zooms to location so that user can easily edit map and insert tags. This plugin works on Geotagged Images with Gpx tracks. After importing images and gpx track, geoimage layer is created after corelation process dialog. One can choose process these images by using option on geoimagelayer. Using plugin dialog one can navigate through images and chooses to process them for sign detection. Respective training file has to be given before processing the images for sign/speed limit board detection. Process for generating training file is explained below on this page. As plugin dialog displays image, map is zoomed to the location of image displayed at center. Plugin inserts a MarkerLayer on map. This is used to show the current position of image on the map. It is recommended to use Continuous Download plugin or download data from server before using plugin. If sign is detected in set of images, the are shown in separate dialog which can be used to zoom to location of those images. This makes it easy to add respective information at these location.
- VideoPlayer
Plugin Plays and processes video for speed limit/road sign detection and locate frame position on map corresponding to gpx file and zooms to location so that user can easily edit map and insert tags. After opening corresponding gpx layer, this plugin can be used to import video. One can play or browse video frame by frame using plugin dialog. Frames of the video are synchronized with gpx waypoints and map is zoomed to nearest waypoint for the ease of editing. Marker is also shown on map to represent current position. image processing part is similar to that of ImageProcessor plugin. Plugin supports most of the common video formats '.mp4', '.avi', '.mpg', '.mov', '.mpeg' etc. Currently only forward playback of video is possible due to limitation of video libraries used by plugins.
- Training Images
Tools required for training can be found at here.
Training images require 2 type of samples: positive samples negative samples
positive samples contain images with required sign and negative samples contain images without required sign. Around 300-400 positive and 500-600 negative samples are required for descent training. More the samples better is performance.
Follow the steps for training:
- Install opencv
- Use objectmarker program to set region of interests in each positive image and create info.txt.
This file is used to mark objects for training images. Put positive samples in folder 'positive' all the images should be in .bmp format and file name should be like '1.bmp', '2.bmp' etc. After running this program each images will pop up. Select region of interest using mouse (creating rectagle while left click pressed) use 'space' to add this data to file and 'enter' to save this data and load next file. Press 'escape' exit. 'info.txt' file will be generated which can be used to create samples for training.
- Directory Structure
Directories will look like following:
/positive
1.bmp 2.bmp
positive.info
/negative
1.bmp 2.bmp
negative.txt
positive.info will look like
positive/1.bmp 1 140 100 45 45 positive/2.bmp 1 100 200 50 50
- Use opencv-createsamples utility for generating positive vec file.
use following command to generate positive.vec file
opencv-createsamples -info positive.txt -vec positive.vec -w 30 -h 30
- Use opencv-trainincascade for training samples.
use following command to start training
opencv_traincascade -data <sing name> -vec positive.vec -bg negative.txt -numPos <number of positive samples> -numNeg <number of negative samples> -w 30 -h 30 -mode ALL
It will generate .xml file which can be used to plugins mentioned above. Training process can take more than 5-6 hour depending on processesing power of the system.