Visual Servoing Platform  version 3.3.0
Tutorial: Keypoint tracking

Introduction

With ViSP it is possible to track keypoints using OpenCV KLT tracker, an implementation of the Kanade-Lucas-Tomasi feature tracker.

All the material (source code and video) described in this tutorial is part of ViSP source code and could be downloaded using the following command:

$ svn export https://github.com/lagadic/visp.git/trunk/tutorial/tracking/keypoint

KLT tracker

The following example code available in tutorial-klt-tracker.cpp shows how to use ViSP vpKltOpencv class to track KLT keypoints. This class is a wrapper over the OpenCV KLT tracker implementation.

#include <visp3/core/vpImageConvert.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/io/vpVideoReader.h>
#include <visp3/klt/vpKltOpencv.h>
int main(int argc, const char *argv[])
{
#ifdef VISP_HAVE_OPENCV
try {
bool opt_init_by_click = false;
for (int i = 0; i < argc; i++) {
if (std::string(argv[i]) == "--init-by-click")
opt_init_by_click = true;
else if (std::string(argv[i]) == "--help") {
std::cout << "Usage: " << argv[0] << " [--init-by-click] [--help]" << std::endl;
return 0;
}
}
vpVideoReader reader;
reader.setFileName("video-postcard.mpeg");
reader.acquire(I);
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
IplImage *cvI = NULL;
#else
cv::Mat cvI;
#endif
vpDisplayOpenCV d(I, 0, 0, "Klt tracking");
vpKltOpencv tracker;
tracker.setMaxFeatures(200);
tracker.setWindowSize(10);
tracker.setQuality(0.01);
tracker.setMinDistance(15);
tracker.setHarrisFreeParameter(0.04);
tracker.setBlockSize(9);
tracker.setUseHarris(1);
tracker.setPyramidLevels(3);
// Initialise the tracking
if (opt_init_by_click) {
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
std::vector<CvPoint2D32f> feature;
#else
std::vector<cv::Point2f> feature;
#endif
do {
vpDisplay::displayText(I, 10, 10, "Left click to select a point, right to start tracking", vpColor::red);
if (vpDisplay::getClick(I, ip, button, false)) {
if (button == vpMouseButton::button1) {
feature.push_back(cv::Point2f((float)ip.get_u(), (float)ip.get_v()));
}
}
} while (button != vpMouseButton::button3);
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
tracker.initTracking(cvI, &feature[0], feature.size());
#else
tracker.initTracking(cvI, feature);
#endif
} else {
tracker.initTracking(cvI);
}
std::cout << "Tracker initialized with " << tracker.getNbFeatures() << " features" << std::endl;
while (!reader.end()) {
reader.acquire(I);
if (opt_init_by_click && reader.getFrameIndex() == reader.getFirstFrameIndex() + 20) {
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
std::vector<CvPoint2D32f> feature;
#else
std::vector<cv::Point2f> feature;
#endif
do {
vpDisplay::displayText(I, 10, 10, "Left click to select a point, right to start tracking", vpColor::red);
if (vpDisplay::getClick(I, ip, button, false)) {
if (button == vpMouseButton::button1) {
feature.push_back(cv::Point2f((float)ip.get_u(), (float)ip.get_v()));
}
}
} while (button != vpMouseButton::button3);
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
tracker.initTracking(cvI, &feature[0], feature.size());
#else
tracker.initTracking(cvI, feature);
#endif
}
tracker.track(cvI);
tracker.display(I, vpColor::red);
}
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
cvReleaseImage(&cvI);
#endif
return 0;
} catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#else
(void)argc;
(void)argv;
#endif
}

The video shows the result of the tracking:

The previous example can be run without command line options. In that case, keypoints are automatically detected before tracking.

$ ./tutorial-klt-tracker

It can also be run with [–init-by-click] option. In that case, the user can select a set of keypoints to track with a left mouse click. A right mouse click stops the keypoints selection and allows to start the tracking.

$ ./tutorial-klt-tracker --init-by-click

Here is the line by line explanation of the source :

#include <visp3/core/vpImageConvert.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/io/vpVideoReader.h>
#include <visp3/klt/vpKltOpencv.h>

We include here the headers that define the corresponding classes. vpImageConvert class will be used to convert ViSP images implemented in vpImage class into OpenCV IplImage or cv::Mat structures used as an entry by the KLT tracker. Then we include the header of vpKltOpencv class which is the wrapper over OpenCV KLT tracker implementation.

Note
  • If OpenCV version is less that 2.8.0, vpKltOpencv class takes as input an IplImage.
  • If OpenCV version is 2.8.0 or ;ore recent, vpKltOpencv class takes as input a cv::Mat image.

We need also to include a device to display the images. We retain vpDisplayOpenCV that works on Unix and Windows since OpenCV is mandatory by the tracker. Finally we include vpVideoReader header that will be used to read an mpeg input stream.

At the beginning of the main() function, we use the following macro to ensure that OpenCV requested by the tracker is available. Note that OpenCV will also be used to render the images and read the input video stream.

#ifdef VISP_HAVE_OPENCV

The program starts by the creation of a vpVideoReader instance able to extract all the images of the video file video-postcard.mpeg. Here, the video should be in the same folder than the binary.

vpVideoReader reader;
reader.setFileName("video-postcard.mpeg");

Then we extract the first image of the video in the gray level ViSP image container I.

This image I is then converted into cvI, an OpenCV image format that will be used by the tracker.

#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
IplImage *cvI = NULL;
#else
cv::Mat cvI;
#endif

We also create a window associated to I, at position (0,0) in the screen, with "Klt tracking" as title, and display image I.

vpDisplayOpenCV d(I, 0, 0, "Klt tracking");

From now we have to create an instance of the tracker and set the parameters of the Harris keypoint detector.

vpKltOpencv tracker;
tracker.setMaxFeatures(200);
tracker.setWindowSize(10);
tracker.setQuality(0.01);
tracker.setMinDistance(15);
tracker.setHarrisFreeParameter(0.04);
tracker.setBlockSize(9);
tracker.setUseHarris(1);
tracker.setPyramidLevels(3);

The tracker is then initialized on cvI image.

tracker.initTracking(cvI);

With the next line the user can know how many keypoints were detected automatically or selected by the user during initialization.

std::cout << "Tracker initialized with " << tracker.getNbFeatures() << " features" << std::endl;
Note
If no keypoints were found, the next call to vpKltTracker::track() will throw an exception.

To detect more keypoints, you may decrease the quality parameter set with the following line:

tracker.setQuality(0.01);

Until the end of the video, we get I the next image in ViSP format, display and convert it in OpenCV format. Then we track the Harris keypoints using KLT tracker before displaying the keypoints that are tracked with a red cross.

while (!reader.end()) {
reader.acquire(I);
if (opt_init_by_click && reader.getFrameIndex() == reader.getFirstFrameIndex() + 20) {
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
std::vector<CvPoint2D32f> feature;
#else
std::vector<cv::Point2f> feature;
#endif
do {
vpDisplay::displayText(I, 10, 10, "Left click to select a point, right to start tracking", vpColor::red);
if (vpDisplay::getClick(I, ip, button, false)) {
if (button == vpMouseButton::button1) {
feature.push_back(cv::Point2f((float)ip.get_u(), (float)ip.get_v()));
}
}
} while (button != vpMouseButton::button3);
#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
tracker.initTracking(cvI, &feature[0], feature.size());
#else
tracker.initTracking(cvI, feature);
#endif
}
tracker.track(cvI);
tracker.display(I, vpColor::red);
}

We are waiting for a mouse click event on image I to end the program.

With the following line, we release the memory allocated for the OpenCV IplImage cvI before ending the program. This has to be done only if OpenCV version is less than 2.8.0.

#if (VISP_HAVE_OPENCV_VERSION < 0x020408)
cvReleaseImage(&cvI);
#endif

KLT tracker with re-initialisation

Once initialized, the number of tracked features decreases over the time. Depending on a criteria, it may sense to detect and track new features online. A possible criteria is for example to compare the number of currently tracked features to the initial number of detected features. If less than a given percentage of features are tracked, you can start a new detection.

To get the number of detected or tracked features just call:

tracker.getNbFeatures();

Then the idea is to add the previously tracked features to the list of features that are detected.

The example tutorial-klt-tracker-with-reinit.cpp shows how to do that. In that example we start a new detection on frame 25. Compared to the previous code available in tutorial-klt-tracker.cpp we add the following lines:

if (reader.getFrameIndex() == 25) {
std::cout << "Re initialize the tracker" << std::endl;
#if (VISP_HAVE_OPENCV_VERSION >= 0x020408)
// Save of previous features
std::vector<cv::Point2f> prev_features = tracker.getFeatures();
// Start a new feature detection
tracker.initTracking(cvI);
std::vector<cv::Point2f> new_features = tracker.getFeatures();
// Add previous features if they are not to close to detected one
double distance, minDistance_ = tracker.getMinDistance();
bool is_redundant;
for (size_t i=0; i < prev_features.size(); i++) {
// Test if a previous feature is not redundant with one of the newly detected
is_redundant = false;
for (size_t j=0; j < new_features.size(); j++){
distance = sqrt(vpMath::sqr(new_features[j].x-prev_features[i].x)
+ vpMath::sqr(new_features[j].y-prev_features[i].y));
if(distance < minDistance_){
is_redundant = true;
break;
}
}
if(is_redundant){
continue;
}
//std::cout << "Add previous feature with index " << i << std::endl;
tracker.addFeature(prev_features[i]);
}
#else
...
#endif
}
// Track the features
tracker.track(cvI);

In this code we do the following:

  • save the features that are tracked until now
  • initialize the tracker to detect new features
  • parse all the saved features and compare them to the newly detected features. If a previous feature is close in terms of geometric distance to a newly detected one, it is rejected (in our case less than 2 pixels). If not, it is added to the list of detected features.

Next tutorial

You are now ready to see the next Tutorial: Moving-edges tracking.

vpKltOpencv::addFeature
void addFeature(const float &x, const float &y)
Definition: vpKltOpencv.cpp:516
vpMath::sqr
static double sqr(double x)
Definition: vpMath.h:113
vpTime::wait
VISP_EXPORT int wait(double t0, double t)
Definition: vpTime.cpp:172
vpKltOpencv::setMinDistance
void setMinDistance(double minDistance)
Definition: vpKltOpencv.cpp:380
vpKltOpencv::getNbFeatures
int getNbFeatures() const
Get the number of current features.
Definition: vpKltOpencv.h:119
vpKltOpencv::track
void track(const cv::Mat &I)
Definition: vpKltOpencv.cpp:146
vpImageConvert::convert
static void convert(const vpImage< unsigned char > &src, vpImage< vpRGBa > &dest)
Definition: vpImageConvert.cpp:78
vpMouseButton::button1
Definition: vpMouseButton.h:52
vpKltOpencv::getFeatures
std::vector< cv::Point2f > getFeatures() const
Get the list of current features.
Definition: vpKltOpencv.h:104
vpKltOpencv::display
void display(const vpImage< unsigned char > &I, const vpColor &color=vpColor::red, unsigned int thickness=1)
Definition: vpKltOpencv.cpp:216
vpKltOpencv::setQuality
void setQuality(double qualityLevel)
Definition: vpKltOpencv.cpp:355
vpVideoReader::getFirstFrameIndex
long getFirstFrameIndex()
Definition: vpVideoReader.h:301
vpVideoReader::end
bool end()
Definition: vpVideoReader.h:259
vpKltOpencv::setHarrisFreeParameter
void setHarrisFreeParameter(double harris_k)
Definition: vpKltOpencv.cpp:363
vpKltOpencv::initTracking
void initTracking(const cv::Mat &I, const cv::Mat &mask=cv::Mat())
Definition: vpKltOpencv.cpp:117
vpVideoReader::getFrameIndex
long getFrameIndex() const
Definition: vpVideoReader.h:294
vpImagePoint::get_v
double get_v() const
Definition: vpImagePoint.h:273
vpDisplayOpenCV
The vpDisplayOpenCV allows to display image using the OpenCV library. Thus to enable this class OpenC...
Definition: vpDisplayOpenCV.h:140
vpKltOpencv
Wrapper for the KLT (Kanade-Lucas-Tomasi) feature tracker implemented in OpenCV. Thus to enable this ...
Definition: vpKltOpencv.h:77
vpVideoReader
Class that enables to manipulate easily a video file or a sequence of images. As it inherits from the...
Definition: vpVideoReader.h:170
vpColor::green
static const vpColor green
Definition: vpColor.h:181
vpVideoReader::setFileName
void setFileName(const std::string &filename)
Definition: vpVideoReader.cpp:91
vpDisplay::display
static void display(const vpImage< unsigned char > &I)
Definition: vpDisplay_uchar.cpp:739
vpDisplay::displayText
static void displayText(const vpImage< unsigned char > &I, const vpImagePoint &ip, const std::string &s, const vpColor &color)
Definition: vpDisplay_uchar.cpp:663
vpKltOpencv::setWindowSize
void setWindowSize(int winSize)
Definition: vpKltOpencv.cpp:342
vpImagePoint::get_u
double get_u() const
Definition: vpImagePoint.h:262
vpKltOpencv::setPyramidLevels
void setPyramidLevels(int pyrMaxLevel)
Definition: vpKltOpencv.cpp:408
vpKltOpencv::setUseHarris
void setUseHarris(int useHarrisDetector)
Definition: vpKltOpencv.cpp:371
vpImagePoint
Class that defines a 2D point in an image. This class is useful for image processing and stores only ...
Definition: vpImagePoint.h:87
vpMouseButton::button3
Definition: vpMouseButton.h:54
vpKltOpencv::getMinDistance
double getMinDistance() const
Definition: vpKltOpencv.h:117
vpDisplay::flush
static void flush(const vpImage< unsigned char > &I)
Definition: vpDisplay_uchar.cpp:715
vpDisplay::displayCross
static void displayCross(const vpImage< unsigned char > &I, const vpImagePoint &ip, unsigned int size, const vpColor &color, unsigned int thickness=1)
Definition: vpDisplay_uchar.cpp:179
vpImage< unsigned char >
vpDisplay::getClick
static bool getClick(const vpImage< unsigned char > &I, bool blocking=true)
Definition: vpDisplay_uchar.cpp:764
vpMouseButton::vpMouseButtonType
vpMouseButtonType
Definition: vpMouseButton.h:51
vpException
error that can be emited by ViSP classes.
Definition: vpException.h:70
vpVideoReader::acquire
void acquire(vpImage< vpRGBa > &I)
Definition: vpVideoReader.cpp:243
vpColor::red
static const vpColor red
Definition: vpColor.h:178
vpKltOpencv::setBlockSize
void setBlockSize(int blockSize)
Definition: vpKltOpencv.cpp:398
vpKltOpencv::setMaxFeatures
void setMaxFeatures(int maxCount)
Definition: vpKltOpencv.cpp:333