InSight SDK 3.2
The facial analysis toolkit
 All Classes Functions Variables Pages
InSight SDK

Introduction

InSight SDK is a cross platform software library which can be used to seamlessly integrate face analysis and facial expression recognition in third party applications. The SDK is extremely flexible, and allows retrieving tailored information depending on the specific requirements of the companies or individuals. Via a simple webcam the InSight SDK can be used to automatically analyze face videos in real time, and communicate the resulting information to a third party application. The information ranges from seven emotional categories (neutral, happy, surprised, sad, afraid, angry and disgusted) to head pose, gaze, eye movements and age and gender of the subjects. The SDK is based on the following state-of-the art technologies developed in collaboration with the University of Amsterdam:

The SDK can be easily integrated in third party software as a collection of C++ shared libraries for Windows, Mac and Linux environments. The SDK requires an internet connection to function properly, as it will communicate with our API to validate the license.

Usage

InSight SDK works on a video sequence of frames. The first frame should be the initialization frame, after that all frames should be processes and the information about the user can be retrieved by using the getter functions. The following is the pseudo-code to retrieve and display the facial expressions of a user in each frame:

Initialize InSight and the capturing device/video input
for (;;)
{
capture >> frame; // get a new frame from the video stream
if (!insight.isInit())
{
insight.init(frame) // initialize InSight with the first frame
}
else
{
insight.process(frame) //process the new frame
insight.getEmotions(emotions) //retrieve the facial expressions
print(emotions) //print them to console
}
}

For the full code, refer to the example implementation provided with the SDK. Additional usage examples can be provided upon request. The following is the pseudo-code to calibrate and obtain eye gaze information on each frame:

// fill a vector with at least nine calibration points
std::vector<cv::Point> calibrationPoints = nine calibration points;
std::vector<cv::Point>::iterator it = calibrationPoints.begin();
// this vector will accumulate processed calibration points
std::vector<CalibInfo> calibCollection;
cv::Mat frame;
bool isCalibrated = false;
int drawing = 0;
while( grab( frame ) )
{
// initialize insight if necessary
if( !insight->isInit() && !insight->init( frame ) )
{
printError( insight->getError() );
break;
}
// always process a frame
if( !insight->process( frame ) )
{
printError( "Failed to process" );
break;
}
if( !isCalibrated )
{
cv::Point & calibrationPoint = *it;
if( it != calibrationPoints.end() )
{
// draw calibrationPoint on screen for 25 frames
// to allow the user to fixate on the point
if( drawing < 25 )
{
drawDotOnScreen( calibrationPoint );
drawing++;
}
else
{
drawing = 0;
// process calibration point
if( !insight->addCalibrationPoint( calibrationPoint, ci ) )
{
printError( "Failed to add calibration point" );
break;
}
// add processed calibration point to collection
calibCollection.push_back( ci );
// move to next calibration point
it++;
}
}
else
{
// calibrate using the processed calibration point collection
std::vector<cv::Point2f> calibErrors;
if(!insight->calibrate( calibCollection, calibErrors) )
{
printError( "Failed to calibrate" );
break;
}
isCalibrated = true;
// do something with the reprojection errors if required
printReprojectionErrors( calibErrors );
}
}
else
{
// insight get gaze
cv::Point estimatedGaze;
insight->getEyeGaze( estimatedGaze );
// do something with the acquired gaze point
drawOnScreen( estimatedGaze )
}
}

Requirements

Platform requirements

The minimum platform requirements.

User requirements

InSight SDK is designed for a single-user scenario, in which the user is around 60 cm away from the camera (eg: like in front of a desktop or a laptop computer). The following user requirements should be met in order to use the InSight SDK.

Environment requirements

The following environment conditions should be met in order to use the InSight SDK.