In this tutorial you'll learn how to perform liveness detection in a video stream with Face SDK and an RGBD sensor. As a rule, liveness detection is used to prevent spoofing attacks (when a person tries to subvert or attack a face recognition system by using a picture or a video and thereby gaining illegitimate access).
With Face SDK you can perform liveness detection by analyzing a depth map or an RGB image from your sensor. The first method is more accurate, that's why we'll consider it in this tutorial.
This tutorial is based on Face Recognition in a Video Stream and the corresponding project. In this project we will also use a ready-made database of faces for recognition. After you run the project, you'll see RGB and depth maps, which you can use to correct your position relative to the sensor: to ensure the stable performance of a liveness detector, your face should be at a suitable distance from a sensor, and the quality of the depth map should be sufficient.
A detected and recognized face will be highlighted with a green rectangle on an RGB image. Next to the detected face, there'll be a picture and a person's name from the database. Also, you'll see the
REAL liveness status. If a person is not recognized, the liveness status will be
REAL, but the bounding rectangle will be red. If a detected face is taken from a picture or a video, the bounding rectangle will be red and recognition won't be performed. In this case the liveness status will be
Besides Face SDK and Qt, you'll need the following:
- An RGBD sensor with OpenNI2 or RealSense2 support (for example, ASUS Xtion or RealSense D415).
- OpenNI2 or RealSense2 distribution package.
You can find the tutorial project in Face SDK: examples/tutorials/depth_liveness_in_face_recognition
- First of all, import necessary libraries to work with the depth camera. You can use either an OpenNI2 sensor (for example, ASUS Xtion) or a RealSense2 sensor. Depending on the camera used, specify the
- [For OpenNI2 sensors] Specify the path to the OpenNI2 distribution package and also the paths to the necessary OpenNI2 libraries and headers.
Note: For Windows install OpenNI2 and specify the path to the installation directory. For Linux just specify the path to the unpacked archive.
- [For RealSense sensors] Specify the path to the RealSense2 distribution package and the paths to the necessary RealSense2 libraries and headers. In the
win32block, we determine the platform bitness to set the correct paths to the RealSense libraries.
Note: For Windows install RealSense2 and specify the path to the installation directory. For Linux install RealSense2 as described at the Intel RealSense website.
- At this stage, retrieve a depth frame from an RGBD sensor using OpenNI2 API or RealSense2 API, depending on the camera used. We won't elaborate on retrieving the depth frames. Instead, we'll use the headers from one of Face SDK samples (video_recognition_demo). In the project profile specify the path to the examples/cpp/video_recognition_demo/src folder from Face SDK.
- Specify the necessary headers to work with OpenNI2 and RealSense2 cameras. You can find the detailed information about retreving the depth frames in the specified files (
- To use mathematical constants, define
cmathis already imported in
- In previous projects we retrieved the image from a webcam using the
QCameraCaptureobject. However, in this project we need to retrieve both RGB and depth frames. To do this, create a new class
DepthSensorCapture: Add New > C++ > C++ Class > Choose… > Class name – DepthSensorCapture > Base class – QObject > Next > Project Management (default settings) > Finish.
depthsensorcapture.h, import the
ImageAndDepthSourceheader. Also import
QSharedPointerto handle pointers, import
QThreadto process threads, import
QByteArrayto work with byte arrays, import
atomicto handle smart pointers and atomic types, respectively. In
depthsensorcapture.cpp, import the
RealSenseSourceheaders to retrieve the depth frames, and also import
assert.hto handle errors, and use
QMessageBoxto display the error message.
RGBFramePtr, which is a pointer to an RGB frame, and
DepthFramePtr, which is a pointer to a depth frame. The
DepthSensorCaptureclass constructor takes a parent widget and also a pointer to worker. The sensor data will be received in an endless loop. To prevent the main thread, where the interface is rendered, from waiting for the cycle completion, create another thread and move the
DepthSensorCaptureobject into this new thread.
DepthSensorCapture::start, start the thread, where the data is received, and stop it in
DepthSensorCapture::frameUpdatedThread, process a new frame from the sensor in an endless loop and pass it to
addFrame. If an error occurs, an error message box will be displayed.
VideoFrameobject should contain an RGB frame from the sensor.
videoframe.h, import the
depthsensorcaptureheader to work with the depth sensor.
IRawImageinterface allows to receive a pointer to the image data, its height and width.
- Start the camera in the
runProcessingmethod, and stop the camera in the
worker.h, import the
SharedImageAndDepthstructure contains the pointers to an RGB frame and a depth frame from the sensor, and also the
pbio::DepthMapRawstructure with the information about depth map parameters (width, height, etc.). The pointers are used in
Worker. Due to some delay in frame processing, a certain number of frames is queued for rendering. To save the memory space, store the pointers to frames instead of the frames.
SharedImageAndDepthframe, which is RGB and depth frames for rendering, to the
TrackingCallback, we extract the image corresponding to the last result received from the frame queue.
Worker::Worker, override the values of some parameters of the
VideoWorkerobject to process the depth map, namely:
depth_data_flag(to confirm face liveness, "1" turns on depth frame processing);
weak_tracks_in_tracking_callback("1" means that all samples, even the ones flagged as
weak=true, are passed to
weak flag becomes
true if a sample doesn't pass certain tests, for example:
- if there are too many shadows on a face (insufficient lighting)
- if an image is blurry
- if a face is turned at a great angle
- if the size of a face in the frame is too small
- if a face didn't pass the liveness test (for example, if the face is taken from a photo or a video)
You can find the detailed information about lighting conditions, camera positioning, etc. in Guidelines for Cameras. As a rule, samples that didn't pass the tests, are not processed and not used for recognition. However, in this project we want to highlight all the faces, even if they are taken from the picture (those that didn't pass the liveness test). Therefore, pass all samples to
TrackingCallback, even if they are flagged as
- In the
worker.h, specify the enumeration
pbio::DepthLivenessEstimator, which is the result of liveness estimation. All in all, there are four liveness statuses:
- NOT_ENOUGH_DATA means that face information is insufficient. This situation may occur if the depth map quality is poor or the user is too close/too far from the sensor.
- REAL means that the face belongs to a real person.
- FAKE means that the face is taken from a picture or a video.
- NOT_COMPUTED means that the face was not checked. This situation may occur, for example, if the frames from the sensor are not synchronized (an RGB frame is received but a corresponding depth frame is not found in a certain time range).
The liveness test result is stored in the
face.liveness_status variable for further rendering.
Worker::addFrame, pass the last depth frame to Face SDK using the
VideoWorker::addDepthFramemethod and store it for further processing.
- Prepare and pass the RGB image to Face SDK using
VideoWorker::addVideoFrame. If the format of the received RGB image is BGR instead of RGB, the byte order is changed, so that the image colors are displayed correctly. If a depth frame is not received together with an RGB frame, the last received depth frame is used. A pair of the depth and RGB frames is queued in
_framesin order to find the data, which corresponds to the processing result in
framefield of the
Worker::DrawingDatastructure contains the pointers to RGB frame data and depth frame data, as well as depth frame parameters (width, height, etc.). For convenience, create the references
const QImage& color_image,
const QByteArray& depth_array, and
const pbio::DepthMapRaw& depth_optionsto refer to these data. The RGB image and depth map will be displayed in
QImage result, which can be considered as a sort of a "background" and contains both images (an RGB image at the top and a depth map at the bottom). Before that, to display the depth map correctly (in grayscale), convert 16-bit depth values to 8-bit depth values. In the
max_depth_mmvalue, specify the maximum distance from the sensor to the user (the distance is usually 10 meters).
- Form the depth image from the converted values. Create the
resultobject, which will be used to display an RGB image (at the top) and a depth map (at the bottom). Render these images.
- Display the liveness status next to the face, depending on the information received from the liveness detector. Specify the label parameters (color, line, size). You'll see a bounding rectangle in the depth map, so that you can make sure that the RGB frame and depth frame are aligned.
- Run the project. You'll see an RGB image and a depth map from the sensor. Also, you'll see the information about the detected face:
- detection and recognition status (which is indicated by the color of the bounding rectangle: green means that a person is detected and found in the database; red means that a person is not recognized, or the face is taken from an image or a video).
- information about the recognized person (a person's image and name from the database).
- liveness status; real means that a person is real; fake means that a face is taken from an image or a video; not_enough_data means that the depth map quality is poor, or a person is too close/too far away from the sensor; not_computed means that RGB and depth frames are not synchronized.