In this tutorial you'll learn how to recognize faces in a video stream. For recognition, use a ready-made database of faces from Face SDK distribution package. The database includes the images of several famous people. Recognized faces are highlighted with a green rectangle. The name and image of a recognized person are displayed next to the person's face in a video stream. This tutorial is based on Face Detection and Tracking in a Video Stream and the corresponding project.
You can find the tutorial project in Face SDK: examples/tutorials/face_recognition_with_video_worker
- In Face Detection and Tracking in a Video Stream, we set only two parameters of Face SDK (a path to Face SDK and a configuration file name for the
VideoWorkerobject). However, in this tutorial we need to set several more parameters: we'll add a path to the database, a configuration file name with a recognition method, and FAR. For convenience, we'll modify several files. Specify all the parameters in the
facesdkparameters.hspecify the path to the
- Pass the
face_sdk_parametersstructure to the constructor of the
- In this project we focus only on face detection in a video stream (creating a bounding rectangle) and face recognition. Note that in the first project (
detection_and_tracking_with_video_worker), which you can use as a reference for this project, we also displayed anthropometric points and angles. If you don't want to display this information, you can just remove unnecessary visualization from the first project.
First of all, create a database of faces. To check face recognition, you can use the ready-made database from Face SDK. This database includes images of three famous people (Elon Musk, Emilia Clarke, Lionel Messi). To check recognition, copy the database to the project root folder (next to a .pro file), run the project, open an image from the database, and point a camera at the screen. You can also add your picture to the database. To do this, create a new folder in the database, specify your name in a folder name, and copy your picture to the folder (in the same way as other folders in the database).
Create a new
Databaseclass to work with the database: Add New > C++ > C++ Class > Choose... > Class name – Database > Next > Project Management (default settings) > Finish. In
QStringheaders to work with images and strings and
libfacerec.hto integrate Face SDK.
videoframe.hheaders (implementation of the
IRawImageinterface, used by
VideoWorkerto receive the frames). Also, include necessary headers for working with the file system, debugging, exception handling, and working with files.
database.hadd a constructor and set the path to the database. Specify the
Recognizerobject to create templates, and specify the
Capturerobject to detect faces and
far. FAR is frequency that the system makes false accepts. False accept means that a system claims a pair of pictures is a match, when these are actually the pictures of different individuals. The
vw_elementsvector contains the elements of the
namesvectors contain the previews of images and names of people from the database.
Databaseconstructor, declared in the previous subsection. The
distance_thresholdvalue means the recognition distance. Since this distance is different for different recognition methods, we get it based on the
FARvalue using the
- Specify the path to the database with faces in the
database_dirvariable. If this path doesn't exist, you'll see the
"database directory doesn't exist"exception. Create a new
person_idvariable to store the id of a person from the database (folder name in the database) and the
element_idvariable to store the id of an element in the database (a person's image from the database). In the
dirslist, create a list of all subdirectories of the specified directory with the database.
Note: See more information about FAR and TAR values for different recognition methods in Identification Performance.
- In the
for(const auto &dir: dirs)loop, process each subdirectory (data about each person). A folder name corresponds to a person's name. Create a list of images in
- In the
for(const auto &person_file: person_files)nested loop, process each image. If an image doesn't exist, the
"Can't read image"warning is displayed.
- Detect a face in an image using the
Capturerobject. If an image cannot be read, a face cannot be found in an image or more than one face is detected, the warning is displayed and this image is ignored.
- Using the
recognizer->processingmethod, create a face template used for recognition.
- In the
pbio::VideoWorker::DatabaseElement vw_elementstructure, specify all the information about the database element that will be passed for processing to the
VideoWorkerobject (element id, person id, face template, recognition threshold). Using the
push_backmethod, add an element to the end of the list.
database.h, add the
makeThumbnailmethod to create a preview of a picture from the database.
database.cpp, implement the method using
makeThumbnailto create a preview of a picture from the database, which will be displayed next to the face of a recognized person. Set the preview size (120 pixels) and scale it (keep the ratio if the image is resized).
- In the .pro file, set the path to the database.
facesdkparameters.h, set the path to the database and the FAR value.
facesdkparameters.hset the path to the configuration file with the recognition method. In this project we use the method 6.7 because it is suitable for video stream processing and provides optimal recognition speed and good quality. You can learn more about recommended recognition methods in Face Identification.
Note: If you want to recognize faces in a video stream using low-performance devices, you can use the method 9.30. In this case recognition speed is higher but recognition quality is lower compared to the method 6.7.
match_database_indexvariable to the
FaceDatastructure. This variable will store the database element, if a person is recognized, or
"-1"if a person isn't recognized. Add
Databaseand a callback indicating that a person is recognized (
worker.cpp, override the parameter value in the configuration file so that
MatchFoundCallbackis received for non-recognized faces too. Set the parameters of the
VideoWorkerobject: in the first tutorial we didn't recognize faces, that's why we set only the value of
streams_count. Since in this project we'll recognize faces in a video stream, we need to specify in the constructor the path to the configuration file with the recognition method, and also the values of
processing_threads_count(number of threads to create templates) and
matching_threads_count(number of threads to compare the templates). In this project we use only one stream (a webcam connected to our PC). Connect the database: pass the path to the database, create
Capturerto detect faces and
Recognizerto create templates, and also specify the
FARcoefficient. Using the
setDatabasemethod, set the database for
VideoWorker. Using the
addMatchFoundCallbackmethod, add the
MatchFoundrecognition event handler.
- In the
MatchFoundCallback, the result is received in the form of the
MatchFoundCallbackDatastructure that stores the information about recognized and unrecognized faces.
- When a template for a tracked person is created, this template is compared with each template from the database. If the distance to the closest element is less than
distance_thresholdspecified in this element, then it is a match. If a face in a video stream is not recognized, then you'll see the message
"Match not found". If a face is recognized, you'll see the message
"Match found with..."and the name of the matched person.
- Save the data about the recognized face to display a preview.
- Run the project. The recognition results will be displayed in the console. If a face is recognized, you'll see the face id and name of a recognized person from the database. If a face isn't recognized, you'll see the message
"Match not found".
- Display the image and name of a person from the database next to the face in a video stream. In
drawfunction.h, add a reference to the database, because we will need it when rendering the recognition results.
DrawFunction::Drawfunction by passing the database to it.
- Save the bounding rectangle in the
pbio::RawSample::Rectanglestructure. Pass its parameters (x, y, width, height) to the
- Create a boolean variable
recognizedthat indicates whether a face is recognized or unrecognized. If a face is recognized, the bounding rectangle is green, otherwise it is red.
- Get a relevant image from the database for a preview by
face.match_database_index. Calculate the position of a preview in the frame.
- Draw an image from the database in the preview. Create the
QImage face_previewobject that is higher than
text_bar_height. The original preview image is drawn in the (0, 0) position. As a result, we get a preview with a black rectangle at the bottom with the name of a person. Set the font parameters, calculate the position of a text and display the text in the preview.
face_previewin the frame using the
worker.h, add a method that returns the reference to the database.
- Modify the call to
DrawFunction::Drawby passing the database to it.
- Run the project. If a face is recognized, it will be highlighted with a green rectangle, and you'll see a preview of an image from the database and a person's name. Unrecognized faces will be highlighted with a red rectangle.