Face Recognition in a Video Stream
In this tutorial, you'll learn how to recognize faces in a video stream. For recognition, you can use a ready-made database of faces from the Face SDK distribution package. The database includes the images of several famous people. Recognized faces are highlighted with a green rectangle. The name and image of a recognized person are displayed next to his/her face in a video stream. This tutorial is based on Face Detection and Tracking in a Video Stream and the corresponding project.
You can find the tutorial project in Face SDK: examples/tutorials/face_recognition_with_video_worker
#
Setting Up the Project- In Face Detection and Tracking in a Video Stream, we set only two parameters of Face SDK (a path to Face SDK and a configuration file name for the
VideoWorker
object). However, in this tutorial, we need to set several more parameters: we will add a path to the database, a configuration file name with a recognition method, and FAR. For convenience, we'll modify several files. Specify all the parameters in theFaceSdkParameters
structure. Infacesdkparameters.h
, specify the path to thevideo_worker_lbf.xml
configuration file.
facesdkparameters.h
- Pass the
face_sdk_parameters
structure to the constructor of theWorker
object.
viewwindow.h
viewwindow.cpp
worker.h
worker.cpp
- In this project, we're interested only in face detection in a video stream (creating a bounding rectangle) and face recognition. Please note that in the first project (
detection_and_tracking_with_video_worker
), which you can use as a reference for this project, we also displayed anthropometric points and angles. If you don't want to display this info, you can just remove unnecessary visualization from the first project.
#
Creating the Database of FacesFirst of all, we have to create a database of faces. To check face recognition, you can use the ready-made database from Face SDK. It includes images of three famous people (Elon Musk, Emilia Clarke, Lionel Messi). To check recognition, you should copy the database to the project root folder (next to a .pro file), run the project, open an image from the database, and point a camera at the screen. You can also add your picture to the database. To do this, you have to create a new folder in the database, specify your name in a folder name, and copy your picture to the folder (in the same way as other folders in the database).
Create a new
Database
class to work with the database: Add New > C++ > C++ Class > Choose... > Class name โ Database > Next > Project Management (default settings) > Finish. Indatabase.h
, include the headersQImage
andQString
to work with images and strings andlibfacerec.h
to integrate Face SDK.
database.h
- In
database.cpp
, include the headersdatabase.h
andvideoframe.h
(implementation of theIRawImage
interface, which is used byVideoWorker
to receive the frames). Also include necessary headers for working with the file system, debugging, exception handling, and working with files.
database.cpp
- In
database.h
, add a constructor and set the path to the database. Specify theRecognizer
object to create templates, theCapturer
object to detect faces andfar
. What is FAR? FAR is frequency that the system makes false accepts. False accept means that a system claims a pair of pictures are a match, when they are actually pictures of different individuals. Thevw_elements
vector contains the elements of theVideoWorker
database. Thethumbnails
andnames
vectors contain the previews of images and names of people from the database.
database.h
- In
database.cpp
, implement theDatabase
constructor, which was declared in the previous subsection. Thedistance_threshold
value means the recognition distance. Since this distance is different for different recognition methods, we get it based on theFAR
value using thegetROCCurvePointByFAR
method.
database.cpp
- In the
database_dir
variable, specify the path to the database with faces. If this path doesn't exist, you'll see the exception"database directory doesn't exist"
. Create a newperson_id
variable to store the id of a person from the database (name of a folder in the database) and theelement_id
variable to store the id of an element in the database (an image of a person from the database). In thedirs
list, create a list of all subdirectories of the specified directory with the database.
database.cpp
Note: See more information about FAR and TAR values for different recognition methods in Identification Performance.
- In the loop
for(const auto &dir: dirs)
, process each subdirectory (data about each person). The name of a folder corresponds to the name of a person. Create a list of images inperson_files
.
database.cpp
- In the nested loop
for(const auto &person_file: person_files)
, process each image. If an image doesn't exist, the warning"Can't read image"
is displayed.
database.cpp
- Detect a face in an image using the
Capturer
object. If an image cannot be read, a face can't be found in an image or more than one face is detected, the warning is displayed and this image is ignored.
database.cpp
- Using the
recognizer->processing
method, create a face template, which is used for recognition.
database.cpp
- In the structure
pbio::VideoWorker::DatabaseElement vw_element
, specify all the information about the database element that will be passed for processing to theVideoWorker
object (element id, person id, face template, recognition threshold). Using thepush_back
method, add an element to the end of the list.
database.cpp
- In
database.h
, add themakeThumbnail
method to create a preview of a picture from the database.
database.cpp
- In
database.cpp
, implement the method usingmakeThumbnail
to create a preview of a picture from the database, which will be displayed next to the face of a recognized person. Set the preview size (120 pixels) and scale it (keep the ratio if the image is resized).
database.cpp
- In the .pro file, set the path to the database.
face_recognition_with_video_worker.pro
- In
facesdkparameters.h
, set the path to the database and the value of FAR.
facesdkparameters.h
#
Searching a Face in the Database and Displaying the Result- In
facesdkparameters.h
, set the path to the configuration file with the recognition method. In this project, we use the method 6.7 because it's suitable for video stream processing and provides optimal recognition speed and good quality. You can learn more about recommended recognition methods in Face Identification.
facesdkparameters.h
Note: If you want to recognize faces in a video stream and you use low-performance devices, you can use the method 9.30. In this case, recognition speed is higher but recognition quality is lower compared to the method 6.7.
- In
worker.h
, add the variablematch_database_index
to theFaceData
structure. This variable will store the database element, if a person is recognized, or"-1"
if a person is not recognized. AddDatabase
and a callback indicating that a person is recognized (MatchFoundCallback
).
worker.h
- In
worker.cpp
, override the value of the parameter in the configuration file so thatMatchFoundCallback
is received for non-recognized faces too. Set the parameters of theVideoWorker
object: in the first tutorial, we didn't recognize faces, that's why we set only the value ofstreams_count
. Since in this project we're going to recognize faces in a video stream we have to specify in the constructor the path to the configuration file with the recognition method, and also the values ofprocessing_threads_count
(number of threads to create templates) andmatching_threads_count
(number of threads to compare the templates). In this project, we use only one stream (a webcam connected to our PC). Connect the database: pass the path to the database, createCapturer
to detect faces andRecognizer
to create templates, and also specify theFAR
coefficient. Using thesetDatabase
method, set the database forVideoWorker
. Using theaddMatchFoundCallback
method, add the recognition event handlerMatchFound
.
worker.cpp
- In the destructor
Worker::~Worker()
, removeMatchFoundCallback
.
worker.cpp
- In
MatchFoundCallback
, the result is received in form of the structureMatchFoundCallbackData
that stores the information about recognized and unrecognized faces.
worker.cpp
- When a template for a tracked person is created, it's compared with each template from the database. If the distance to the closest element is less than
distance_threshold
specified in this element, then it's a match. If a face in a video stream is not recognized, then you'll see the message"Match not found"
. If a face is recognized, you'll see the message"Match found with..."
and the name of the matched person.
worker.cpp
- Save the data about the recognized face to display a preview.
worker.cpp
- Run the project. The recognition results will be displayed in the console. If a face is recognized, you'll see the face id and name of a recognized person from the database. If a face isn't recognized, you'll see the message
"Match not found"
.
#
Displaying the Preview of the Recognized Face from the Database- Let's make our project a little nicer. We'll display the image and name of a person from the database next to the face in a video stream. In
drawfunction.h
, add a reference to the database, because we'll need it when rendering the recognition results.
drawfunction.h
- In
drawfunction.cpp
, modify the functionDrawFunction::Draw
by passing the database to it.
drawfunction.cpp
- Save the bounding rectangle in the structure
pbio::RawSample::Rectangle
. Pass its parameters (x, y, width, height) to theQRect rect
object.
drawfunction.cpp
- Create a boolean variable
recognized
that indicates whether a face is recognized or unrecognized. If a face is recognized, the bounding rectangle is green, otherwise it's red.
drawfunction.cpp
- Get a relevant image from the database for a preview by
face.match_database_index
. Calculate the position of a preview in the frame.
drawfunction.cpp
- Draw an image from the database in the preview. Create the object
QImage face_preview
that is higher thanthumbnail
ontext_bar_height
. The original preview image is drawn in the position (0, 0). As a result, we get a preview with a black rectangle at the bottom with the name of a person. Set the font parameters, calculate the position of a text and display the text in the preview.
drawfunction.cpp
- Draw
face_preview
in the frame using thedrawPixmap
method.
drawfunction.cpp
- In
worker.h
, add a method that returns the reference to the database.
worker.h
- Modify the call to
DrawFunction::Draw
by passing the database to it.
viewwindow.cpp
- Run the project. If a face is recognized, it will be highlighted with a green rectangle and you'll see a preview of an image from the database and a person's name. Unrecognized faces will be highlighted with a red rectangle.