In this tutorial you'll learn how to detect and track faces in a video stream from your camera using the
VideoWorker object from Face SDK API. Tracked faces are highlighted with a green rectangle.
Besides Face SDK and Qt, you'll need a camera connected to your PC (for example, a webcam). You can build and run this project either on Windows or Ubuntu (v16.04 or higher).
Find the tutorial project in Face SDK: examples/tutorials/detection_and_tracking_with_video_worker
- Run Qt and create a new project: File > New File or Project > Application > Qt Widgets Application > Choose...
- Name it, for example, 1_detection_and_tracking_with_video_worker and choose the path. Click Next and choose the necessary platform for your project in the Kit Selection section, for example, Desktop. Click Details and select the Release build configuration ( Debug is not required in this project).
- Leave settings as default in the Class Information window and click Next. Then leave settings as default in the Project Management window and click Finish.
- Title the main window of our application: double-click the file Forms > mainwindow.ui in the project tree. Specify the window name in the Properties tab (the right part of the editor): windowTitle > Face SDK Tracking.
- To lay out widgets in a grid, drag-and-drop the Grid Layout object to the MainWindow widget. Call context menu of MainWindow by right-clicking and select Layout > Lay Out in a Grid. The Grid Layout object will be stretched to the size of the MainWindow widget. Rename the Layout: layoutName > viewLayout.
- To run the project, click Run (Ctrl+R). You'll see an empty window with Face SDK Tracking title.
- To use a camera in our project, add Qt multimedia widgets. To do this, add the following line to the .pro file:
- To receive the image from a camera, create a new class
QCameraCapture: Add New > C++ > C++ Class > Choose… > Class name – QCameraCapture > Base class – QObject > Next > Project Management (default settings) > Finish. Create a new class
qcameracapture.h, which will provide the frames from camera via the
- Describe the implementation of this class in
qcameracapture.cppfile. Designate the
CameraSurface::CameraSurfaceconstructor and the
supportedPixelFormatsmethod. All the image formats in
CameraSurface::supportedPixelFormatslist are supported by Face SDK (RGB24, BGR24, NV12, NV21). With some cameras the image is received in the RGB32 format, so we add this format to the list. This format is not supported by Face SDK, so convert the image from RGB32 to RGB24.
- Check the image format in the
CameraSurface::startmethod. If the format is supported, start the camera. Otherwise, handle the exception.
- Process a new frame in the
CameraSurface::presentmethod. If the frame is successfully verified, send the signal
frameUpdatedSignalto update the frame. Next, connect this signal to the
frameUpdatedSlotslot, where the frame will be processed.
QCameraCaptureconstructor takes the pointer to a parent widget (
parent), camera id and image resolution (width and height), which will be stored in the relevant class fields.
m_surfacecamera objects to the
- To throw exceptions, include the
stdexceptheader file to
qcameracapture.cpp. Save the pointer to a parent widget, camera id and image resolution in the initializer list of the
QCameraCapture::QCameraCaptureconstructor. Get the list of available cameras in the constructor body. The list of cameras should contain at least one camera, otherwise, the
runtime_errorexception will be thrown. Make sure that the list contains a camera with the requested id. Create a camera and connect the camera signals to the slots processing the object. When the camera status changes, the camera sends the
statusChangedsignal. Create the
CameraSurfaceobject to display the frames from the camera. Connect the
CameraSurface::frameUpdatedSignalsignal to the
- Stop the camera in the
- Add the
QCameraCapture::frameUpdatedSlotmethod, which processes the
CameraSurface::frameUpdatedSignalsignal. Using this method convert the
QImageand send a signal that a new frame is available. Create a pointer to the
FramePtrimage. If the image is received in the RGB32 format, convert it to RGB888.
- Add the methods to start and stop the camera to
- In the
QCameraCapture::onStatusChangedmethod process the change of the camera status to
LoadedStatus. Check if the camera supports the requested resolution. Set the requested resolution, if it is supported by the camera. Otherwise, set the default resolution (640 x 480), specified by the
- Display the camera error messages in the
cameraErrormethod if occurred.
- Create a new
Workerclass: Add New > C++ > C++ Class > Choose… > Class name - Worker > Next > Finish. Through the
Workerclass will save the last frame from the camera and pass this frame through the
- Frames will be displayed in the
ViewWindowclass. Create a ViewWindow widget: Add > New > Qt > Designer Form Class > Choose... > Template > Widget (default settings) > Next > Name – ViewWindow > Project Management (default settings) > Finish.
- In the editor (Design) drag-and-drop the Grid Layout object to the widget. To do this, call ViewWindow context menu by right-clicking and select Layout > Lay Out in a Grid. The Grid Layout object lets you place widgets in a grid and is stretched to the size of the ViewWindow widget. Then add the Label object to gridLayout and name it frame: QObject > objectName > frame.
- Delete the default text in QLabel > text.
- Add the
_qCameracamera to the
ViewWindowclass and initialize it in the constructor. Using the
camera_image_heightstatic fields, set the required image resolution to 1280x720. The
_runningflag stores the camera status:
truemeans that the camera runs,
falsemeans that the camera is stopped.
- Add the
Workerobject to the
ViewWindowclass and initialize it in the constructor.
- Frames will be passed to
QCameraCapture. Modify the
QCameraCapture::newFrameAvailablesignal is processed in the
ViewWindow::drawslot, which displays the camera image on the frame widget.
- Start the camera in the
runProcessingmethod and stop it in
- Stop the camera in the
- Connect the camera widget to the main application window: create a view window and start processing in the
MainWindowconstructor. Stop the processing in the
- Modify the
mainfunction to catch possible exceptions.
- Run the project. You'll see a window with the image from your camera.
Note: On Windows the image from some cameras can be flipped or mirrored, due to some peculiarities of the image processing by Qt. In this case process the image, for example, using QImage::mirrored().
- Download and extract Face SDK distribution as described in the section Getting Started. The distribution root folder contains the bin and lib folders, depending on your platform.
- To detect and track faces on the image from your camera, integrate Face SDK into your project. In the .pro file, specify the path to Face SDK root folder in the variable
FACE_SDK_PATH, which includes necessary headers. Also, specify the path to the
includefolder (from Face SDK). If the paths are not specified, the exception “Empty path to Face SDK” will be thrown.
Note: When you specify the path to Face SDK, please use a slash ("/").
- [Linux only] To build the project with Face SDK, add the following option to the .pro file:
- Besides, specify the path to the
facereclibrary and configuration files. Create the
FaceSdkParametersclass, which will store the configuration (Add New > C++ > C++ Header File > FaceSdkParameters) and use it in
- Integrate Face SDK: add necessary headers to
initFaceSdkServicemethod to initialize Face SDK services. Create a
FacerecServiceobject, which is a component used to create Face SDK modules, by calling the
FacerecService::createServicestatic method. Pass the path to the library and path to the folder with the configuration files in a
try-catchblock to catch possible exceptions. If the initialization is successful, the
initFaceSdkServicefunction will return
true. Otherwise, it will return
falseand you'll see a window with an exception.
- Add a service initialization call in the
MainWindow::MainWindowconstructor. In case of an error, throw the
FacerecServiceand Face SDK parameters to the
ViewWindowconstructor, where they will be used to create the
VideoWorkertracking module. Save the service and parameters to the class fields.
- Modify the
Workerclass for interaction with Face SDK. The
Workerclass takes the
FacerecServicepointer and name of the configuration file of the tracking module. The
Workerclass creates the
VideoWorkercomponent from Face SDK, responsible for face tracking, passes the frames to it and processes the callbacks, which contain the tracking results. Imlement the constructor – create the
VideoWorkerobject, specifying the configuration file, recognizer method (in this case it is empty, as faces are not to be recognized in this project), number of video streams (in this case it is 1, as we use one camera only).
Note: In addition to the face detection and tracking,
VideoWorker can be used for face recognition on several video streams. In this case specify the recognizer method and the
- Subscribe to the callbacks from the
TrackingCallback(a face is detected and tracked),
TrackingLostCallback(a face is lost). Delete them in the destructor.
- Include the
cassertheader to handle exceptions. The result in
TrackingCallbackis received in the form of the
TrackingCallbackDatastructure, which stores data about all faces being tracked. The preview output is synchronized with the result output. We cannot immediately display the frame, passed to
VideoWorker, as it will be processed a bit later. Therefore, frames are stored in a queue. When the result is obtained, find a frame that matches this result. Some frames may be skipped by
VideoWorkerunder heavy load, which means that sometimes there is no matching result for some frames. In the algorithm below, the image corresponding to the last received frame is extracted from the queue. Save the detected faces for each frame to be able to use them later for visualization. To synchronize the shared data changes in
TrackingLostCallback, where we mark that the tracked face left the frame.
VideoWorkerreceives the frames via the
pbio::IRawImageinterface. Create the
VideoFrameheader file: Add New > C++ > C++ Header File > VideoFrame. Include it into the
videoframe.hfile and implement the
pbio::IRawImageinterface for the
pbio::IRawImageinterface allows to get the pointer to image data, its format, width and height.
- In the
addFramemethod, pass the frames to
VideoWorker. Any exceptions occurred during the callback processing are thrown again in the
checkExceptionsmethod. Create the
_framesqueue to store the frames. This queue will contain the frame id and the corresponding image, so that we can find the frame, which matches the processing result in
TrackingCallback. To synchronize the changes in shared data, use
- Modify the
getDataToDrawmethod - do not draw the faces, which
TrackingLostCallbackwas called for.
- Modify the
QCameraCaptureclass to catch the exceptions, which can be thrown in
- Create the
DrawFunctionclass, which will contain a method to draw the tracking results in the image: Add New > C++ > C++ Class > Choose… > Class name – DrawFunction.
- In the
ViewWindowconstructor, pass the
FacerecServicepointer and the name of the configuration file of the tracking module when creating
Worker. In the
Drawmethod, draw the tracking result in the image by calling
- Run the project. Now you can see that faces in the image are detected and tracked (they are highlighted with a green rectangle). You can find more information about using the
VideoWorkerobject in Video Stream Processing section.