Processing Block API

The Processing Block API is an alternative, scalable interface that replaces existing APIs for easier integration of SDK capabilities into your application.

Key features of Processing Block API:

• Combining many components into a single integration
• Simplicity and ease of learning
• Rapid implementation
• Long term support and updates

The Processing Block API is a part of the upcoming 3DiVi solutions. For more details, contact your 3DiVi Sales representative.

Set of Processing Blocks

note

An example of using Processing Block API is Processing Block Demo Sample.

Face Detector

Face Detector is a component used to detect human faces on an image. The detection result is a bounding rectangle (a frame) around the detected face.

Timing characteristics of Face Detector:

• CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~115ms
• GPU (GeForce GTX 1080 Ti) ~16ms

You can find the information about using Face Detector on the page Face Detection.

Human Body Detector

Human Body Detector is a component used to detect human bodies on an image. The detection result is a bounding rectangle (a frame) around the detected body.

Timing characteristics of Human Body Detector:

• CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~170ms
• GPU (GeForce GTX 1080 Ti) ~16ms

You can find the information about using Human Body Detector on the page Body Detection.

Object Detector

Object Detector is a component used to detect multiple objects on an image. The detection result is a bounding rectangle (a frame) around the detected object with classification name. Object Detector is used to detect the following classes of objects on an image: "body", "bicycle", "car", "motorcycle", "bus", "train", "truck", "traffic_light", "fire_hydrant", "stop_sign", "bird", "cat", "dog", "horse", "sheep", "cow", "bear", "backpack", "umbrella", "handbag", "suitcase", "sports_ball", "baseball_bat", "skateboard", "tennis_racket", "bottle", "wine_glass", "cup", "fork", "knife", "laptop", "phone", "book", "scissors".

Timing characteristics of Object Detector:

• CPU (Intel Core i5-11400H @ 2.70GHz, 1 core) ~1440ms
• GPU (GeForce GTX 1080 Ti) ~35,6ms

You can find the information about using Object Detector, and a list of objects for detection on the page Object Detection.

Human Pose Estimator

Human Pose Estimator is a component used to estimate human body skeleton keypoints on an image. The detection result is a list of keypoints with their coordinates and confidence score of the detected human body. Human Pose Estimator is used to detect the following keypoints on an image: "nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle"

Timing characteristics of Human Pose Estimator:

• CPU (Intel Core i5-11400H @ 2.70GHz, 1 core) ~55ms
• GPU (GeForce GTX 1080 Ti) ~15ms

You can find the information about using Human Pose Estimator on the page Human Pose Estimator.

Emotion Estimator

Emotion Estimator is a component used to estimate human emotions from the cropped face image. The estimator result is a confidence for every emotion estimated.

Timing characteristics of Emotion Estimator:

• CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~18ms
• GPU (GeForce GTX 1080 Ti) ~8ms

You can find the information about using Emotion Estimator on the page Emotion Estimation.

Age Estimator

Age Estimator is a component used to estimate a human age from the cropped image with a face. The estimator result is a human age.

Timing characteristics of Age Estimator:

• CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~5ms
• GPU (GeForce GTX 1080 Ti) ~1ms

You can find the information about using Age Estimator on the page Age Estimation.

Gender Estimator

Gender Estimator is a component used to estimate a human gender from the cropped image with a face. The estimator result is a verdict about gender identity.

Timing characteristics of Gender Estimator:

• CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~5ms
• GPU (GeForce GTX 1080 Ti) ~1ms

You can find the information about using Gender Estimator on the page Gender Estimation.

2D RGB Liveness Estimator

2D RGB Liveness Estimator is a component used to detect human liveness on a single colored image. The detection result is a bounding rectangle (a frame) around the detected human face with liveness verdict and score. You can find the information about using Gender Estimator on the page Liveness Estimation.

Quality Assessment Estimator

Quality Assessment Estimator is a component used to assess the quality of a face in an image for identification tasks on a single colored image. The detection result is a list of the detected human faces with a verbose quality score. You can find the information about using Quality Assessment Estimator on the page Quality Assessment Estimation.

Context

Processing Block API is based on the use of Context.

Context is a heterogeneous container that consists of a set of hierarchically organized data presented in the form of key–value pairs. The closest analogue of Context is a JSON object. Each Context object can contain a scalar object (integer, real, boolean, string), a memory area or pointer, a sequential array of Context objects, or an associative container of string-Context pairs, with unlimited nesting.

How to create and use a Context object

1. Create a FacerecService.

2. Then create a Context container:

auto array_elem0 = service->createContext();

3. Common set of operations with a Context container:

• creating an associative container by calling ["key"] on empty Context:

array_elem0["name"] = "Julius Zeleny"; // put string

array_elem0["phone"] = 11111111111l; // put integer (long)

array_elem0["social_score"] = 0.999; // put double

array_elem0["verified"] = true; // put bool

• getters:

ASSERT_EQ( array_elem0["name"].getString(), "Julius Zeleny" );

ASSERT_EQ( array_elem0["phone"].getLong(), 11111111111l );

ASSERT_EQ( array_elem0["social_score"].getDouble(), 0.999 );

ASSERT_EQ( array_elem0["verified"].getBool(), true );

• creating a sequence array by calling push_back on empty Context:

auto service->createContext();

array.push_back(array_elem0);

• iterating over array:

// get by index

ASSERT_EQ( array[0]["phone"].getLong(), 11111111111l );

// iterate with index

size_t array_sz = array.size();

for(size_t i = 0; i < array_sz; ++i)

array[i]["phone"];

// or with iterators

for(auto iter = array.begin(); iter != array.end(); ++iter)

(*iter)["phone"]; // deference returns nested Context

// with foreach

for(auto val : array)

val["phone"];

• operations with a nested associative container:

auto full = service->createContext();

full["friends"] = std::move(array); // move assignment without copying

// access nested object

ASSERT_EQ( full["friends"][0]["social_score"].getDouble(), 0.999 );

// iterate over associative containers values

for(auto iter = full.begin(); iter != full.end(); ++iter) {

iter.key(); // get key value from iterator

(*iter)[0]["social_score"].getDouble(); // get value

}

// with foreach

for(auto val : full)

val[0]["social_score"].getDouble();

• other Context's convenient methods:

void clear()

bool contains(const std::string& key) // for an assosiative container

Context operator[](size_t index) // for a sequence array, access specified element with bounds checking

Context operator[](const std::string& key) // for an assosiative container, access or insert

Context at(const std::string& key) // for an assosiative container, with bounds checking

size_t size() // return elements count for a container

bool isNone() // is empty

bool isArray() // is a sequence array

bool isObject() // is an assosiative container

bool isLong(), isDouble(), isString(), isBool() // check if contains a certain scalar type

Binary Image Format

Most of the processing blocks operates on Context with an image in binary format:

/*

{

"image" : { "format": "NDARRAY",

"blob": <data pointer>,

"dtype": "uint8_t",

"shape": [height, width, channels] }

}

*/

The "blob" key contains a smart pointer to data. The pointer is set by the function void Context::setDataPtr(void* ptr, int copy_sz), where copy_sz is the size of memory in Bytes, that will be copied, and then automatically released when Context object's lifetime ends. Copying will not perform if 0 is passed as argument copy_sz. In this case the Context object does not control the lifetime of the object it points to.

You can also allocate a raw memory, f.e. to copy data later, passing nullptr and size as arguments of setDataPtr.

The "dtype" can contain one of these values: "uint8_t", "int8_t", "uint16_t", "int16_t", "int32_t", "float", "double". This is according to OpenCV types: CV_8U, CV_8S, CV_16U, CV_16S, CV_32S, CV_32F, CV_64F.

Creating a Context container with RGB-image

1. Create a FacerecService.

2. Create a Context container for image using the createContext() method:

auto imgCtx = service->createContext();

4. Read an RGB-image from file:

// read the image from file

std::string input_image_path = "<path_to_image>";

cv::Mat image = cv::imread(input_image_path, cv::IMREAD_COLOR);

cv::Mat input_image;

cv::cvtColor(image, input_image, cv::COLOR_BGR2RGB);

5. a. Put an image into container OR

// using pbio::context_utils::putImage(Context& ctx, unsigned char* data, size_t height, size_t width, pbio::IRawImage::Format format, bool copy)

pbio::context_utils::putImage(imgCtx, input_image.data, input_image.rows, input_image.cols, pbio::IRawImage::FORMAT_RGB, true);

5. b. OR copy an image from pbio::RawImage, pbio::CVRawImage, pbio::InternalImageBuffer to binary format and put it to Context container:

// constructing pbio::RawImage

pbio::RawImage input_rawimg(input_image.cols, input_image.rows, pbio::RawImage::Format::FORMAT_RGB, input_image.data);

// using void putImage(Context& ctx, const RawImage& raw_image)

pbio::context_utils::putImage(imgCtx, input_rawimg);

Creating a Processing Block

This template can be used to create any processing block. Keys unit_type and model_path must be specified according to block you want to use (see description of specific processing block).

1. Create a FacerecService.

2. Create a Context container:

auto configCtx = service->createContext();

3. Define fields in the created context container for creating a Processing Block:

// mandatory, specify the name of processing block

configCtx["unit_type"] = "<name_of_processing_block>";

// mandatory, specify the path to model file of processing block

configCtx["model_path"] = "<path_to_model_file>";

// default location of the onnxruntime library in <ShortProductName /> distributive: the "lib" folder for Linux platfrom or the "bin" folder for Windows platfrom

// you can specify your own path to onnxruntime library

// if value is not specified, the os-specific default search order will be used

configCtx["ONNXRuntime"]["library_path"] = "../lib";

// optional, "true" if you want to use GPU acceleration (CUDA) for processing block that support it

configCtx["use_cuda"] = false;

4. Create a Processing Block:

pbio::ProcessingBlock processing_block = service->createProcessingBlock(configCtx);

GPU Acceleration

Processing Blocks can be used with GPU acceleration (CUDA). To activate acceleration you need to define the ["use_cuda"] key with the true value for Processing Block configuration container (see Creating a Processing Block).

Requirements

• Software requirements:
  • Nvidia GPU Driver >= 410.48
  • CUDA Toolkit 10.1
  • cuDNN 7
  • (For Windows) Microsoft Visual C++ Redistributable for Visual Studio 2019
• Hardware requirements:
  • CUDA compatible GPU (NVIDIA GTX 1050 Ti or better)