CV Task

Computer Vision (CV) Tasks, a.k.a Vision Tasks, focus on analysing and understanding the content of visual data in the same way as human visual system does. The goal is to make a computer/device provide description for the data as complete and accurate as possible. Some classic CV Tasks include image classification, object detection, image segmentation and keypoint detection. These primitive tasks are the foundation for building many real-world industrial vision applications.

Intrigued? Refer to Prepare Models to Learn about how to prepare your models for CV Tasks supported by VDP.

#Standardise CV Tasks

In a data pipeline, model is the core component designed to solve a specific CV Task. By standardising the data format of model outputs into CV Tasks,

  • model in a pipeline is modularized: you can freely switch to use different models in a pipeline as long as the model is designed for the same CV Task;
  • VDP produces a stream of data from models with standard format for use in a data integration or ETL pipeline.

At the moment, VDP defines the data interface for popular CV Tasks:

  • Image classification - classify images into predefined categories
  • Object detection - detect and localise multiple objects in images
  • Keypoint detection - detect and localise multiple keypoints of objects in images
  • OCR (Optical Character Recognition) - detect and recognise text in images
  • The list is growing ... 🌱

Each CV Task is described in depth in the respective section below.

If you'd like to ask for a new model definition, you can create a topic in Discussions, or request it in the #vdp channel on Discord.

#How to standardise

#Standardise via Protocol Buffers

Currently, the model output is converted to standard format based on the CV Task outputs maintained in Protobuf.

#Standardise via VDP Protocol

The VDP Protocol describes the data schema of model outputs in order to standardise an ETL pipeline for visual data. The data produced by the model component and passed to destination component of a pipeline is done via serialized JSON messages for inter-process communication.

The protocol is still under development. Stay tuned on how the protocol will evolve.

The batched outputs of a model for input images is wrapped in a BatchOutputs. It includes an array of TaskOutput, each of which corresponds to the model output of one input image in the batch. The TaskOutput struct has a required task which describes the CV Task type of the wrapped model output. Based on the task, the corresponding field will be populated, other fields will be null.


ModelInstanceOutput:
type: object
additionalProperties: true
required:
- task
- batch_outputs
properties:
task:
description: "Task type"
type: string
enum:
- TASK_CLASSIFICATION
- TASK_DETECTION
- TASK_KEYPOINT
- TASK_OCR
- TASK_UNSPECIFIED
batch_outputs:
description: "a list of inference output for a specific CV Task"
type: array
items:
"$ref": "#/definitions/TaskOutput"
TaskOutput:
type: object
additionalProperties: true
anyOf:
- required:
- classification
- required:
- detection
- required:
- keypoint
- required:
- ocr
- required:
- unspecified
properties:
classification:
description: "Classify into pre-defined categories"
"$ref": "#/definitions/Classification"
detection:
description: "Detect and localise multiple objects"
"$ref": "#/definitions/Detection"
keypoint:
description: "Detect and localise keypoints of multiple objects"
"$ref": "#/definitions/Keypoint"
ocr:
description: "Detect, localise and recognise texts"
"$ref": "#/definitions/Ocr"
unspecified:
description: "Unspecified task with output in the free form"
"$ref": "#/definitions/Unspecified"

#Image classification

Image classification task is the CV Task to assign a single pre-defined category label to an entire input image. Generally, an image classification model takes an image as the input, and outputs a prediction about what category this image belongs to and a confidence score (usually between 0 and 1) representing the likelihood that the prediction is correct.

Image classification task


{
"task": "TASK_CLASSIFICATION",
"batch_outputs": [
{
"classification": {
"category": "golden retriever",
"score": 0.98
}
}
]
}

#Object detection

Object detection task is the CV Task to localise multiple objects of pre-defined categories in an input image. Generally, an object detection model receives an image as the input, and outputs bounding boxes with category labels and confidence scores on detected objects.

Object detection task


{
"task": "TASK_DETECTION",
"batch_outputs": [
{
"detection": {
"bounding_boxes": [
{
"bounding_box": {
"left": 324,
"top": 102,
"width": 208,
"height": 405,
},
"category": "dog",
"score": 0.97
},
...
]
}
}
]
}

#Keypoint detection

Keypoint detection task is the CV Task to localise multiple objects by identifying their pre-defined keypoints, for example, identifying the keypoints of human body: nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles. Normally, a keypoint detection task takes an image as the input, and outputs the coordinates and visibility of keypoints with confidence scores on detected objects.

Keypoint detection task


{
"task": "TASK_KEYPOINT",
"batch_outputs": [
{
"keypoint": {
"keypoint_groups": [
{
"keypoint_group": [
{
"v": 0.53722847,
"x": 542.82764,
"y": 86.63817
},
{
"v": 0.634061,
"x": 553.0073,
"y": 79.440636
},
...
],
"score": 0.94
}
]
}
}
]
}

#Optical Character Recognition (OCR)

OCR task is the CV Task to localise and recognise text in an input image. The task can be done in two steps by multiple models: a text detection model to detect bounding boxes containing text and a text recognition model to process typed or handwritten text within each bounding box into machine readable text. Alternatively, there are deep learning models that can accomplish the task in one single step.

OCR task


{
"task": "TASK_OCR",
"batch_outputs": [
{
"ocr": {
"bounding_boxes": [
{
"top": 298,
"left": 279,
"width": 134,
"height": 59
},
{
"top": 228,
"left": 216,
"width": 255,
"height": 65
}
],
"texts": ["ENDS", "PAVEMENT"]
}
}
]
}

#What if my task is not standardised by by VDP yet?

VDP is very flexible and allows you to import models even if your task is not standardised yet or the output of the model can't be converted to the format of supported CV Tasks. The model will be classified as an Unspecified CV Task. Send an image to the model as the input, VDP will

  • check the config.pbtxt model configuration file to extract the output names, datatypes and shapes of the model outputs,
  • and wrap these information along with the raw model output in a standard format.

Unspecified task


{
"task": "TASK_UNSPECIFIED",
"batch_outputs": [
{
"unspecified": {
"raw_outputs": [
{
"data": [0.85, 0.1, 0.05],
"data_type": "FP32",
"name": "output_scores",
"shape": [3]
},
{
"data": ["dog", "cat", "rabbit"],
"data_type": "BYTES",
"name": "output_labels",
"shape": [3]
}
]
}
}
]
}

Last updated: 9/13/2022, 2:24:43 PM