In the previous tutorial VDP 101 [4/7] How to trigger a SYNC pipeline?, you have learned how to trigger and send data to the pipeline. However, we haven't talked about how to process it.
In the SYNC mode, VDP returns the triggering pipeline results according to the protocol (i.e., HTTP/gRPC) used for triggering the pipeline. In this tutorial, we will use the HTTP protocol as an example to demonstrate how to parse the responses from a SYNC pipeline.
#How to process pipeline responses
Let's go back to our code in sync-http-base64.py. After triggering a SYNC pipeline, the function will return a response from your triggered pipeline.
# Post HTTP request to the SYNC pipelineresp = trigger_pipeline_base64(api_gateway_url, opt.pipeline_id, img_string)# Parse results from the SYNC pipelineboxes_ltwh, categories, scores = parse_detection_response(resp)
Here, the pipeline responds to our HTTP POST request with resp: request.Response, which is then passed to a user-defined parser parse_detection_response(resp) to extract object detection results.
In the parser, you extract data structs from the standardised response for easy access using the one-line code below.
# Parse JSON into an object with attributes corresponding to dict keys.r = json.loads(resp.text, object_hook=lambda d: SimpleNamespace(**d))
#Pipeline response in SYNC mode
To better understand what is in the resp, let's have a closer look at resp.text, the content of the SYNC mode response in JSON format:
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
data_mapping_indices: is the pipeline trigger operation ID. You probably don't need the IDs in the SYNC mode, as you receive responses right after you trigger pipelines.
However, in the ASYNC and PULL modes, where pipelines do not return with model inference results immediately, users need the IDs to locate corresponding results in the destination (e.g., database).
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
model_instance_outputs: is a list of outputs generated by triggering a pipeline with a batch of data.
Taking object detection as an example, if you pass N images to the pipeline, it will return a list consisting of N outputs.
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
model_instance is the model instance component you used when creating the pipeline.
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
task indicates the AI task model instance designed to solve. Please read the following section about what AI tasks are supported by VDP.
Note that when importing a model, users need to specify the corresponding AI task in the model card.
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
task_outputs: is a list of corresponding task outputs given the input data.
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
index: is the same as the data_mapping_indices received when triggering pipelines. Users can use it to match trigger queries and task outputs.
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
detection: indicates the Object Detection task outputs: a list of detection objects. Read more about the task here
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
To better understand what is in the resp, let's have a closer look at resp.text, the content of the SYNC mode response in JSON format:
data_mapping_indices: is the pipeline trigger operation ID. You probably don't need the IDs in the SYNC mode, as you receive responses right after you trigger pipelines.
However, in the ASYNC and PULL modes, where pipelines do not return with model inference results immediately, users need the IDs to locate corresponding results in the destination (e.g., database).
model_instance_outputs: is a list of outputs generated by triggering a pipeline with a batch of data.
Taking object detection as an example, if you pass N images to the pipeline, it will return a list consisting of N outputs.
model_instance is the model instance component you used when creating the pipeline.
task indicates the AI task model instance designed to solve. Please read the following section about what AI tasks are supported by VDP.
Note that when importing a model, users need to specify the corresponding AI task in the model card.
task_outputs: is a list of corresponding task outputs given the input data.
index: is the same as the data_mapping_indices received when triggering pipelines. Users can use it to match trigger queries and task outputs.
detection: indicates the Object Detection task outputs: a list of detection objects. Read more about the task here
## This is the object returned in SYNC mode.{ "data_mapping_indices":[ "01GPXG4P8MWZD4ZQMXD98CAEGG" ], "model_instance_outputs":[ { "model_instance":"models/yolov7/instances/v1.0-cpu", "task":"TASK_DETECTION", "task_outputs":[ { "index":"01GPXG4P8MWZD4ZQMXD98CAEGG", "detection":{ "objects":[ { "category":"dog", "score":0.9601877, "bounding_box":{ "top":102, "left":324, "width":208, "height":405 } }, { "category":"dog", "score":0.9304089, "bounding_box":{ "top":198, "left":130, "width":198, "height":237 } } ] } } ] } ]}
We will show you how to locate corresponding results in ASYNC pipelines in
[7/7] Create, trigger, and parse an ASYNC
pipeline.
#AI tasks supported by VDP
In addition to Object Detection, Versatile Data Pipeline (VDP) supports extensive AI tasks. A complete list can be found in the Standardise AI tasks documentation.
- Image Classification - classify images into predefined categories
- Object Detection - detect and localise multiple objects in images
- Keypoint Detection - detect and localise multiple keypoints of objects in images
- OCR (Optical Character Recognition) - detect and recognise text in images
- Instance Segmentation - detect, localise and delineate multiple objects in images
- Semantic Segmentation - classify image pixels into predefined categories
- Text to Image - generate images from input text prompts
- Text Generation - generate texts from input text prompts
- The list is growing ... 🌱
For each supported AI task, we define the standard input and output formats maintained via Protocol Buffers. By standardising the data format of model outputs into AI tasks, model in a pipeline is modularized: you can freely switch to use different models in a pipeline as long as the model is designed for the same task. Also, VDP produces a stream of data from models with standard format for use in a data integration or ETL pipeline. That also enables us writing a corresponding parser for the Object Detection task in this tutorial.
#What's next?
We have demonstrated how to use VDP step-by-step in SYNC mode. However, VDP supports other modes, including ASYNC and PULL. Before showing how they work, let's closely look at their differences in our following tutorial → VDP 101 [6/7] Pipeline Modes: SYNC, ASYNC, and PULL.
↓↓↓ VDP 101 - Get familiar with the basics ↓↓↓
