Back

Object Detection

HTTP

HTTP

Build a shareable Object Detection application with VDP and Streamlit

VDP and Streamlit are a perfect match if you work with ML/Data and would like to build AI prototypes fast to share with your team, clients or the world.
Xiaofei Du's github avatar

Published by

Xiaofei Du

on 9/2/2022

The theme of this tutorial

When YOLOv7 was out, we were so excited to test it out. Therefore, we built a web app to side-by-side compare the classic YOLOv4 and the freshly released YOLOv7. Once completed, we shared the app with our team and then deployed it online to share with the community.

The app was built with two best-in-class machine learning tools:

  • VDP as the backbone of the Vision task solver, and
  • Streamlit as the application framework to build beautiful UI components.

For anyone who is not familiar with VDP, it is an unstructured data ETL tool that we've been working on. The goal of VDP is to streamline the end-to-end unstructured data flow, with the transform component being able to flexibly import AI models to process the unstructured data for a specific task for Vision, Language and more.

It is the future for unstructured data ETL, where developers won't need to build their own data connectors, high-maintenance model serving platform or ELT pipeline automation tool. ā€” From Introducing VDP, open-source unstructured data ETL

Streamlit removes the barriers for Data/ML practitioners to build shareable web apps. No need to write HTML, CSS and Javascript to create beautiful UIs, you can just write everything in pure Python.

This tutorial will demonstrate how to replicate the YOLOv4 vs. YOLOv7 web app. It shows that VDP and Streamlit are a perfect match if you work with ML/Data and would like to build AI prototypes fast to share with your team, clients or the world.

#Prerequisites

  • Docker and Docker Compose
  • Python 3.8+ with an environment-management tool such as Conda

#Build Object Detection pipelines

INFO

VDP standardises outputs for AI tasks. Therefore, a model is modularised in a pipeline, and model outputs are in standard format for use in data integration or ETL pipeline.

Vision tasks focus on analysing and understanding the content of unstructured visual data in the same way as the human visual system does. Some classic Vision tasks include Image Classification, Object Detection, image segmentation and Keypoint Detection. These primitive Vision tasks are the foundation for building many real-world industrial computer vision applications.

In the following section, we will build two Object Detection pipelines with YOLOv4 and YOLOv7 in VDP, respectively. The pipelines will serve as the AI backbone for the Streamlit app.

#Run VDP locally


$ git clone https://github.com/instill-ai/vdp.git && cd vdp
$ make all

Once the services are up, the Console is ready to go at http://localhost:3000.

#Build via no-code Console

A pipeline in SYNC mode responds to a request synchronously. It is suitable for our Streamlit app to perform real-time inference where low latency is of concern. Check here for more details.

INFO

No matter where your model stores, we want to keep your models in the same place without changes. VDP integrates with many model platforms and tools to make importing models as easy as possible.

After onboarding, you will be redirected to the Pipeline page on the left sidebar, where you can build a SYNC pipeline with YOLOv4. Please follow VDP 101 [3/7] Create your first pipeline on VDP with a few alterations:

  1. add an HTTP source,

  2. import and deploy a model from the v1.0-cpu tag of the GitHub repository instill-ai/model-yolov4-dvc with ID yolov4,

  3. add an HTTP data destination, and

  4. set up a pipeline with ID yolov4.

Pipeline is empty view
Add sync http source
Add yolov4 model
Add sync http destination
Add sync yolov4 pipeline

Swipe to see other images.

#Build via low-code

You could build a pipeline with YOLOv7 in the same way by importing instill-ai/model-yolov7-dvc via no-code Console. Or, you can build it via REST API.

INFO

VDP is implemented with API-first design principle. It enables seamless integration to your data stack at any scale.

Programmatically build a SYNC pipeline via REST API:


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Step 1:

Add an HTTP data source


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Step 2:

Import a model from GitHub


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Step 3:

Deploy the model


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Step 4:

Add a HTTP data destination


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Step 5:

Set up a pipeline with ID yolov7


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Programmatically build a SYNC pipeline via REST API:

Step 1:

Add an HTTP data source

Step 2:

Import a model from GitHub

Step 3:

Deploy the model

Step 4:

Add a HTTP data destination

Step 5:

Set up a pipeline with ID yolov7


curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
"id": "source-http",
"source_connector_definition": "source-connector-definitions/source-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
"id": "yolov7",
"model_definition": "model-definitions/github",
"configuration": {
"repository": "instill-ai/model-yolov7-dvc",
"tag": "v1.0-cpu"
}
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
"id": "destination-http",
"destination_connector_definition": "destination-connector-definitions/destination-http",
"connector": {
"configuration": {}
}
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
"id": "yolov7",
"recipe": {
"version": "v1alpha",
"components": [
{
"id": "source",
"resource_name": "source-connectors/source-http"
},
{
"id": "model",
"resource_name": "models/yolov7"
},
{
"id": "destination",
"resource_name": "destination-connectors/destination-http"
}
]
}
}'

Now you should see two pipelines yolov4 and yolov7 in the Console.

Pipeline page on the VDP Console
Pipeline page on the VDP Console

In the next section, we will build a Streamlit app to send requests triggering the pipelines and visualise the detection outputs with a beautiful UI.

#Build the Streamlit app

#1. Create a Python virtual environment

In this tutorial, we'll use Conda as the package management system. You can install Conda via anaconda or miniconda. Using a virtual environment is not required but recommended.

Create and activate an environment named vdp-streamlit with Python 3.8:


conda create --name vdp-streamlit python=3.8
conda activate vdp-streamlit

Once activated, you can run scripts from this environment.

#2. Install app dependencies

Go to /examples/streamlit-object-detection-sync-http-http directory of the VDP project.


cd examples/streamlit-object-detection-sync-http-http

The directory of the app will look like the following:


ā”œā”€ā”€ Dockerfile
ā”œā”€ā”€ README.md
ā”œā”€ā”€ main.py
ā”œā”€ā”€ requirements.txt
ā””ā”€ā”€ utils.py

where requirements.txt file contains all the app dependencies. Install all the dependencies required to run the app from the activated virtual environment.


pip install -r requirements.txt

#3. Trigger the pipelines

In the main app script main.py, we use a Streamlit text.input to enable user to provide an image URL for inference.


image_url = st.text_input( label="Feed me with an image URL and press ENTER",
value="https://artifacts.instill.tech/imgs/dog.jpg")

The pipelines we built are SYNC with HTTP connectors, so we create a trigger_detection_pipeline function to trigger a pipeline by sending a HTTP request with payload constructed with the provided image_url.


def trigger_detection_pipeline(api_gateway_url: str, pipeline_id: str, image_url: str) -> requests.Response:
r""" Trigger a pipeline composed with a detection model using remote image URL
Args:
api_gateway_url (str): VDP API base URL
pipeline_id (str): pipeline ID
image_url (str): remote image URL, e.g., `https://artifacts.instill.tech/imgs/dog.jpg`
Returns: requests.Response
pipeline trigger result
"""
body = {
"task_inputs": [
{
"detection": {
'image_url': image_url
}
}
]
}
return requests.post("{}/pipelines/{}/triggerSync".format(api_gateway_url, pipeline_id), json=body)

Since the pipeline output is standardised, we also create a parse_detection_response function to parse the response into a list of bounding boxes, categories and scores according to the standardised format. Learn more about standardising Object Detection task.

Standardise Object Detection task in VDP
Standardise Object Detection task in VDP

def parse_detection_response(resp: requests.Response) -> Tuple[List[Tuple[float]], List[str], List[float]]:
r""" Parse a detection response in to bounding boxes, categories and scores
Args:
resp (`requests.Response`): response for standardised object Detection task
Returns: parsed outputs, a tuple of
List[Tuple[float]]: a list of detected bounding boxes in the format of (left, top, width, height)
List[str]: a list of category labels, each of which corresponds to a detected bounding box. The length of this list must be the same as the detected bounding boxes.
List[float]: a list of scores, each of which corresponds to a detected bounding box. The length of this list must be the same as the detected bounding boxes.
"""
if resp.status_code != 200:
return [], [], []
else:
# Parse JSON into an object with attributes corresponding to dict keys.
r = json.loads(resp.text, object_hook=lambda d: SimpleNamespace(**d))
boxes_ltwh = []
categories = []
scores = []
for v in r.model_outputs[0].task_outputs[0].detection.objects:
boxes_ltwh.append((
v.bounding_box.left,
v.bounding_box.top,
v.bounding_box.width,
v.bounding_box.height))
categories.append(v.category)
scores.append(v.score)
return boxes_ltwh, categories, scores

In the main function, the input image is sent to trigger both pipelines for a side-by-side comparison.


# Trigger VDP pipelines
pipeline_ids = [opt.yolov4, opt.yolov7]
pipeline_results = []
for pipeline_id in pipeline_ids:
resp = trigger_detection_pipeline(
api_gateway_url, pipeline_id, image_url)
boxes_ltwh, categories, scores = parse_detection_response(resp)
pipeline_results.append((resp, boxes_ltwh, categories, scores))

#4. Visualise the detections

Thanks to Streamlit's powerful visualisation features, we create and use functions in utils.py to visualise the detections in different ways:

  • draw the detections on the input image
  • display the detections as pandas.Dataframe in an interactive table

# Visualise detections on input image
cols = st.columns(len(pipeline_ids))
captions = ["YOLOv4", "YOLOv7"]
for col, (resp, boxes_ltwh, categories, scores), cap in zip(cols, pipeline_results, captions):
if resp.status_code == 200:
# Show image overlaid with detection results
img_draw = draw_detection(img, boxes_ltwh, categories, scores)
col.image(img_draw, use_column_width=True, caption=cap)
else:
col.error("{} inference error".format(cap))
# Display detections with scores >= 0.5 in a table
cols = st.columns(len(pipeline_ids))
detection_thres = 0.5
for col, (resp, boxes_ltwh, categories, scores) in zip(cols, pipeline_results):
if resp.status_code == 200:
_, df = gen_detection_table(
boxes_ltwh, categories, scores)
if len(df):
col.dataframe(df.style.highlight_between(
subset='Score', left=detection_thres, right=1.0))
else:
col.dataframe(df)
st.caption(
"Highlight detections with score >= {}".format(detection_thres))

#5. Run the app


streamlit run main.py
You can now view your Streamlit app in your browser.
Local URL: http://localhost:8501
Network URL: http://192.168.0.10:8501

Now go to http://localhost:8501 in the browser and have some fun with your app!

Fill the input field with a random image URL and press Enter to see the detection results of YOLOv4 and YOLOv7 side-by-side.

YOLOv4 vs. YOLOv7 live demo screenshot
YOLOv4 vs. YOLOv7 live demo screenshot

#Conclusion

šŸ„³ Congratulations! You've built a beautiful app to showcase STOA object detectors using Streamlit powered by VDP.

#What's next?

By the end of the demo, we hint that you can manipulate the detection results using other structured data toolings in the modern data stack. Check the Build a Cow Counter dashboard using VDP, Postgres and Metabase tutorial to transform unstructured images into analysable structured insights, and send the structured insights to a PostgreSQL database.

If you enjoyed VDP, we're building a fully managed service for VDP - Instill Cloud (Alpha):

  • Painless setup
  • Maintenance-free infrastructure
  • Start for free, pay as you grow

We also invite you to join our Discord community to share your use cases and showcase your work with Data/AI practitioners.

Last updated: 5/29/2023, 12:50:07 AM