Build a shareable Object Detection application with VDP and Streamlit

When YOLOv7 was out, we were so excited to test it out. Therefore, we built a web app to side-by-side compare the classic YOLOv4 and the freshly released YOLOv7. Once completed, we shared the app with our team and then deployed it online to share with the community.

👉 YOLOv4 vs. YOLOv7 live demo

The app was built with two best-in-class machine learning tools:

VDP as the backbone of the Vision task solver, and
Streamlit as the application framework to build beautiful UI components.

For anyone who is not familiar with VDP, it is an unstructured data ETL tool that we've been working on. The goal of VDP is to streamline the end-to-end unstructured data flow, with the transform component being able to flexibly import AI models to process the unstructured data for a specific task for Vision, Language and more.

It is the future for unstructured data ETL, where developers won't need to build their own data connectors, high-maintenance model serving platform or ELT pipeline automation tool. — From Introducing VDP, open-source unstructured data ETL

Streamlit removes the barriers for Data/ML practitioners to build shareable web apps. No need to write HTML, CSS and Javascript to create beautiful UIs, you can just write everything in pure Python.

This tutorial will demonstrate how to replicate the YOLOv4 vs. YOLOv7 web app. It shows that VDP and Streamlit are a perfect match if you work with ML/Data and would like to build AI prototypes fast to share with your team, clients or the world.

#Prerequisites

Docker and Docker Compose
Python 3.8+ with an environment-management tool such as Conda

#Build Object Detection pipelines

INFO

VDP standardises outputs for AI tasks. Therefore, a model is modularised in a pipeline, and model outputs are in standard format for use in data integration or ETL pipeline.

Vision tasks focus on analysing and understanding the content of unstructured visual data in the same way as the human visual system does. Some classic Vision tasks include Image Classification, Object Detection, image segmentation and Keypoint Detection. These primitive Vision tasks are the foundation for building many real-world industrial computer vision applications.

In the following section, we will build two Object Detection pipelines with YOLOv4 and YOLOv7 in VDP, respectively. The pipelines will serve as the AI backbone for the Streamlit app.

#Run VDP locally

$ git clone https://github.com/instill-ai/vdp.git && cd vdp
$ make all

Once the services are up, the Console is ready to go at http://localhost:3000.

#Build via no-code Console

A pipeline in SYNC mode responds to a request synchronously. It is suitable for our Streamlit app to perform real-time inference where low latency is of concern. Check here for more details.

INFO

No matter where your model stores, we want to keep your models in the same place without changes. VDP integrates with many model platforms and tools to make importing models as easy as possible.

After onboarding, you will be redirected to the Pipeline page on the left sidebar, where you can build a SYNC pipeline with YOLOv4. Please follow VDP 101 [3/7] Create your first pipeline on VDP with a few alterations:

add an HTTP source,
import and deploy a model from the v1.0-cpu tag of the GitHub repository instill-ai/model-yolov4-dvc with ID yolov4,
add an HTTP data destination, and
set up a pipeline with ID yolov4.

Swipe to see other images.

Build a SYNC Object Detection pipeline with YOLOv4 via no-code Console

#Build via low-code

You could build a pipeline with YOLOv7 in the same way by importing instill-ai/model-yolov7-dvc via no-code Console. Or, you can build it via REST API.

INFO

VDP is implemented with API-first design principle. It enables seamless integration to your data stack at any scale.

Programmatically build a SYNC pipeline via REST API:

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Step 1:

Add an HTTP data source

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Step 2:

Import a model from GitHub

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Step 3:

Deploy the model

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Step 4:

Add a HTTP data destination

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Step 5:

Set up a pipeline with ID yolov7

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Programmatically build a SYNC pipeline via REST API:

Step 1:

Add an HTTP data source

Step 2:

Import a model from GitHub

Step 3:

Deploy the model

Step 4:

Add a HTTP data destination

Step 5:

Set up a pipeline with ID yolov7

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "yolov7",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-yolov7-dvc",
    "tag": "v1.0-cpu"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/yolov7/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "yolov7",
  "recipe": {
    "version": "v1alpha",
    "components": [
      {
        "id": "source",
        "resource_name": "source-connectors/source-http"
      },
      {
        "id": "model",
        "resource_name": "models/yolov7"
      },
      {
        "id": "destination",
        "resource_name": "destination-connectors/destination-http"
      }
    ]
  }
}'

Now you should see two pipelines yolov4 and yolov7 in the Console.

Pipeline page on the VDP Console

In the next section, we will build a Streamlit app to send requests triggering the pipelines and visualise the detection outputs with a beautiful UI.

#Build the Streamlit app

#1. Create a Python virtual environment

In this tutorial, we'll use Conda as the package management system. You can install Conda via anaconda or miniconda. Using a virtual environment is not required but recommended.

Create and activate an environment named vdp-streamlit with Python 3.8:

conda create --name vdp-streamlit python=3.8
conda activate vdp-streamlit

Once activated, you can run scripts from this environment.

#2. Install app dependencies

Go to /examples/streamlit-object-detection-sync-http-http directory of the VDP project.

cd examples/streamlit-object-detection-sync-http-http

The directory of the app will look like the following:

├── Dockerfile
├── README.md
├── main.py
├── requirements.txt
└── utils.py

where requirements.txt file contains all the app dependencies. Install all the dependencies required to run the app from the activated virtual environment.

pip install -r requirements.txt

#3. Trigger the pipelines

In the main app script main.py, we use a Streamlit text.input to enable user to provide an image URL for inference.

image_url = st.text_input( label="Feed me with an image URL and press ENTER",
    value="https://artifacts.instill.tech/imgs/dog.jpg")

The pipelines we built are SYNC with HTTP connectors, so we create a trigger_detection_pipeline function to trigger a pipeline by sending a HTTP request with payload constructed with the provided image_url.

def trigger_detection_pipeline(api_gateway_url: str, pipeline_id: str, image_url: str) -> requests.Response:
    r""" Trigger a pipeline composed with a detection model using remote image URL
    Args:
        api_gateway_url (str): VDP API base URL
        pipeline_id (str): pipeline ID
        image_url (str): remote image URL, e.g., `https://artifacts.instill.tech/imgs/dog.jpg`
    Returns: requests.Response
        pipeline trigger result
    """
    body = {
        "task_inputs": [
            {
                "detection": {
                    'image_url': image_url
                }
            }
        ]
    }
    return requests.post("{}/pipelines/{}/triggerSync".format(api_gateway_url, pipeline_id), json=body)

Since the pipeline output is standardised, we also create a parse_detection_response function to parse the response into a list of bounding boxes, categories and scores according to the standardised format. Learn more about standardising Object Detection task.

Standardise Object Detection task in VDP

def parse_detection_response(resp: requests.Response) -> Tuple[List[Tuple[float]], List[str], List[float]]:
    r""" Parse a detection response in to bounding boxes, categories and scores
    Args:
        resp (`requests.Response`): response for standardised object Detection task
    Returns: parsed outputs, a tuple of
        List[Tuple[float]]: a list of detected bounding boxes in the format of (left, top, width, height)
        List[str]: a list of category labels, each of which corresponds to a detected bounding box. The length of this list must be the same as the detected bounding boxes.
        List[float]: a list of scores, each of which corresponds to a detected bounding box. The length of this list must be the same as the detected bounding boxes.
    """
    if resp.status_code != 200:
        return [], [], []
    else:
        # Parse JSON into an object with attributes corresponding to dict keys.
        r = json.loads(resp.text, object_hook=lambda d: SimpleNamespace(**d))
        boxes_ltwh = []
        categories = []
        scores = []
        for v in r.model_outputs[0].task_outputs[0].detection.objects:
            boxes_ltwh.append((
                v.bounding_box.left,
                v.bounding_box.top,
                v.bounding_box.width,
                v.bounding_box.height))
            categories.append(v.category)
            scores.append(v.score)
        return boxes_ltwh, categories, scores

In the main function, the input image is sent to trigger both pipelines for a side-by-side comparison.

# Trigger VDP pipelines
pipeline_ids = [opt.yolov4, opt.yolov7]
pipeline_results = []
for pipeline_id in pipeline_ids:
    resp = trigger_detection_pipeline(
        api_gateway_url, pipeline_id, image_url)
    boxes_ltwh, categories, scores = parse_detection_response(resp)
    pipeline_results.append((resp, boxes_ltwh, categories, scores))

#4. Visualise the detections

Thanks to Streamlit's powerful visualisation features, we create and use functions in utils.py to visualise the detections in different ways:

draw the detections on the input image
display the detections as pandas.Dataframe in an interactive table

# Visualise detections on input image
cols = st.columns(len(pipeline_ids))
captions = ["YOLOv4", "YOLOv7"]
for col, (resp, boxes_ltwh, categories, scores), cap in zip(cols, pipeline_results, captions):
    if resp.status_code == 200:
        # Show image overlaid with detection results
        img_draw = draw_detection(img, boxes_ltwh, categories, scores)
        col.image(img_draw, use_column_width=True, caption=cap)
    else:
        col.error("{} inference error".format(cap))
# Display detections with scores >= 0.5 in a table
cols = st.columns(len(pipeline_ids))
detection_thres = 0.5
for col, (resp, boxes_ltwh, categories, scores) in zip(cols, pipeline_results):
    if resp.status_code == 200:
        _, df = gen_detection_table(
            boxes_ltwh, categories, scores)
        if len(df):
            col.dataframe(df.style.highlight_between(
                subset='Score', left=detection_thres, right=1.0))
        else:
            col.dataframe(df)
st.caption(
    "Highlight detections with score >= {}".format(detection_thres))

#5. Run the app

streamlit run main.py
  You can now view your Streamlit app in your browser.
  Local URL: http://localhost:8501
  Network URL: http://192.168.0.10:8501

Now go to http://localhost:8501 in the browser and have some fun with your app!

Fill the input field with a random image URL and press Enter to see the detection results of YOLOv4 and YOLOv7 side-by-side.

YOLOv4 vs. YOLOv7 live demo screenshot

#Conclusion

🥳 Congratulations! You've built a beautiful app to showcase STOA object detectors using Streamlit powered by VDP.

⭐️ Star VDP on GitHub

#What's next?

By the end of the demo, we hint that you can manipulate the detection results using other structured data toolings in the modern data stack. Check the Build a Cow Counter dashboard using VDP, Postgres and Metabase tutorial to transform unstructured images into analysable structured insights, and send the structured insights to a PostgreSQL database.

If you enjoyed VDP, we're building a fully managed service for VDP - Instill Cloud (Alpha):

Painless setup
Maintenance-free infrastructure
Start for free, pay as you grow

🚀 Try Instill Cloud Free

We also invite you to join our Discord community to share your use cases and showcase your work with Data/AI practitioners.

Build a shareable Object Detection application with VDP and Streamlit

#Prerequisites

#Build Object Detection pipelines

#Run VDP locally

#Build via no-code Console

#Build via low-code

#Build the Streamlit app

#1. Create a Python virtual environment

#2. Install app dependencies

#3. Trigger the pipelines

#4. Visualise the detections

#5. Run the app

#Conclusion

#What's next?

Similar Articles

Build a Text Generation Discord app with VDP

Build a Cow Counter dashboard using VDP, Postgres and Metabase

On this page

Contribution