Back

Object Detection

HTTP

HTTP

VDP 101 [3/7] Create your first pipeline on VDP

In this tutorial, we will build our first unstructured data pipeline with VDP, which allows you to process your data for a specific AI task with minimum effort.
Po-Yu Chen's github avatar

Published by

Po-Yu Chen

on 3/6/2023

The theme of this tutorial

In this tutorial, we will build our first unstructured data pipeline with VDP, which allows you to process your data for a specific AI task with minimum effort.

As we demonstrated in VDP 101 - Introduction, VDP aims to streamline the end-to-end unstructured data pipeline. An unstructured data pipeline is defined by a recipe which consists of three components:

  • source - where the pipeline starts to ingest unstructured data to be processed.
  • model instances - deployed AI models to process the ingested unstructured data and generate meaningful outputs.
  • destination - where to send the processed outputs.

The following tutorial will demonstrate how to set up these three components and build a SYNC pipeline for Object Detection with YOLOv7 via Console.

#Prerequisites

Before diving into the details, please ensure you have launched VDP on your machine. If not, you can set it up following the tutorial VDP 101 [2/7] Launch VDP on your local machine.

After launching VDP, you can access the Console via http://localhost:3000. If this is your first time setting up VDP, you should see the onboarding page. Enter your email, and you are all set!

#Step 1: Create a new pipeline

Click on the Pipeline page on the left sidebar. Since we have not yet set up any pipeline, this page will be empty. To create our first pipeline via VDP Console, click the Set up your first pipeline button.

Empty pipeline list page of the VDP Console.

#Step 2: Add an HTTP source

VDP currently supports two sources, HTTP and gRPC. An HTTP source accepts HTTP requests with payloads to be processed by a pipeline.

INFO

Check our growing list of Source Connectors.

To set up a Source Connector,

  1. click the Pipeline mode ▾ drop-down and choose Sync,
  2. click the Source type ▾ drop-down and choose HTTP, and
  3. click Next.
Add an HTTP source to set up a SYNC pipeline in the VIP Console.

#Step 3: Import and deploy a model from GitHub

To fulfil objective detection tasks, we import a model from our public GitHub repository instill-ai/model-yolov7-dvc.

To set it up,

  1. give your model a unique ID,
  2. add a description (optional),
  3. click the Model source ▾ drop-down and choose GitHub,
  4. fill in the GitHub repository URL instill-ai/model-yolov7-dvc, and
  5. click Set up to fetch the AI model to VDP.
Import a model from a GitHub repository via VDP Console.

VDP will fetch all the releases of the GitHub repository. Each release is converted into one Model Instance, using the release tag as the corresponding model instance ID.

INFO

A Model Instance represents a tagged snapshot of a Model. A model may have multiple model instances. The tag of a model instance depends on the model source and what versioning control tool the model source uses.

Check out the documentation.

ff9c78e (chore: update links)

Once the model is imported, we can choose one model instance to deploy. For simplicity,

  1. click the Model instances ▾ drop-down,
  2. pick v1.0-cpu to deploy on CPU or v1.0-gpu to deploy on GPU, and
  3. click Deploy to deploy it on VDP.
Deploy a model instance via VDP Console.

#Step 4: Add an HTTP destination

Since we are building a SYNC pipeline, the HTTP destination is paired automatically with the HTTP source as we set up for the source. Click Next.

Add an HTTP destination to create a SYNC pipeline in the VDP Console.
INFO

When creating pipelines under sync mode, source and destination connectors in VDP must be the same, which means:

  • HTTP source → HTTP destination
  • gRPC source → gRPC destination

#Step 5: Set up the pipeline

We are almost there! We have created Source, Model, and Destination. The last step is to give this pipeline an ID, and we are ready to go! Just

  1. give your pipeline a unique ID vdp-101-sync,
  2. add a description (optional), and
  3. click Set up.
Set up a pipeline in the VDP Console.

🎉 CONGRATULATIONS! You have your first VDP pipeline setup. You should see it on the Pipeline page.

Find your first pipeline on the Pipeline page.

You can find further details about this pipeline by clicking on the one you just created. The green light indicates the pipeline is Active and can be triggered via sending HTTP requests.

INFO

Check out the documentation to understand all the pipeline states.

Find further details about this pipeline.

#What's next?

You may notice the REST request examples to trigger the pipeline in the Trigger section at the bottom of the page. No worry about this now. You will learn how to trigger this pipeline in the following tutorial → VDP 101 [4/7] How to trigger a SYNC pipeline?.



↓↓↓ VDP 101 - Get familiar with the basics ↓↓↓

Last updated: 3/11/2023, 2:21:57 AM