Build a SYNC classification pipeline

In this tutorial you'll build your first SYNC classification pipeline and trigger the pipeline to process some images.

#Build via no-code Console

After onboarding, you will be redirected to the Pipeline page on the left sidebar, where you can build your first VDP pipeline by clicking Set up your first pipeline and forever change the way you approach unstructured data processing workflow development.

Empty pipeline list page of the VDP Console

TIP

You can follow the tutorial via the Pipeline page. Or, you can navigate to the Source, Model and Destination page to create each component first and use them to configure a pipeline in Pipeline page later.

#Add a HTTP source

A HTTP source accepts HTTP requests with image payloads to be processed by a pipeline.

To set it up,

click the Pipeline mode ▾ drop-down and choose Sync,
click the Source type ▾ drop-down and choose HTTP, and
click Next.

Add a HTTP source to set up a SYNC pipeline in the VDP Console

INFO

Check our growing list of Source Connectors.

#Import a model from GitHub

To process images, here we import a model from our public GitHub repo instill-ai/model-mobilenetv2.

To set it up,

give your model a unique ID,
[optional] add description,
click the Model source ▾ drop-down and choose GitHub,
fill in the GitHub repository URL instill-ai/model-mobilenetv2, and
click Set up.

Import a model from a GitHub repo via VDP Console

VDP will fetch all the releases of the GitHub repository. Each release is converted into one model instance, using the release tag as the corresponding model instance ID.

INFO

Check our growing list of model sources to learn about how to import models from other platforms.

#Deploy a model instance of the imported model

Once the model is imported,

click the Model instances ▾ drop-down,
pick one model instance, and
click Deploy to put it online.

Deploy a model instance via VDP Console

#Add a HTTP destination

Since we are building a SYNC pipeline, the HTTP destination is automatically paired with the HTTP source.

Just

click Next.

Add a HTTP destination to create a SYNC pipeline in the VDP console

INFO

The paired source and destination connectors for pipelines in SYNC mode:

HTTP source → HTTP destination
gRPC source → gRPC destination

#Set up the pipeline

Almost done! Just

give your pipeline a unique ID,
[optional] add description, and
click Set up.

Set up a pipeline in the VDP Console

Now you should see the newly created SYNC pipeline on the Pipeline page 🎉

Pipeline list page of the VDP Console

#Build via low-code

You can programmatically build a SYNC pipeline via REST API.

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

Step 1: Add a HTTP source

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

Step 2: Import a model from GitHub

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

Step 3: Deploy a model instance

Choose the model instance v1.0-cpu to deploy.

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

Step 4: Add a HTTP destination

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

Step 5: Set up your first pipeline

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

You can programmatically build a SYNC pipeline via REST API.

Step 1: Add a HTTP source

Step 2: Import a model from GitHub

Step 3: Deploy a model instance

Choose the model instance v1.0-cpu to deploy.

Step 4: Add a HTTP destination

Step 5: Set up your first pipeline

curl -X POST http://localhost:8080/v1alpha/source-connectors -d '{
    "id": "source-http",
    "source_connector_definition": "source-connector-definitions/source-http",
    "connector": {
        "configuration": {}
    }
}'
curl -X POST http://localhost:8080/v1alpha/models -d '{
  "id": "mobilenetv2",
  "model_definition": "model-definitions/github",
  "configuration": {
    "repository": "instill-ai/model-mobilenetv2"
  }
}'
curl -X POST http://localhost:8080/v1alpha/models/mobilenetv2/instances/v1.0-cpu/deploy
curl -X POST http://localhost:8080/v1alpha/destination-connectors -d '{
  "id": "destination-http",
  "destination_connector_definition": "destination-connector-definitions/destination-http",
  "connector": {
      "configuration": {}
  }
}'
curl -X POST http://localhost:8080/v1alpha/pipelines -d '{
  "id": "classification",
  "description": "A magic pipeline to classify images",
  "recipe": {
    "source": "source-connectors/source-http",
    "model_instances": [
      "models/mobilenetv2/instances/v1.0-cpu"
    ],
    "destination": "destination-connectors/destination-http"
  }
}'

#Trigger your pipeline for the first time

INFO

All SYNC pipelines are automatically activated.

Now that the classification pipeline is automatically activated, you can make a request to trigger the pipeline to process multiple images within a batch via remote image URLs, Base64 or multipart:

cURL(url)

cURL(base64)

cURL(multipart)

curl -X POST http://localhost:8080/v1alpha/pipelines/classification/trigger -d '{
  "inputs": [
    {
      "image_url": "https://artifacts.instill.tech/imgs/dog.jpg"
    },
    {
      "image_url": "https://artifacts.instill.tech/imgs/polar-bear.jpg"
    }
  ]
}'

in which http://localhost:8080 is the api-gateway default URL.

INFO

api-gateway is the single point of entry into the backend services. Please check the VDP system architecture.

A HTTP response will return

{
  "data_mapping_indices": [
    "01GPRNBGGFTE4AWK6RM825RJG0",
    "01GPRNBGGFTE4AWK6RMB2WXQWA"
  ],
  "model_instance_outputs": [
    {
      "model_instance": "models/mobilenetv2/instances/v1.0-cpu",
      "task": "TASK_CLASSIFICATION",
      "task_outputs": [
        {
          "index": "01GPRNBGGFTE4AWK6RM825RJG0",
          "classification": {
            "category": "golden retriever",
            "score": 0.898938
          }
        },
        {
          "index": "01GPRNBGGFTE4AWK6RMB2WXQWA",
          "classification": {
            "category": "ice bear",
            "score": 0.999554
          }
        }
      ]
    }
  ]
}

🙌 That's it! You just built your first SYNC pipeline and triggered it to convert unstructured image data into structured and analysable insight.

#What's next?

By now, you should have a basic understanding of how VDP streamlines the end-to-end ETL pipelines for unstructured data. This tutorial only shows the tip of what VDP is capable of and is just the beginning of your VDP journey.

Next, check out how to Build an ASYNC object detection pipeline. If you have any problem at all, join our Discord to get community support.