Core Concepts Overview

Modern data stack misses unstructured data processing.

Versatile Data Pipeline (VDP) is to fill in the missing piece. At a high-level perspective, VDP streamlines processing unstructured data from various data sources by AI models and sending results to pre-configured data destinations.

In other words, VDP is a tool to extract, transform and load (ETL) unstructured data to their meaningful representations.

To benefit a broader set of users and maximise the tool's usability and affordability, VDP is built with an open heart to make the most of open source software in each component:

VDP overview
VDP overview


Pipeline consists of Source, Model and Destination, altogether defined in a recipe. One pipeline defines one processing unit for a certain AI task. A pipeline can be in either SYNC or ASYNC mode, depending on whether the pipeline task is time-critical or not. See Pipeline for more details.

Pipeline is maintained in pipeline-backend.


Source is the component at the beginning of a VDP pipeline, which streams unstructured data (e.g., image, video, text, audio) from IoT devices (e.g., IP cameras) or cloud storage services (e.g., GCP Cloud Storage, AWS S3).

In the VDP project, the unstructured data source connectors are developed and maintained by the Instill AI community.

Source is maintained in connector-backend.


Model is the core component to process the unstructured data to solve a certain AI task. VDP uses Triton Inference Server under the hood to optimise AI model inference for various frameworks including TensorRT, TensorFlow, PyTorch, ONNX, Python, and more.

VDP uses model definition to abstract a model source (i.e., codebase) so the users can adopt different approaches to import models. Different approaches can provide different features. For example, some approaches support version control but some do not. It is up to the users to adopt approach they are already familiar with. We strive to support as many popular model sources as possible. See Import Models for more details.

Model is maintained in model-backend.


Destination is the component at the end of a VDP pipeline, which loads the structured data provided by the Model component to a target destination.

VDP implements Airbyte Protocol to adopt Airbyte's 30+ destination connectors to write the structured data result from Model to the destination. VDP users can thus directly benefit from the abundant community resources from there to further perform data movement or transformation.

Destination is maintained in connector-backend.


Console is the no-code VDP platform to manage resources and monitor service metrics. The goal is to provide a unified, clean and intuitive user experience of VDP.

Console is maintained in console.


VDP is implemented with API-first design principle. It enables seamless integration to your data stack at any scale.

VDP uses Protocol Buffers version 3 (proto3) as the Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. The same interface definitions are used for both REST (via gRPC-Gateway) and RPC. This enables the access to the API over different protocols:

  • JSON over HTTP
  • Protocol Buffers over gRPC

The interface definitions are maintained in protobufs with auto-generated Go codes in protogen-go and Python code in protogen-python. The genuine protobuf documentation can be found in our Buf Scheme Registry (BSR).

#Release stage

For Instill AI developed and maintained components, we use release stage defined as below to indicate their readiness:

AlphaAn alpha component indicates the component is under development and Instill AI is collecting its early feedback and issues reported by early adopters. We strongly discourage using alpha components for production.
BetaA beta component is considered stable and reliable with no further backwards incompatible changes but has not been validated by a broader group of users. We expect to find and fix a few issues and bugs in the release before it is ready for the next stage.
Generally AvailableA generally available component has been thoroughly tested in the battlefield and is ready for use in a production environment. Its documentation is considered sufficient to support widespread adoption.

Last updated: 1/14/2023, 8:35:23 PM