💧 Instill VDP

#About

Instill VDP (Versatile Data Pipeline) is a powerful tool designed to build end-to-end unstructured data pipelines. It leverages the capabilities of 3rd-party data, AI, and applications functionalities and seamlessly connect with Instill Model and Instill Artifact via Instill Component components.

#Create Pipeline

#Pipeline Recipe

A pipeline is defined by a Recipe, which is essentially a YAML composed of sections on, variable, output and multiple components. Here is a sample recipe demonstrating a pipeline that accepts a prompt as input, forwards the prompt to OpenAI, and retrieves the response text.

variable:
  prompt:
    title: input
    format: string
output:
  text:
    title: text
    value: ${op-0.output.texts[0]}
component:
  op-0:
    type: openai
    task: TASK_TEXT_GENERATION
    input:
      model: gpt-3.5-turbo
      n: 2
      prompt: ${variable.prompt}
      response-format:
        type: text
      system-message: You are a helpful assistant.
      temperature: 1
      top-p: 1
    setup:
      api-key: "${secret.my-openai-key}"

#Instill Format

Instill VDP defines several Instill Formats as data type identifiers. These formats simplify the creation of pipelines by eliminating the complexity of converting unstructured data formats. The supported Instill Formats include primitive data types such as boolean, string, integer, number, and json, as well as unstructured data types like document, image, audio, and video, along with array data types.

These Instill Formats enable users to efficiently build pipelines that manage unstructured data in ETL workflows.

#Create Pipeline

Instill VDP offers multiple methods for creating a pipeline, including HTTP and gRPC APIs, the Python/TypeScript SDK, and the web Console. For more details, please refer to the Create Pipeline page.

#Pipeline Management

Instill VDP also provides a set of methods for pipeline management, including listing, updating, and deleting pipelines. For more information, please refer to the Pipeline Management page.

#Secret Management

In a pipeline, it may be necessary to hide certain credential data from the recipe. Instill VDP provides a method to set up secret values and reference them in a recipe, rather than directly exposing sensitive information. For more information, please refer to the Secret Management page.

#Integrations

Integrations allow users to set up components that connect with external services. For more information, please refer to the Integrations page.

#Release Pipeline

Version Control provides enhanced flexibility and management by enabling tracking of modifications in your pipelines and applications over time. This functionality empowers users to leverage various versions of the same pipeline for purposes such as backup, recovery, A/B testing, among others. Instill VDP offers a pipeline release feature that allows users to manage versions and control deployments.

#Run Pipeline

Instill VDP provides multiple methods to run a pipeline, including:

Run-on-Trigger: Offers HTTP, gRPC, and HTTP with multipart/form-data APIs for executing a pipeline.
Run-on-Event: Provides webhook endpoints to receive events from vendors.

Instill VDP offers the capability to disseminate your creations either via a share link or by setting them to public visibility. This allows other users to view your pipeline. For more information, please refer to the Share Pipeline page.

#View Runs

Instill VDP enables users to list and search historical pipeline runs, facilitating monitoring and analysis of pipeline execution. For more information, please refer to the View Runs page.

#About

#Create Pipeline

#Pipeline Recipe

#Instill Format

#Create Pipeline

#Pipeline Management

#Secret Management

#Integrations

#Release Pipeline

#Run Pipeline

#Share Pipeline

#View Runs