Getting started with VDP

Versatile Data Pipeline (VDP) is an unstructured data ETL tool to streamline the end-to-end unstructured data processing pipeline:

  • Extract unstructured data from pre-built data sources such as cloud/on-prem storage, or IoT devices
  • Transform it into analysable or meaningful data representations by AI models
  • Load the transformed data into warehouses, applications, or other destinations


  • 🚀 The fastest way to build end-to-end unstructured data pipelines
  • ⚡️ High-performing backends implemented in Go
  • 🖱️ One-click import & deploy models
  • 📦 Standardise AI task output formats for data integration
  • 🔌 Pre-built ETL data connectors for extensive data access
  • 🪢 Perform in SYNC for real-time inference and ASYNC for on-demand workload
  • 🧁 Scalable API-first microservice design for great developer experience
  • 🤠 Built for every AI and Data practitioner with no-/low-code interfaces

#Quick start

We've made it easy to get started with VDP either on a local machine or on our demo playground.

#Local run

Make sure you have the prerequisites set up:

  • macOS or Linux - VDP works on macOS or Linux, but does not support Windows at the moment.
  • Docker and Docker Compose - VDP uses Docker Compose to run all services at local.

Get started locally by running:

git clone && cd vdp
make all

Once the services are up, the Console UI is ready to go at http://localhost:3000.

⚠︎ Downloading the Triton server image will take a while, but it should be just a one-time effort.

To shut down all running services:

make down

#Demo playground

Just visit the demo website to try out VDP.

#Deploy on Kubernetes

Please follow the instructions to deploy VDP on local Kubernetes using Helm.

#Start building with VDP

If this is your first time setting up VDP, access the Console (http://localhost:3000) and you should see the onboarding page. Please enter your email and you are all set!

Onboarding page of the VDP Console
Onboarding page of the VDP Console

Jump right in VDP 101 [3/7] Create your first pipeline on VDP and explore other VDP tutorials.

#Learn VDP

📔 Dive into the core concepts

📔 Check out the FAQ and tutorials

📔 Explore the Source Connectors and Destinations Connectors to find the connectors you want

📔 Explore the APIs of Protobufs or OpenAPI (after make all or make doc)


👐 We love contribution to VDP in any forms:


Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.

#Community support

VDP is still in Alpha. If you encounter any issues, we'd love to help. Join us on Discord to share with and get help from an active and friendly community!

👋 Pick a role in #welcome and say hi in #introduce-yourself channel

👋 Ask VDP questions in our #vdp channel

👋 Share what you've been working on with VDP in #showcase channel

#Learn more

🌟 GitHub

📚 Blog

📖 Newsletter

🚀 Product Website

Last updated: 5/29/2023, 12:50:07 AM