Versatile Data Pipeline (VDP) is an unstructured data ETL tool to streamline the end-to-end unstructured data processing pipeline:
- Extract unstructured data from pre-built data sources such as cloud/on-prem storage, or IoT devices
- Transform it into analysable or meaningful data representations by AI models
- Load the transformed data into warehouses, applications, or other destinations
#Highlights
- 🚀 The fastest way to build end-to-end unstructured data pipelines
- ⚡️ High-performing backends implemented in Go
- 🖱️ One-click import & deploy models
- 📦 Standardise AI task output formats for data integration
- 🔌 Pre-built ETL data connectors for extensive data access
- 🪢 Perform in
SYNCfor real-time inference andASYNCfor on-demand workload - 🧁 Scalable API-first microservice design for great developer experience
- 🤠 Built for every AI and Data practitioner with no-/low-code interfaces
#Quick start
We've made it easy to get started with VDP either on a local machine or on our demo playground.
#Local run
Make sure you have the prerequisites set up:
- macOS or Linux - VDP works on macOS or Linux, but does not support Windows at the moment.
- Docker and Docker Compose - VDP uses Docker Compose to run all services at local.
Get started locally by running:
git clone https://github.com/instill-ai/vdp.git && cd vdpmake all
Once the services are up, the Console UI is ready to go at http://localhost:3000.
⚠︎ Downloading the Triton server image will take a while, but it should be just a one-time effort.
To shut down all running services:
make down
#Demo playground
Just visit the demo website to try out VDP.
#Deploy on Kubernetes
Please follow the instructions to deploy VDP on local Kubernetes using Helm.
#Start building with VDP
If this is your first time setting up VDP, access the Console (http://localhost:3000) and you should see the onboarding page. Please enter your email and you are all set!
Jump right in VDP 101 [3/7] Create your first pipeline on VDP and explore other VDP tutorials.
#Learn VDP
📔 Dive into the core concepts
📔 Check out the FAQ and tutorials
📔 Explore the Source Connectors and Destinations Connectors to find the connectors you want
📔 Explore the APIs of Protobufs or OpenAPI (after make all or make doc)
#Contribution
👐 We love contribution to VDP in any forms:
- Please refer to the guideline for local development
- Please open a topic in the repository Discussions for any feature requests
- Please open issues for bug report in the repository
- vdp for general issues;
- pipeline-backend, connector-backend, model-backend, mgmt-backend, console, etc., for specific issues.
- Please refer to the VDP project board to track progress.
Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.
#Community support
VDP is still in Alpha. If you encounter any issues, we'd love to help. Join us on Discord to share with and get help from an active and friendly community!
👋 Pick a role in #welcome and say hi in #introduce-yourself channel
👋 Ask VDP questions in our #vdp channel
👋 Share what you've been working on with VDP in #showcase channel
#Learn more
🌟 GitHub
📚 Blog