Versatile Data Pipeline (VDP) is an open-source data infrastructure tool to streamline the end-to-end unstructured data pipeline with Artificial Intelligence (AI). Before we start, let's briefly introduce Why we need VDP.
#Why VDP?
When people say they are data-driven, most of the time, it means structured data, although 80% of the world's data are unstructured. Due to the proliferation of AI, more and more Organisations wish to adopt AI (especially advanced Deep Learning technologies) to process their growing unstructured data for specific use cases, such as extracting business intelligence or delivering AI features to improve customer experience.
Unfortunately, unstructured data are more challenging to analyse. It requires a robust and scalable framework that can build flexible data pipelines connecting data from different sources and serve up-to-date AI models converting the unstructured data into meaningful data presentations. Not many companies know or have the resources to build this kind of complex framework. That's why we introduce Versatile Data Pipeline (VDP) to tackle the problem.
We believe VDP is the next-gen AI-powered unstructured data infrastructure.
— Instill AI Team
#What is VDP
VDP streamlines the end-to-end unstructured data pipeline:
- Extract unstructured data (e.g., images) from pre-built data sources such as cloud/on-prem storage or IoT devices.
- Transform it into analysable data representations (e.g., labels) with AI models.
- Load the transformed data into warehouses, applications, or other destinations.
With VDP, developers won't need to build or maintain their own data connectors, high-performing model serving platform, or data pipeline automation tools.
#Highlights
You don't need to know these when using VDP. But we are proud to say VDP is:
- 🚀 The fastest way to build end-to-end unstructured data pipelines
- ⚡️ High-performing backends implemented in Go
- 🖱️ One-click import & deploy models
- 📦 Standardise AI task output formats for data integration
- 🔌 Pre-built ETL data connectors for extensive data access
- 🪢 Perform in
SYNCfor real-time inference andASYNCfor on-demand workload - 🧁 Scalable API-first microservice design for great developer experience
- 🤠 Built for every AI and Data practitioner with no-/low-code interfaces
Okay, now you know what VDP is. Let's start with our following tutorial → VDP 101 [2/7] Launch VDP on your local machine!
↓↓↓ VDP 101 - Get familiar with the basics ↓↓↓
