Versatile Data Pipeline (VDP) is a source available unstructured data infrastructure tool to streamline the end-to-end unstructured data processing pipeline:
- Extract unstructured data from pre-built data sources such as cloud/on-prem storage, or IoT devices
- Transform it into analysable or meaningful data representations by AI models
- Load the transformed data into warehouses, applications, or other destinations
We've made it easy to get started with VDP either through Instill Cloud or self-hosting.
Instill Cloud is a fully-managed public cloud service for VDP. The service is currently in Open Alpha. Follow the guideline to register and use an Instill Cloud account.
Next, dive into the VDP core cocepts, explore the Source Connectors and Destinations Connectors to find the connectors you want, and build unstructured data pipelines for your use cases.
VDP is source available on GitHub. Check out the quick start and understand the core concepts. Then, learn about how to deploy VDP in your infrastructure.
VDP is still in Alpha. If you encounter any issues, we'd love to help. Join us on Discord to share with and get help from an active and friendly community!
👋 Pick a role in #welcome and say hi in #introduce-yourself channel
👋 Ask VDP questions in our #vdp channel
👋 Share what you've been working on with VDP in the #showcase channel
👐 We love contribution to VDP in any forms:
- Please refer to the guideline for local development
- Please open a topic in the repository Discussions for any feature requests
- Please open issues for bug reports in the repository
- vdp for general issues;
- pipeline-backend, connector-backend, model-backend, mgmt-backend, console, etc., for specific issues.
- Please refer to the VDP project board to track progress.
Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.
#We're open to collaboration
Our mission is to make AI accessible to everyone. If you are from an academic group and have any showcase of AI in your research, book a meeting to tell us about yourself and your project, and we will be in touch 👐.