Kubernetes Using Helm

Kubernetes offers an elastic way of managing VDP to persist the services and scale resources horizontally and vertically.

#Setup

VDP currently supports deploying on Kubernetes.

#Cluster setup

We recommend using one of the local Kubernetes solutions:

Docker Desktop (macOS) with Kubernetes enabled
minikube
kind

#`kubectl` setup

Make sure the Kubernetes CLI tool kubectl has been installed in your environment. If not, please follow the official instructions to install.

Make sure kubectl is configured to connect to the local cluster by

kubectl use-context <cluster-name>

#Helm setup

Make sure Helm has been installed in your environment. If not, please follow the official instructions to install.

Check out the VDP Helm repository on ArtifactHub.

To add VDP Helm repository, simply run

helm repo add instill https://helm.instill.tech

, and

helm repo update

You can then run

helm search repo vdp --devel

to see the charts:

NAME            CHART VERSION   APP VERSION     DESCRIPTION
instill/vdp     <chart-version> <app-version>   Versatile Data Pipeline (VDP) empowers the mode...

INFO

The VDP and its Helm chart are still in a pre-release Alpha version, so the --devel flag is necessary for the access of Helm charts.

#Install

To install the chart:

helm install vdp instill/vdp --devel --namespace vdp --create-namespace

And visit the api-gateway (http://localhost:8080) and Console (http://localhost:3000) by port forwarding:

export APIGATEWAY_POD_NAME=$(kubectl get pods --namespace vdp -l "app.kubernetes.io/component=api-gateway,app.kubernetes.io/instance=vdp" -o jsonpath="{.items[0].metadata.name}")
export APIGATEWAY_CONTAINER_PORT=$(kubectl get pod --namespace vdp $APIGATEWAY_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
export CONSOLE_POD_NAME=$(kubectl get pods --namespace vdp -l "app.kubernetes.io/component=console,app.kubernetes.io/instance=vdp" -o jsonpath="{.items[0].metadata.name}")
export CONSOLE_CONTAINER_PORT=$(kubectl get pod --namespace vdp $CONSOLE_POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
kubectl --namespace vdp port-forward $APIGATEWAY_POD_NAME 8080:${APIGATEWAY_CONTAINER_PORT} &
kubectl --namespace vdp port-forward $CONSOLE_POD_NAME 3000:${CONSOLE_CONTAINER_PORT} &

#Uninstall

To uninstall the chart:

helm uninstall vdp --namespace vdp

, and further clean up all VDP-related persistent volumes:

kubectl delete pvc --all -n vdp
kubectl delete namespace vdp

#Troubleshooting

If you encounter any issues, please create an issue on GitHub or come to our Discord channel #ask-for-help. The community loves to help!