Set up local development environment

VDP is built with the microservice architecture. To develop each service independently, we assign profiles to each service in the docker-compose.dev.yml file in VDP. This allows us to selectively enable services for various purposes, e.g., debugging, development.

INFO

Services are associated with profiles through the profiles attribute, which takes an array of profile names. A service will be started only if one of its profile names is activated. A service without profiles will always be started.

VDP assigns seven different profile names:

  • console - start all the dependent services for console
  • mgmt - start all the dependent services for mgmt-backend
  • connector - start all the dependent services for connector-backend
  • model - start all the dependent services for model-backend
  • pipeline - start all the dependent services for pipeline-backend
  • api-gateway - start all the dependent services for api-gateway
  • all - start all services

Use one of the profile name to develop the corresponding service:


# In the vdp repository folder
make build PROFILE=<profile-name>
make latest PROFILE=<profile-name> ITMODE=true

The following guideline shows a specific example of how to develop the connector-backend.

#Start dependent VDP services for connector-backend

On the local machine, clone the vdp repository in your workspace, move to the repository folder, and launch all dependent services:


# Clone the vdp repository
git clone https://github.com/instill-ai/vdp.git && cd vdp
# Use profile `connector` to launch all dependent services
# for `connector-backend` and set ITMODE=true
# for local integration test
make build PROFILE=connector
make latest PROFILE=connector ITMODE=true

#Run dev connector-backend

Clone the connector-backend repository in your workspace and move to the repository folder:


git clone https://github.com/instill-ai/connector-backend.git
cd connector-backend

Build & run the dev image


make build
make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

Start the connector-backend server


docker exec -it connector-backend /bin/bash
go run ./cmd/migration
go run ./cmd/init
go run ./cmd/main

Start the Temporal worker


docker exec -it connector-backend /bin/bash
go run ./cmd/worker

Run the integration test

During local development, you can run the integration test to make sure your latest connector-backend works as intended:


docker exec -it connector-backend /bin/bash
make integration-test

#Stop the dev container


make stop

#To shut down all dependent VDP services:


# In the vdp repository folder
make down

Last updated: 5/29/2023, 12:50:07 AM

by

Xiaofei Du

Previous page

System architecture