Set Up Local Development Environment

Instill Core is currently under rapid development. The system architecture shown below is subject to change but will always follow the general guidance described in Tech.

The Instill Core system architecture
The Instill Core system architecture

The project is built with the microservice architecture. To develop each service independently, we assign profiles to each service in the docker-compose.latest.yml file in Core. This allows us to selectively enable services for various purposes, e.g., debugging, development.

INFO

Services are associated with profiles through the profiles attribute, which takes an array of profile names. A service will be started only if one of its profile names is activated. A service without profiles will always be started.

Core assigns seven different profile names:

  • console - start all the dependent services for console
  • mgmt - start all the dependent services for mgmt-backend
  • api-gateway - start all the dependent services for api-gateway
  • pipeline - start all the dependent services for pipeline-backend
  • controller-vdp - start all the dependent services for controller-vdp
  • model - start all the dependent services for model
  • controller-model - start all the dependent services for controller-model
  • all - start all services

Use one of the profile name to develop the corresponding service:


# In the core repository folder
make build-latest
make latest PROFILE=<profile-name>

The following guideline shows a specific example of how to develop the pipeline-backend.

#Start dependent services for pipeline-backend

On the local machine, clone the core repository in your workspace, move to the repository folder, and launch all dependent services:


# Clone the core repository
git clone https://github.com/instill-ai/core.git && cd core
# Use profile `pipeline` to launch all dependent services
# for `pipeline-backend`
make build-latest
make latest PROFILE=pipeline

#Run dev pipeline-backend

Clone the pipeline-backend repository in your workspace and move to the repository folder:


git clone https://github.com/instill-ai/pipeline-backend.git
cd pipeline-backend

Build & run the dev image


make build
make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

Start the pipeline-backend server


docker exec -it pipeline-backend /bin/bash
go run ./cmd/migration
go run ./cmd/main

Start the Temporal worker


docker exec -it pipeline-backend /bin/bash
go run ./cmd/worker

Run the integration test

During local development, you can run the integration test to make sure your latest pipeline-backend works as intended:


docker exec -it pipeline-backend /bin/bash
make integration-test

#Stop the dev container


make stop

#To shut down all dependent Core services:


# In the core repository folder
make down

Last updated: 1/15/2024, 2:25:21 PM