Set Up Local Development Environment

Instill Core is currently under rapid development. The project is built with the microservice architecture. To develop each service independently, we assign profiles to each service in the docker-compose.latest.yml file in Core. This allows us to selectively enable services for various purposes, e.g., debugging, development.


Services are associated with profiles through the profiles attribute, which takes an array of profile names. A service will be started only if one of its profile names is activated. A service without profiles will always be started.

Core assigns seven different profile names:

  • console - start all the dependent services for console
  • mgmt - start all the dependent services for mgmt-backend
  • api-gateway - start all the dependent services for api-gateway
  • pipeline - start all the dependent services for pipeline-backend
  • model - start all the dependent services for model
  • controller-model - start all the dependent services for controller-model
  • all - start all services

Use one of the profile name to develop the corresponding service:

# In the core repository folder
make build-latest
make latest PROFILE=<profile-name>

The following guideline shows a specific example of how to develop the pipeline-backend.

#Start dependent services for pipeline-backend

On the local machine, clone the core repository in your workspace, move to the repository folder, and launch all dependent services:

# Clone the core repository
git clone && cd instill-core
# Use profile `pipeline` to launch all dependent services
# for `pipeline-backend`
make build-latest
make latest PROFILE=exclude-pipeline

#Run dev pipeline-backend

Clone the pipeline-backend repository in your workspace and move to the repository folder:

git clone
cd pipeline-backend

Build & run the dev image

make build
make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

Start the pipeline-backend server

docker exec -it pipeline-backend /bin/bash
go run ./cmd/migration
go run ./cmd/main

Start the Temporal worker

docker exec -it pipeline-backend /bin/bash
go run ./cmd/worker

Run the integration test

During local development, you can run the integration test to make sure your latest pipeline-backend works as intended:

docker exec -it pipeline-backend /bin/bash
make integration-test

#Stop the dev container

make stop

#To shut down all dependent Core services:

# In the core repository folder
make down

Last updated: 3/21/2024, 1:42:31 PM