Streaming | Documentation

A complex pipeline may take time to execute. To keep users updated during the process, Instill VDP offers response streaming, delivering real-time updates of the pipeline run.

#Enable Response Streaming

To enable response streaming, simply add the header Accept: text/event-stream to the request. Instill VDP will return responses in text/event-stream format, providing real-time updates.

cURL

export INSTILL_API_TOKEN=********
curl -X POST 'https://api.instill.tech/v1beta/namespace/NAMESPACE_ID/pipelines/PIPELINE_ID/trigger' \
  --header "Content-Type: application/json" \
  --header "Accept: text/event-stream" \
  --header "Authorization: $INSTILL_API_TOKEN" \
  --data '{
    "data": [
      {
        "variable": {
          "prompt": "hello world"
        }
      }
    ]
  }'

#Example Stream

event: PIPELINE_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:31.800966675Z","batchIndex":0,"status":{"completed":false,"errored":false,"started":true}}
event: COMPONENT_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:32.128045472Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true}}
event: COMPONENT_INPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:32.128772046Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true},"input":{"model":"gpt-4o-mini","n":1,"prompt":"hello world","response-format":{"type":"text"},"system-message":"You are a helpful assistant.","temperature":1,"top-p":1}}
event: COMPONENT_OUTPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.181563573Z","componentID":"openai-0","batchIndex":0,"status":{"completed":false,"errored":false,"skipped":false,"started":true},"output":{"texts":["Hello! How can I assist you today?"],"usage":{"completion-tokens":0,"prompt-tokens":0,"total-tokens":0}}}
event: PIPELINE_OUTPUT_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.182456024Z","batchIndex":0,"status":{"completed":false,"errored":false,"started":true},"output":{"o":["Hello! How can I assist you today?"]}}
event: PIPELINE_STATUS_UPDATED
data: {"updateTime":"2024-10-01T19:40:33.397610043Z","batchIndex":0,"status":{"completed":true,"errored":false,"started":true}}

#Supported Events

Instill VDP currently provides the following events:

PIPELINE_STATUS_UPDATED: Updates the pipeline's status
PIPELINE_OUTPUT_UPDATED: Updates the pipeline's output
PIPELINE_ERROR_UPDATED: Updates error messages in the pipeline
COMPONENT_STATUS_UPDATED: Updates a component's status
COMPONENT_INPUT_UPDATED: Updates a component's input
COMPONENT_OUTPUT_UPDATED: Updates a component's output
COMPONENT_ERROR_UPDATED: Updates error messages in a component

Each event shares the following common fields:

updateTime: The timestamp of the event
batchIndex: Indicates the data index in the batch
status: Reflects the status of the pipeline or component

#In-Component Streaming

Instill VDP not only supports component-level streaming (events for when a component starts, errors, or completes) but also offers in-component streaming. If a component itself supports streaming, such as text generation from LLM models, VDP can stream these updates to the user in real time.

Currently, in-component streaming uses an override mode, meaning each new output replaces the previous one for the same component. In future updates, delta mode will be introduced to reduce transmission overhead and improve performance.