At the two ends of a VDP pipeline, there are Source component (i.e., the E in ETL) and Destination component (i.e., the L in ETL).
#Source
A Source component is a data connector in charge of ingesting unstructured data into Pipeline.
VDP supports HTTP and gRPC source connector for both SYNC and ASYNC pipelines.
Source connectors for the pipeline PULL mode are coming soon.
#Destination
A Destination component is a data connector to write the standardised AI task output from Model to the destination data warehouse or notification.
VDP supports HTTP and gRPC destination connector for the SYNC pipeline.
For the ASYNC pipeline, VDP adopts Airbyte Protocol to inject standardised AI task output from Model into the AirbyteRecordMessage JSON object, to employ Airbyte destination connectors.
Specifically, to trigger a destination connector's container write operation,
connector-backend assembles the ConfiguredAirbyteCatalog on the fly,
with the corresponding connector configuration JSON object used in check operation,
to start a Temporal workflow to write the structured data to the destination.
As far as Airbyte connections and sync modes are concerned, VDP currently supports full_refresh sync mode and append destination sync mode for an ASYNC pipeline.
VDP currently does not support Airbyte's Namespaces.
The release stages of destination connectors developed and maintained by Airbyte community are reported here.
#Definition
VDP uses ConnectorDefinition to define the basic properties and detailed configuration of Source and Destination.
Please check out Source Connectors and Destination Connectors to learn more.
#State
When a connector is initially created, the connector state is checked by the connector container check operation.
The exit code of the container check operation decides the connector state.
If the exit code is 0, the state will be CONNECTED.
If the exit code is 1, the state will be ERROR.
Before the container check return the result, the connector state will remain UNSPECIFIED.
A connector can be switched to DISCONNECTED state by invoking the connector-backend endpoint /disconnect only when its original state is CONNECTED.
A connector can be switched to CONNECTED state by invoking the connector-backend endpoint /connect only when its original state is DISCONNECTED.
If the connector state ends up with ERROR, the connector configuration will need to be updated via the UPDATE connector-backend endpoint.
