Airbyte Data Destinations

The Airbyte Data Connectors allow users to leverage the power of Airbyte and its 30+ destination connectors to facilitate data movement and transformation. VDP implements the Airbyte Protocol to enable seamless integration with Airbyte's destination connectors for writing data results.

#Release Stage

To check the current release status and supported connectors, please consult the Airbyte Destinations Connector Catalog table.

#Resource Configuration

Please refer to each of Airbyte Destinations for configuration details.

#No-code Setup

To create an Airbyte data connector (e.g., PostgreSQL) effortlessly, follow these steps:

Go to the Resource page and click Add Resource
Select Airbyte PostgreSQL
Fill in the required fields
Optionally, give a short description in the Description field

#Low-code Setup

For advanced users who prefer a more hands-on approach, you can create a PostgreSQL data connector using the following cURL commands:

Create a Postgres data connector:

cURL

curl  --location http://localhost:8080/vdp/v1alpha/users/<user-id>/connectors
      --header 'Authorization: Bearer <Your-API-Token>' \
      --data '{
        "id": "postgres-db",
        "connector_definition_name": "connector-definitions/airbyte-destination-postgres",
        "description": "The PostgreSQL database in your basement",
        "configuration": {
          "host": <PostgreSQL host address>,
          "port": <port>,
          "database": <database>,
          "schema": "public",
          "username": <username>,
          "password": <password>,
          "ssl": false
        }
      }'

For other operations, please refer to the VDP Protobufs.

#Connector Configuration

The connector configuration is used for setting up the input data of this component. The configuration is configured in pipeline recipe, please refer to pipeline for more details.

Field	Type	Note
input.<key>*	any	The data you want to write into Airbyte destination

#No-code Setup

#Low-code Setup

This is a sample configuration in the pipeline recipe.

{
  "configuration": {
    "input": {
      "a_text_field": "${ start.prompt }",
      "a_object_field": "${ compA.object }"
    }
  }
}

To enable Airbyte destination connectors and inject data, VDP utilizes the AirbyteRecordMessage JSON object. This facilitates triggering the container's write operation for the Airbyte destination connector.

For this purpose, connector assembles the ConfiguredAirbyteCatalog on-the-fly, utilizing the corresponding connector configuration JSON object used in the check operation. This initiates a Temporal workflow to write the structured data to the destination.

Please note that VDP does not currently support Airbyte's Namespaces.