Issue with creating multiple streams in Airbyte API

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

When creating a connection between a Postgres origin with two schemas and an S3 destination via the Airbyte API, only one stream is enabled even though there are two streams (one from each schema). User is looking for a solution to handle multiple streams in this scenario.


Question

Hello community, how are you? I have a problem using the Airbyte API, I will detail it below:
1 - I am creating a Postgres origin, where I am filtering two schemas;
2 - I am creating a destination in S3;
3 - I am creating a connection between both.

All of this creation is being done through the API.

What happens is that the same stream exists in both schemas, and when I create it via the API, I only have the option of passing the stream name parameter. In this case, in Airbyte, only one stream is enabled, but there are two (one from each schema).

Any solution?

Thanksssssss


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airbyte-api", "postgres-origin", "s3-destination", "multiple-streams"]
2

However, I can configure the prefix only at the connection level, not at the data schema level, so when I change the connection prefix, it changes for all streams, becoming duplicated again. There is no possibility to add a variable as a prefix.

3

the connection configuration allows you to add a prefix to the stream names. it applies to all streams in the connection. can you add prefixes for the schema names or something else to deduplicate?