Creating streams using Airbyte API

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

The user is inquiring about the capability of the Airbyte API to create streams.


Question

Does the Airbyte API allow for creation of streams? Looked at https://github.com/airbytehq/airbyte-api-python-sdk/blob/main/docs/sdks/streams/README.md but can only see https://github.com/airbytehq/airbyte-api-python-sdk/blob/main/docs/sdks/streams/README.md#get_stream_properties|get_stream_properties


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte-api", "creating-streams", "get-stream-properties"]
2

What exactly are you trying to achieve?

Depending on connector, they are discovered or defined when configuring new source

3

Yes, this is difficult from a terminology standpoint. It depends on what you mean. I could see the following interpretations:

  1. Making an already available stream active on a connection? Yes. (This would be handled in the StreamConfigurations when creating/patching the Connection.)
  2. Running a discovery to see if here are changes in the streams/schemas? Yes. (You’d get the stream properties with ignoreCache=false)
  3. Creating a new stream that doesn’t already exist? Typically no (although there are some dynamic configs for certain connectors that take a report spec)
    If you’re talking about actually modifying a source connector definition like you would in Builder, that’s different and unlikely to be covered in the Airbyte API (but may be able to be achieved with the unsupported config API the Airbyte frontend uses). It’s worth noting that it wouldn’t just be adding the stream though; you’d also have to publish the new connector version and set it as default for existing connections, and then run a discovery on each and select the stream.
4

Thank you so much <@U035912NS77> this solved it for me!