Assigning Environment Variables to Custom Sources

Hi everyone, we have around 5 connectors that we built using the CDK, and they all require the same set of ENV Variables. What is the best way to handle this? We can add them to the docker files for each environment but it would be better if there was a way to assign these variables without having to define the variables in the docker image. Also it is not something that we want the users to see so we can’t use it as part of the spec of the container

Hey when you say same ENV variables if you meant configuration then you can handle it with spec.json and pass them over API https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/sources/create

Hey, actually its not really related to the spec, as these are basically some environment related information related to the K8 Cluster that are needed by the Connector. I didn’t want to include them in the Spec file as they would have to be defined per Source, while they are more of a Source Definition kind of setting , not sure if this is clear

Also if there was any updates needed then I would have to go through all the sources and update them, so its more of a Connector configuration rather than a Source configuration. I am currently hardcoding them into the docker file for the custom connector , but I would rather have a cleaner way of handling that.

Got it.

  1. If it is constant for source right now best way is to hardcode in the source docker
  2. If it changes then handle it in the spec

We don’t have a out of box solution for this would suggest you to create a github issue for this so that team can comment

Alright , so I guess for now the workaround i implemented should do the trick, and will just rely on the proper tags for the build pipeline when pushing images to the various environments.

Thanks a lot for your help