While testing locally, I’ve been passing in tokens using docker run -E <secret_token> to connect to some internal stuff, but I don’t want to include these secrets in the dockerfile for my connector, as keeping secrets in images is generally bad practice. Also have some specific mounts that I’m currently using with --mount
Is there a method where I can provide these environment variables when starting up an airbyte instance w/ docker-compose that will be available to the connectors, or what is the suggested workflow for this usecase?I intentionally do not want these to be passed in by the user as part of the spec.json
Hey if this is about source or destination and you want to connect with them using specific keys?
Sort of – I’m not sure I understand your question.
Example: I’m developing a customer connector which will communicate with an internal endpoint, my_endpoint.com/api/v1. I want this endpoint to be configurable, but NOT by the user, only by the airbyte operator. So lets say we also have a my_endpoint.com/api/v2, I would want to be able to change an environment variable on the machine airbyte is running as an airbyte operator to modify this endpoint, without the user knowing (or caring)
Got it, I don’t think we have an out of box solution for this. How about keeping it in the config and you set it through API while create the source? We have an API doc here
Thats an interesting idea @harshith, could you elaborate on how to do this for a destination connector (specifically in java in this case) for example? So it’s in the config (spec.json) as optional, but then I fetch the env variable when actually saving the destination setup?
I’m assuming the place to inject the env fetching would be before the /v1/destinations/create call happens, but im not really sure how the actual “destination creation” works. This approach you suggest could work as long as the env fetching only happens during config creation and not during nominal execution of the connector
Yeah if you want to do that then you have to write custom code in the airbyte-server container. That’s where the request would go and the destination is created.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.