Summary
When building a custom connector with 2 streams, the user is facing issues providing different values of properties inside secrets/config.json for each stream. The flow executes successfully for the first stream but throws an error for the second stream.
Question
hi! I am building a custom connector with 2 streams. how can I provide different values of properties inside secrets/config.json for these streams. When I am running this command python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json the flow is successfully executing for first stream and then throwing error for the second stream.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connector", "multiple-streams", "config.json", "python", "error"]
my integration_tests/configured_catalog.json is configured properly but the problem is with secrets/config.json file since it expects only one set of input.
Can you give more information about your use case, how you built your spec and is trying to consume the config parameters?
Yeah sure!
So my source connector has 2 streams (A&B) both with different schemas but same set of credentials. The value of the credentials differ for each of them.
My spec has account id and token. When I am running python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json. , it is getting run for both the streams and the credentials are being fetched from secrets/config.json file which only has credentials for stream A, hence when it is using this credentials for stream B it is failing.
Maybe a solution is to have:
"apikey_endpoint_a: "...",
"apikey_endpoint_b: "..."
}```
Without more information or sharing more about the structure is not possible to help you
i have created an API based source connector A, which has 2 streams A1 and A2. Both the streams need to request the same API with same set of parameters. The only difference is the value of parameters, Stream A1 has to send { AccountID: 1, Token: XXX } while Stream A2 has to send { AccountID: 2, Token: YYY } . I have already built the custom connector and now when I am trying to run python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json this command, it is reading the credentials from config file and I can only provide one config at a time. Is it possible to pass more than one config here, if yes then how and if no then how should i handle the failure ?