Parameterized Airbyte job

Q&A
1

Hi, a question about how to do:
I have a case that requires running a job to extract data from Stripe, same job, but for multiple different Stipe accounts, i.e. different tokens.
The list of accounts (tokens) might be changed over time.
How to run an Airbyte job with a different token parameter each time?

2

Hey @datayoshi you can create sources for individual stripe accounts and use them to pull the data. In case you don’t need them later you can delete the source. You can use API if that can help

4

Thanks @harshith ,
I would like to avoid manually creating an individual job for every account…
Is there a way to achieve it automatically? - let’s say by creating a single job that has {tokan} parameter, receiving the value from a file or similar

5

I don’t think we have an out of box solution for this. But would suggest to extend the current connector into your need and use it as a custom connector if that can help

6

Can I attach AirFlow as an orchestrator which executes Airbyte’s same job with a different stripe_acc_token each time?

7

You can create it in 2 flows as for my knowledge

  1. You can use the API to configure that create source in runtime and then delete it later
  2. Create a custom connector which can take a bunch of ids and run all of those every time you start sync
8

Can you please direct me to the docs about option (1) API ?

9

https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/workspaces/create here you go

1 Like
10

Hey @datayoshi,
I’d suggest you try out our CLI. With a bit of scripting on top of it, you might be able to automatically generate multiple Stripe sources, one for each account you have to deal with.

11

Thanks @harshith @alafanechere
Are there any limitations around spawning sources via the CLI ?
such as: performance, amounts, etc…

12

Hey @datayoshi didn’t get you over performance? Ideally it’s hitting the API thus there are no limitations

13

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.