Parameterized Airbyte job

Hi, a question about how to do:
I have a case that requires running a job to extract data from Stripe, same job, but for multiple different Stipe accounts, i.e. different tokens.
The list of accounts (tokens) might be changed over time.
How to run an Airbyte job with a different token parameter each time?

Hey @datayoshi you can create sources for individual stripe accounts and use them to pull the data. In case you don’t need them later you can delete the source. You can use API if that can help

Thanks @harshith ,
I would like to avoid manually creating an individual job for every account…
Is there a way to achieve it automatically? - let’s say by creating a single job that has {tokan} parameter, receiving the value from a file or similar

I don’t think we have an out of box solution for this. But would suggest to extend the current connector into your need and use it as a custom connector if that can help

Can I attach AirFlow as an orchestrator which executes Airbyte’s same job with a different stripe_acc_token each time?

You can create it in 2 flows as for my knowledge

  1. You can use the API to configure that create source in runtime and then delete it later
  2. Create a custom connector which can take a bunch of ids and run all of those every time you start sync

Can you please direct me to the docs about option (1) API ?

https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/workspaces/create here you go

1 Like

Hey @datayoshi,
I’d suggest you try out our CLI. With a bit of scripting on top of it, you might be able to automatically generate multiple Stripe sources, one for each account you have to deal with.

Thanks @harshith @alafanechere
Are there any limitations around spawning sources via the CLI ?
such as: performance, amounts, etc…

Hey @datayoshi didn’t get you over performance? Ideally it’s hitting the API thus there are no limitations