Airbyte CICD Best Practice

Q&A
1

Hi Folks,

I’m curious as to the approaches to SLDC people are using, i’m specifically interested in using Azure DevOps pipelines. I can see that we can use Octavia CLI to maintain a repo of configs that could could be deployed from dev to prod airbyte instances, assuming state files would be ignored. Are people also using airbyte API to stop existing connections before applying updates? Bit confused.

Thanks,
Y

2

What is SLDC? You can check this repo https://github.com/airbytehq/airflow-summit-airbyte-2022 it’s using Airbyte/Airflow/dbt with CI/CD

3

Hi,
SDLC - Software Development Lifecyle

So what i’m thinking is, having two instances of Airbyte;
dev_test
prod

A manual SDLC would involve setting up dev sources, dev destinations and connections in UI or via Octavia CLI in dev_test, complete testing (for example check data in destination). Then in order to push to production we would mimic the above but use prod sources, prod destinations in our prod instance.

I’m curious as to how people are applying devOps for this process. I’m conscious that the export config > import config wouldn’t work here as we would need to change source/destination references from dev to prod and we would not want any state information overwritten in prod with dev source state information.

Perhaps i’m over complicating things in my head…

4

Yep, did you check the repo I share with you? It’s one example where we can have a dev/prod with same connectors and different credentials

5

Ah right. Thanks I’ll review this evening. :+1:

6

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.