I am considering using an orchestrator to trigger airbyte syncs.
I wanted to follow this post Using the Airflow Airbyte Operator | Airbyte Documentation
but it mentions an HTTP request sent to localhost. How can I adapt it so that Cloud Composer can launch a sync in an Airbyte instance that is deployed on a different server ?
Note that Cloud Composer is the Google Cloud managed Airflow service, so the question is the same for other Airflow deployments.
You must allow Airflow Composer be able to connect to your Airbyte server. You must change the network settings between both instances. There is a demo hour I showed how to do it: https://www.youtube.com/watch?v=ouQNYQDWI28
Thanks @marcosmarxm , it’s exactly what I needed.
Though in my case, after I created the Cloud Composer environment I don’t see any new VM in my list of GCP VM Instances. It’s like they are hidden or not showing. Hence, I am not able to see their IP and so I cannot whitelist them to access my Airbyte server.
Hi @marcosmarxm ,
Do you see any reason for this ?
I suspect that it is because I am using Composer v2 with autoscaling and hence workers are created dynamically and don’t have fixed IP.
Have you faced this issue ? Do you see anyway to circumvent it ?