Hi,
I am considering using an orchestrator to trigger airbyte syncs.
I wanted to follow this post Using the Airflow Airbyte Operator | Airbyte Documentation
but it mentions an HTTP request sent to localhost. How can I adapt it so that Cloud Composer can launch a sync in an Airbyte instance that is deployed on a different server ?
Note that Cloud Composer is the Google Cloud managed Airflow service, so the question is the same for other Airflow deployments.
Thanks
You must allow Airflow Composer be able to connect to your Airbyte server. You must change the network settings between both instances. There is a demo hour I showed how to do it: https://www.youtube.com/watch?v=ouQNYQDWI28
Thanks @marcosmarxm , it’s exactly what I needed.
Though in my case, after I created the Cloud Composer environment I don’t see any new VM in my list of GCP VM Instances. It’s like they are hidden or not showing. Hence, I am not able to see their IP and so I cannot whitelist them to access my Airbyte server.
Hi @marcosmarxm ,
Do you see any reason for this ?
I suspect that it is because I am using Composer v2 with autoscaling and hence workers are created dynamically and don’t have fixed IP.
Have you faced this issue ? Do you see anyway to circumvent it ?
Thanks
Hey @anatole @marcosmarxm I have the same issue with composer v2. Were you able to figure it out?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.