Deprecation of Docker Compose in Airbyte 1.0 release and performance issues with abctl

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte is deprecating Docker Compose in future versions after 1.0 release, leading to questions about the functionality of deployments using Docker Compose post-deprecation. Additionally, a user is experiencing significant performance issues with abctl compared to Docker Compose, with long sync times and failures for a small number of records.


Question

I know that docker compose is being deprecated (or is deprecated?) and all https://github.com/airbytehq/airbyte/discussions/40599|communications reference after 1.0 release:
> Today, we are announcing that Docker Compose will be deprecated from future versions of Airbyte OSS after version 1.0 is released (expected in early August 2024).
> Support for Docker Compose and running Airbyte via the run_ab_platform.sh script will end in August 2024.
I got an email today that 1.0 release is set for September 24, 2024. Does this mean that deployments using docker compose will no longer function after this date? Or just that updates will no longer be issued?

We tried to migrate to abctl using the limited resources command option but consistently see sync times of 3+ hours for an incredibly small number of records (10-20 records), and they often fail anyway. In comparison, using docker compose, we can sync 800k+ records in five minutes. It seems truly infeasible to move to abctl from docker compose, given the number of problems we face there and the extremely long load times in comparison to docker compose. Is anyone else experiencing extremely long load times using abctl? Are there any known solutions to this?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["docker-compose", "deprecation", "1.0-release", "abctl", "performance-issues", "sync-times", "load-times"]
2

Hey Cailey, sorry to hear that. Do you mind sharing what is the configuration of server (cpu/memory)?

3

<@U07FZ6RPCR1> do you have this info?

4

Old versions of Airbyte won’t stop working. But you won’t be able to get updates and new versions of connectors.

5

When did you try to migrate? There were some known performance issues in the beta versions of abctl, but we’ve now done performance testing such that we’re confident that abctl can always be faster that its docker compose predecessor. To accomplish that there may be a environment variable for resources that needs to be set. We’d be happy to work with you to get that configured. It’s probably something we could document better as well.

6

hello!
We are using an t3.large EC2 instance. We were using docker and after the depredcation advice we went for abctl with --low-resource-mode

7

we migrated and also updated abctl to it’s latest version

8

after updating the abctl version the syncs take more than 1 hour and the output is this