Syncs and connections tests start failing suddenly

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu / GCP VM / n1-standard-2
  • Memory / Disk: n1-standard-2
  • Deployment: Docker
  • Airbyte Version: 0.39.21-alpha
  • Source name/version: Shopify 0.1.37
  • Destination name/version: BigQuery 1.1.9
  • Step: The issue is happening during sync
  • Description: I have been using Airbyte to sync data from Shopify and Google Sheets to BigQuery. It has been running smoothly for almost 5 months. Since the 26th of November, all the syncs, source tests and destination tests are failing. Not sure whether it’s related to the cause of the issue, but two days before syncs started failing I changed the frequency of the syncs for the Shopify connections from 6 to 3 hours. Also, Since the the 26th of November, I noticed a new error message when navigating the UI, saying: “Cannot reach server. The server may still be starting up”. This message appears randomly after a few clicks on the UI.
  • Could you please advise on how to fix this issue?
  • If I decide to restart the machine, am I going to lose all the connections credentials?
  • If I decide to update the destination and the source, is it going to trigger a full refresh sync even though the schema of the source didn’t change? Indeed, I am currently using an incremental sync mode.

I have attached the UI logs of the first attempt that failed.
logs-1456_attempt_1.txt (769.9 KB)

When trying to get the server logs in a file using the command docker logs airbyte-worker > airbyte-worker.log I get the following error: No space left on device . I therefore ran docker logs airbyte-server | grep ERROR and copied the results in this file: server_logs_airbyte.txt (423.7 KB)

Moreover, I did run docker system prune but it didn’t solve the problem.

UPDATE: Currently I am not even able to have the UI displayed. The page keeps loading.

Thanks for your help!
Jaafar

3

Hello, i have found the question about the support to fix the issue from the upwork.
As i can see from the log it is looks like on closed connection from a while. You need to keep connection open / reopen. A superficial look, since I do not know the architecture. Also, you need to check the process of the write and close to file ang clean operative memory to avoid stack overflow.
Check the container by this tutorial and it can take to you some more information

4

@datasquad-ai did you check for docker cache?

5

Thanks for your help. After some investigation, our hypothesis is that the jobs database grew too much and there was no more space in the disk. As a consequence, the airbyte-db container went down. It probably means I will have to set up again all the sources, destinations and connections. Also, it means I will have to resync 3+ years of data history which may take more than 48h…

It would have been great to know about the need to truncate manually the jobs database. I thought it was done automatically. I really don’t need to keep 5 months of jobs history…

If you see a way to restart the airbyte-db container without losing all the connections data, please let me know.

7

Hello jaafar, it’s been a while without an update from us. Are you still having problems or did you find a solution?

8

For now we have been deleting the logs folders manually to free space in the instance. We implemented the variable that deals with deleting the logs after X number of days in the yml file and updated Airbyte to 0.40.22.

We will let you know if it works.