Error with Shopify connector in self-managed Airbyte on Azure VM

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Error ‘could not find image: airbyte/source-shopify:2.0.8’ occurs after 12-24 hours in automated job. Manually syncing after restarting docker image resolves the issue. Need help identifying the root cause.


Question

Hi I’m using Shopify connector in the self-managed version of Airbyte deployed on an azure VM.
Once in a while, usually after 12-24 hours, I will get a “could not find image: airbyte/source-shopify:2.0.8” error on an automated job.
I have setup 128gb memory on my vm. And if I down then up the docker image, and manually sync, it will be able to find the image and proceed with the sync.
Can’t figure out what the issue is. Any ideas?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["shopify-connector", "self-managed", "airbyte", "azure-vm", "docker-error"]
2 
2024-05-11 20:19:19 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-05-11 20:19:19 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-05-11 20:19:19 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-05-11 20:19:19 platform > Checking if airbyte/source-shopify:2.0.8 exists...
2024-05-11 20:19:19 platform > airbyte/source-shopify:2.0.8 not found locally. Attempting to pull the image...
2024-05-11 20:19:19 platform > Image does not exist.
2024-05-11 20:19:19 platform > Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/source-shopify:2.0.8
        at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:117) ~[io.airbyte-airbyte-commons-worker-0.59.0.jar:?]
        at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:149) ~[io.airbyte-airbyte-commons-worker-0.59.0.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.59.0.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.59.0.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.59.0.jar:?]
        at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:226) ~[io.airbyte-airbyte-workers-0.59.0.jar:?]
        at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.59.0.jar:?]
        at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:211) ~[io.airbyte-airbyte-workers-0.59.0.jar:?]
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]```
3

<@U072UGA0162> hi I am facing the exact same issue right now. Did you manage to find any solution?