Sync Job Not Starting After Upgrade to Airbyte v1

slack-user-airbyte · December 9, 2024, 2:34pm

Summary

User reports an issue with sync jobs not starting after upgrading to Airbyte v1 using abctl. Kubernetes pods show errors, particularly with source-postgres-check pods indicating ImagePullBackOff and other errors. User seeks guidance on troubleshooting the issue.

Question

Hello,

I have upgraded to Airbyte v1 using abctl but now, the sync wont start. Looking at the kubeters pods there seems to be an error, though it not clear what the issue might be.

NAME                                                     READY   STATUS             RESTARTS       AGE
airbyte-abctl-airbyte-bootloader                         0/1     Completed          0              19h
airbyte-abctl-connector-builder-server-f7b8c4d7b-6w7qh   1/1     Running            0              19h
airbyte-abctl-cron-7f6dc9484-2d98n                       1/1     Running            1 (18h ago)    19h
airbyte-abctl-pod-sweeper-pod-sweeper-856cf644f9-zf2tl   1/1     Running            0              19h
airbyte-abctl-server-8477ffd8d6-97xrx                    1/1     Running            12 (18h ago)   19h
airbyte-abctl-temporal-857dd9f6f4-xkwtq                  1/1     Running            0              19h
airbyte-abctl-webapp-cccb469b8-b95rb                     1/1     Running            0              19h
airbyte-abctl-worker-54df9686cc-qnvzq                    1/1     Running            0              19h
airbyte-abctl-workload-api-server-6867bffd8b-b2mvf       1/1     Running            0              19h
airbyte-abctl-workload-launcher-5f6bc4cd55-nnpl4         1/1     Running            0              19h
airbyte-minio-0                                          1/1     Running            0              19h
source-postgres-check-22394-1-ikhqn                      0/2     Error              0              120m
source-postgres-check-22394-2-oyrpg                      0/2     Error              0              110m
source-postgres-check-22394-3-mpzgz                      0/2     Error              0              99m
source-postgres-check-22394-4-dmcio                      0/2     Error              0              86m
source-postgres-check-22395-0-wvktf                      1/2     ImagePullBackOff   0              85s```
It there a way to dig deeper into the issue?

As for the sync job logs, they stay in a loop like this:

```2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CLAIM — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-11-01 14:39:38 INFO i.a.w.l.c.WorkloadApiClient(claim):75 - Claimed: true for decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check via API for local
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CHECK_STATUS — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.CheckStatusStage(applyStage):59 - No pod found running for workload decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check. Continuing...
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - ```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1730472253783009) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['airbyte-v1', 'abctl', 'kubernetes', 'sync-job', 'imagepullbackoff', 'error']
</sub>

slack-user-airbyte · December 9, 2024, 2:48pm

you can debug it with k9s

Install kubectl (https://kubernetes.io/docs/tasks/tools/#kubectl) and kubectx + kubens (https://github.com/ahmetb/kubectx) and k9s (https://k9scli.io/)

then execute commands:

KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten &gt; ~/.kube/config
kubectx kind-airbyte-abctl
kubens airbyte-abctl
k9s```
with k9s you can faster which pods are not running, their logs, etc.

slack-user-airbyte · December 9, 2024, 2:48pm

I wouldn’t be surprised if you hit limits for pulling from Docker Hub

slack-user-airbyte · December 9, 2024, 2:48pm

check this thread
https://airbytehq.slack.com/archives/C021JANJ6TY/p1729100805535889?thread_ts=1729097397.162399&cid=C021JANJ6TY

slack-user-airbyte · December 9, 2024, 2:48pm

another option could be https://airbytehq.slack.com/archives/C021JANJ6TY/p1728562294190379?thread_ts=1728561176.853139&cid=C021JANJ6TY
get more details, so you can narrow down the issue

slack-user-airbyte · December 9, 2024, 2:50pm