Destination Snowflake - Normalization Architecture Exec Issue

Connector Questions & Issues
, , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Amazon Linux 2
  • Memory / Disk: 16Gb
  • Deployment: Kubernetes (EKS)
  • Airbyte Version: 0.40.2
  • Source name/version: Postgres 1.0.4
  • Destination name/version: Snowflake 0.4.34
  • Step: The issue is happening during sync
  • Description:

We’re deployed on t4g arm64 instances in EKS and trying to sync data from postgres to snowflake. However, the normalize step seems to be stuck. After looking at the pods in k8s, I can see an error on the main container for our pod normalization-snowflake-normalize-72-0-dsyle and logs stating standard_init_linux.go:228: exec user process caused: exec format error which makes me think it’s a processor architecture issue, but the airbyte/normalization-snowflake:0.2.18 image hosted on dockerhub seems to be tagged for both amd64 and arm64.

NAME                                           READY   STATUS      RESTARTS   AGE
normalization-snowflake-normalize-72-0-dsyle   3/4     Error       0          74m

However, on my M1 mac I can run docker run -it --platform=linux/arm64 --entrypoint sh airbyte/normalization-snowflake:0.2.18 and navigate the image and attempt to run the entrypoint script without seeing the exec format error.

Additionally, we have check and write pods for snowflake that completed without issue:

NAME                                     READY   STATUS      RESTARTS   AGE
destination-snowflake-check-72-0-dkxwu   0/4     Completed   0          79m
destination-snowflake-write-72-0-cxghx   0/5     Completed   0          79m

Image info from the pod:

image: airbyte/normalization-snowflake:0.2.18
imageID: docker-pullable://airbyte/normalization-snowflake@sha256:d6d5ddbeb457a90d16a204e6baf73798bbd4be605d3c9ffa3e1537aa9aab2527

Let me know if I can provide any additional information, any advice is greatly appreciated.

2

Hello, thanks for the thorough bug report.

We are aware of this issue and it is currently being evaluated. All progress updates will be reported here: https://github.com/airbytehq/airbyte/issues/15498

You’re right that it is specific to architecture, this issue appears to only affect arm64 systems that do not use Rosetta (like Mac M1) or possibly if they use a different compatibility layer.

Let me know if you have any further questions.

1 Like