Error with api/v1/source_definitions/create when running on GCP

Troubleshooting
#1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Linux(GCP compute engine)
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.39.5-alpha
  • Source name/version: Docker Hub
  • Destination name/version:
  • Step: The issue is happening during importing the connector
  • Description:
    Hi, I’m facing an issue while importing an home-built connector to an airbyte instance running on GCP compute engine.
    Once I clicked on the add button, I get a 500 error:
    java.lang.IllegalStateException: Get Spec job failed.

I trouble shooted the issue on my side:

  • CE instance can connect to docker hub as it’s working with other containers
  • I can import the mentionned container on airbyte running locally (:sweat_smile:)
  • The API call is not returning more information than the message added to the description
  • I’m exploring the possibility of a blocked process by the firewall …

Do you have any idea of what’s happening ? Or an idea to restrict the search scope ?

Thanks a lot for your help,

#2

The issue could be due to the OS/Arch (linux/arm64/v8) of this image.
It would explain why it’s working locally and not remotely

#3

@alexisc could you try to run docker run connector-image:tag spec on your GCP VM and share the container output?
It might indeed be an architecture-related problem.

#4

It’s confirmed it was due to an image built on a Apple M1 not running properly on the VM.
It will probably be interresting to display a more explicit error than Get Spec job failed.
Display the docker run error ?

This is not an issue with airbyte. Sorry for the noise!
Thanks for the support though

#5

@alexisc you might want to use Docker Buildx to create a multi architecture image for your connector.
And you are right that the error message should be more explicit!