Error adding a source (custom connector) to UI

Connector Development
1

Hi,

I know there were similar questions asked, but the proposed answers don’t seem to help in my case.

I’ve developed a custom connector some time ago and now need to improve it. However, I can’t get it uploaded to my Airbyte instance. During the period between the last time I’ve got it working and now I’ve changed my PC I work on - this might be relevant. I’m now using M1 Mac.

What I did: first tried to copy a source folder to my local instance of Airbyte, create a docker image, push, update on my target Airbyte instance. Doesn’t work. I get the following unhelpful error:

“Get spec job failed.”

Ok, I then went through generator to create a new source, copied out all the code, schemas, specs etc. and did that carefully.

I then test it locally (through venv, command line) and it seems to work. I build a docker image, push it, and then try to update my connector - same error again.

Since the error seems to be absolutely generic (same error if I enter a random docker repo linking to a non-existing image), I’ve tried accessing server logs - no success, still some generic “unhandled exception” error.

One suspicious thing that I’ve noticed is that the current image is ±28 mb. while the old one used to be around ±37 - but I am an idiot what concerns Docker, so this doesn’t indicate to me anything particular.

Any ideas? THANKS!

P.S. What is NOT the case here:

  1. Image naming - no stacked folders in Docker repo, no “_” or other “bad” symbols
  2. Issues with actual specs - docker run --rm -i repo/image-name:tag spec goes through
2

UPDATE:

OK, so one definite issue was that on M1 docker by default builds on arm processor. I’ve changed that using experimental features. Get a nice little image (actually, the size is back to what it once was). Still no success adding it - same “Get spec job failed” error.

3

UPDATE 2

Turns out, it was it, after all. Maybe this thread will be helpful for others building on M1, M2 or other arm processors.

What you need to do:

  1. Go to docker desktop settings. Find “Experimental features”. Enable.
  2. run docker buildx create --use
  3. build and push your image using
    docker buildx build --platform linux/amd64 --push -t repo/image-name:tag .

In my case, tag overwriting didn’t work, use a distinct tag.