Summary
The user is facing an ‘IllegalStateException: Get Spec job failed’ error when trying to install a custom connector using a Dockerfile on Airbyte instance. The error persists even when using ‘airyte-ci’ to build the image.
Question
Hello everyone.
I am trying to build my first custom connector and after preparing the custom Dockerfile, I was not able to install it on my Airbyte instance.
I got the following error message:
java.lang.IllegalStateException: Get Spec job failed.
I thought the problem could be related to the Dockerfile I created, so I decided to use the airyte-ci to build my image. It did not work and I got the same error message.
The Dockerfile is working as expected when I execute it.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connector", "dockerfile", "illegalstateexception", "get-spec-job", "airbyte-ci"]
I’m sorry to hear that you’re having trouble with your custom connector. The error message java.lang.IllegalStateException: Get Spec job failed is a common issue when trying to add a custom connector to Airbyte. Here are a few things you could check based on the information from the extracts: 1. Docker Image Naming: Ensure that your Docker image name does not contain any “bad” symbols like _ and does not have stacked folders in the Docker repository [source]. 2. Docker Platform: If you’re using Docker on different platforms (like Mac and Windows), there might be compatibility issues. Make sure that your Docker platform is compatible with the one used to build the image [source]. 3. Docker Image Tag: If you’re using the latest tag for your Docker image, try switching to a real version number [source]. 4. Private Docker Registry: If your Docker image is hosted on a private registry, make sure that it’s pulled on the machine prior to upgrading the connector [source]. 5. Custom Dockerfile: Some connectors like destination-s3 have custom Dockerfiles for the installation of system dependencies. If your connector is one of them, you might need to implement custom build logic for it [source]. If none of these suggestions work, you might want to check the server logs for more detailed error messages. If you’re still stuck, consider reaching out to the Airbyte community on their discussion forum or GitHub for further assistance.
<@U04SE163WC8> It looks like the problem is related to the worker not being able to find the Docker image:
Could not find image: ...
I am using Azure container registry to store the image and I followed this guide to make the authentication: https://docs.airbyte.com/operator-guides/using-custom-connectors#on-azure---container-registry
Based on the information provided, it seems like you’ve done the correct steps to authenticate to your Azure Container Registry using the az CLI as mentioned in the Airbyte guide. However, it’s important to note that you need to authenticate both your local/CI environment (where your image is built) and your Azure Virtual Machine instance where the Airbyte instance is running. This is to ensure that both environments have the necessary permissions to access and pull the Docker image from your Azure Container Registry. If you’ve done this and you’re still encountering issues, it might be worth checking the following: 1. Ensure that the Docker image you’re trying to pull actually exists in your Azure Container Registry and that the image name and tag are correct. 2. Check if there are any network restrictions or firewall rules that might be preventing your environments from accessing the Azure Container Registry. 3. Try pulling the Docker image manually from your environments to see if there are any specific error messages that might give more insight into the issue. If the issue persists, you might want to consider reaching out to the Airbyte community or support for further assistance.
Hey <@U0668M7G22W> - one thing to check would be the architecture of your docker image versus the architecture of the machine running your airbyte (e.g. if you have linux VM cluster running airbyte, but built the image on a mac)
Based on your “not able to find” issue I’m not 100% sure if thats the case (I’m not very familiar with the azure container registry setup), but still probably worth checking
The image is being built using the azure CLI: az acr build , so the build is following the Azure architecture
A decent sanity check might be connecting to your cluster, execing into a pod and trying to pull the image directly from there
I am a docker-compose deploy on a VM
We have tried to deploy into our K8s cluster but it looks like Azure is not supported yet.
In that case, doing the same but in a container (e.g. the container-orchestrator container) would work
For me it looks like the worker is trying to pull the image from another repository and not the one that was logged in the VM. I am not sure if I need to change something in the docker-compose to make this repo accessible inside the containers
because I am able to run the image from the VM.
> pull the image from another repository and not the one that was logged in the VM
With what pull command? you’ll have to specify the repository (including the full azure repository path)
I think I managed to make it work. It was necessary to access the worker container and make the docker login from there. After that, I as able to install the image into Airbyte.
Ah, glad you got it worked out!
I am just checking if there is something to be updated in the documentation
I guess the
> You will have to authenticate both your local/CI environment/ environment (where your image is built) and your Azure Virtual Machine instance where the Airbyte instance is running
Was too specific to your use case as you aren’t running airbyte in an Azure Virtual Machine
It should probably just say that you need to authenticate both your local building environment and your airbyte instance puling the image
I am running Airbyte in a Azure VM using docker-compose.
So I don’t understand why the instructions did not work on my case.
So you’d already authed in your azure VM? If so, I’m not sure what went wrong, as I’m not familiar, but I could send the feedback along to the right team.