Error adding custom connector to local Airbyte instance

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

When trying to add a custom connector to a local Airbyte instance, an ‘Internal Server Error: Get Spec job failed’ is returned. The error seems to be related to fetching the spec from the custom connector.


Question

Hi! I built a custom connector outside of the airbyte repo and have its docker image published. When I try to add it to my local airbyte it resturns Internal Server Error: Get Spec job failed.

    "message": "Internal Server Error: Get Spec job failed.",
    "exceptionClassName": "java.lang.IllegalStateException",
    "exceptionStack": [
        "java.lang.IllegalStateException: Get Spec job failed.",
        "\tat com.google.common.base.Preconditions.checkState(Preconditions.java:502)",
        "\tat io.airbyte.commons.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:17)",
        "\tat io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.getSpecForImage(ActorDefinitionHandlerHelper.java:153)",
        "\tat io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromCreate(ActorDefinitionHandlerHelper.java:72)",
        "\tat io.airbyte.commons.server.handlers.DestinationDefinitionsHandler.createCustomDestinationDefinition(DestinationDefinitionsHandler.java:240)",
        "\tat io.airbyte.server.apis.DestinationDefinitionApiController.lambda$createCustomDestinationDefinition$0(DestinationDefinitionApiController.java:66)",
        "\tat io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28)",
        "\tat io.airbyte.server.apis.DestinationDefinitionApiController.createCustomDestinationDefinition(DestinationDefinitionApiController.java:66)",
        "\tat io.airbyte.server.apis.$DestinationDefinitionApiController$Definition$Exec.dispatch(Unknown Source)",
        "\tat io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371)",
        "\tat io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594)",
        "\tat io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)",
        "\tat io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)",
        "\tat io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)",
        "\tat io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)",
        "\tat reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)",
        "\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)",
        "\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)",
        "\tat io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62)",
        "\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)",
        "\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)",
        "\tat io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)",
        "\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)",
        "\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
        "\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
        "\tat java.base/java.lang.Thread.run(Thread.java:1583)"
    ]
}```
But when i pulll the docker image and run the spec command it works just fine.

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1707845710859589) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["custom-connector", "docker-image", "internal-server-error", "get-spec-job-failed"]
</sub>
2

The “Get Spec job failed” error you’re encountering when trying to add your custom connector to Airbyte can be due to several reasons. Here are some potential issues and solutions based on the information extracts: 1. Image Naming: Ensure that your Docker image name doesn’t contain any “bad” symbols like underscores or stacked folders in the Docker repository. This was mentioned in a forum post. 2. Docker Image Tag: If you’re using the latest tag for your Docker image, try switching to a real version number. This was a solution provided by a user in a Github issue comment. 3. Dependencies: Make sure all necessary dependencies are included in your setup. A user in a forum post solved the issue by adding certain dependencies to their setup.py. 4. Architecture Compatibility: If you built your Docker image on a different architecture (e.g., Mac M1), it might have issues running in Airbyte. You can try building the image on your current machine or in the cloud to work around this issue. This was suggested by a support agent in a forum post. If none of these solutions work, you might want to check the server logs for more detailed error messages or post your issue on the Airbyte Github or forum for further assistance.

3

If you want to post the spec output, we could take a look. Likely something in the spec format or fields is not compatible?

4

it was an arch issue! my machine is arm64 and my Docker image was for amd64. Thanks!