Trying to add a .NET 6.0 custom destination connector, error with connector Docker container

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte? Yes
  • OS Version / Instance: Windows 11
  • Memory / Disk: 32Gb, 500Gb
  • Deployment: Docker
  • Airbyte Version: 0.40.32
  • Source name/version: NA
  • Destination name/version: Custom .NET 6.0 connector
  • Step: Trying to add a new destination connector
  • Description:
    Hello,

I’ve been testing the capabilities of Airbyte for the last week or so. My goal has been to get it set up in a demo environment, reading changes from a Mysql database, and outputting those changes to a .Net Core application. I’ve been slowly deciphering the setup docs and the .Net example repo, and have got to a point where I seem to have a tiny test application running in Visual Studio. I can make calls to spec and write and they seem to respond as expected.
I’ve created a new Docker container with that application, and am hosting it in a local Docker registry.
I can run this container manually, and the application starts up. (It actually gives me the command line help text requesting the command/config/catalog arguments, so I know it’s capable of running even if it’s not getting further than that. I simply haven’t supplied the arguments to get further.)

When I try to load it in Airbyte, the “Add new connector” modal gives the error “Internal Server Error: Get Spec job failed.”.
In the airbyte-worker logs, there’s the error:

airbyte-worker                    | 2023-02-16 09:40:55 INFO i.a.w.p.DockerProcessFactory(create):125 - Creating docker container = localhost-spec-54d41f18-d699-4c2b-aae6-b7c6dc62b1e8-0-hlnay with resources io.airbyte.config.ResourceRequirements@688a62cb[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null
airbyte-worker                    | 2023-02-16 09:40:55 INFO i.a.w.p.DockerProcessFactory(create):170 - Preparing command: docker run --rm --init -i -w /data/54d41f18-d699-4c2b-aae6-b7c6dc62b1e8/0 --log-driver none --name localhost-spec-54d41f18-d699-4c2b-aae6-b7c6dc62b1e8-0-hlnay --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_CONNECTOR_IMAGE=localhost:5000/airbyteconnectortest:latest -e WORKER_JOB_ATTEMPT=0 -e AUTO_DETECT_SCHEMA=true -e AIRBYTE_VERSION=0.40.32 -e WORKER_JOB_ID=54d41f18-d699-4c2b-aae6-b7c6dc62b1e8 localhost:5000/airbyteconnectortest:latest spec
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 - The command could not be loaded, possibly because:
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 -   * You intended to execute a .NET application:
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 -       The application 'AirbyteConnectorTest.dll' does not exist.
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 -   * You intended to execute a .NET SDK command:
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 -       No .NET SDKs were found.
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 - Download a .NET SDK:
airbyte-worker                    | 2023-02-16 09:40:56 ERROR i.a.c.i.LineGobbler(voidCall):114 - https://aka.ms/dotnet-download

It looks like its not finding the correct version of dotnet? I’m not sure if this is the worker itself erroring, or if it’s my container erroring, but as I said, running my container manually is working.

I wondered if anyone would have any suggestions?

Thanks,

Mike

2

I have a sneaky suspicion that the worker sets the working directory deeper inside the container, but my connector is in it’s default location, hence why it can’t find it.

3

It’s definitely the working folder. If I run the same command as the worker from my command prompt, I get the same error, can’t find the executable. That command is:

docker run --rm --init -i -w /data/54d41f18-d699-4c2b-aae6-b7c6dc62b1e8/0 --log-driver none --name localhost-spec-54d41f18-d699-4c2b-aae6-b7c6dc62b1e8-0-hlnay --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_CONNECTOR_IMAGE=localhost:5000/airbyteconnectortest:latest -e WORKER_JOB_ATTEMPT=0 -e AUTO_DETECT_SCHEMA=true -e AIRBYTE_VERSION=0.40.32 -e WORKER_JOB_ID=54d41f18-d699-4c2b-aae6-b7c6dc62b1e8 localhost:5000/airbyteconnectortest:latest spec

If I simply remove “-w /data/54d41f18-d699-4c2b-aae6-b7c6dc62b1e8/0” it can find the executable. I’m not sure what I need to do to either stop Airbyte using that working folder, or how to deploy my executable into that folder as I won’t know what it is beforehand. I must be missing a step somewhere.

Any suggestions welcome :slight_smile:

4

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

5

Hello mikej, it’s been a while without an update from us. Are you still having problems or did you find a solution?

6

Hello.

Sadly not, I ran out of time investigating. I’m pretty sure this was the last thing I needed to get working, and would still be curious to know what the fix is, but unfortunately we’ve had to make the decision to go with the other CDC system I was investigating and comparing Airbyte to.

For what it’s worth, you’re probably already aware but the .net connector repo is only 50% complete, and there’s almost nothing there for destinations. If I’d got this working, I may have added my own PRs to improve it, but it could do with someone looking at. I’m sure there must be other .net developers who’d find it useful.

Thanks,

Mike