Summary
User wants to make a small change to the MSSQL source connector to allow syncing the CDC schema and build a custom connector image. They are struggling with the Gradle approach and the complexity of the Airbyte repository structure.
Question
Naive question here, I want to make a change to one line in the mssql source connector, and then build my own docker image to use it as if it’s a custom connector. But that connector appears to be using the gradle approach, and when I clone the entire airbyte repo and let vs code spin up the devcontainer, frankly it’s a little over my head in terms of what all is happening, and the repo is huge.
Is there a simple way to just grab the mssql source code, make my small change (I want to allow syncing the cdc schema), and build an image I can use on my server? Or do I have to learn gradle and vs code conteainer development etc to even touch these connectors that don’t have a docker-compose.yaml in them?
Edit - I tried cloning the repo down, removing the connectors I didn’t care about, starting my own fresh branch pointed at a new repo so I don’t mess with anything the community is doing, making the change I wanted, then running ./gradlew :airbyte-integrations:connectors:source-mssql:buildConnectorImage, which seemed to run through although slowly. I have a new file in the container’s build/distributions folder, and it’s a tar that contains a java application, and that I’m not sure what to do with if connectors are typically docker images. What I don’t see is a new image in either my main PC’s docker desktop, or from running docker images in the vs code devcontainer.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["mssql-source-connector", "custom-connector", "gradle", "docker-image", "vs-code", "devcontainer"]
Thanks <@U03MTK4SWH3>. I see. I did make some progress on using the full repo and the java/gradle approach last night, but it was a little painful. Using airbyte-ci looks to be a much simpler approach. Thank you.
Yeah, I think I need to better understand how vs code interacts with java projects, since whatever build or sync process was happening, it wasn’t one I kicked off, just something that started running when the workspace opened for some reason based on the project files and setup.
Got it. Let us know how it goes. Cheers
Hey <@U03MTK4SWH3>. After some false starts where I struggled to get airbyte-ci update to run, and it never would, it said the airbyte-ci file was busy, I got this to work! Ended up just running a pull to grab all the new repo files. Initially it was giving me the message about airbyte-ci not matching versions, and auto-update failed, as did the airbyte-ci update.
Might have been some oddity of being on vs code in wsl2, not sure. ps didn’t show anything using the airbyte-ci file, so not sure why it kept saying it was locked/busy.
I think I’m all set now though is the short of it. Thanks again.
Hi Andrew,
In addition to the suggested above, you can also install our airbyte-ci command line by following this https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1|guide.
It provides a self contained environment that can build any connector.
Once installed you should be able to generate a custom image by running
airbyte-ci connectors --name=source-mssql build
Either with airbyte-ci or with gradle, you shouldn’t need to build the entire repo .
You can always build just the connector(s) you need
Give ./gradlew :airbyte-integrations:connectors:source-mssql:build a shot, and you should end up with a dev tagged docker image to try out
OK, I’ll give that a shot. I presume it’s intended that you’re going to run the gradlew commands from the devcontainer that’s defind in the repo? VS Code finds the java and python devcontainer json files so I let it go ahead and spin up the java version, and ran the command from there. I may have run buildImage, not build though. Which perhaps was the issue.