The user is asking about using airbyte-ci for forking an existing source (slack) and making modifications.
I want to fork an existing source (slack) and make some modifications. Is airbyte-ci the way to go about this?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Based on the information provided in the extracts, it seems that airbyte-ci is used for continuous integration purposes and not specifically for forking and modifying a source. However, there is a mention of a potential issue where changes to airbyte-ci aren’t used on branches, which could impact your work if you’re planning to make changes on a separate branch. If you’re looking to modify an existing source like Slack, you might want to follow the guidelines provided in the Building a Python Source tutorial. This guide provides a checklist for submitting a source to Airbyte, which includes steps like submitting a PR and coordinating with an Airbyte engineer to add test credentials. If you have any questions or need help during the process, you’re encouraged to submit a PR with your progress and any questions you have, or ask for help on the #help-connector-development channel on Slack. Please note that this advice is based on the information provided in the extracts and may not cover all aspects of your task. For more detailed guidance, you might want to refer to the full Airbyte documentation or reach out to the Airbyte community.
<@U04SE163WC8> Is it possible to build a local version of a connector and use it in our own production instance?
Yes, it is possible to build a local version of a connector and use it in your own production instance. Here are the steps you can follow: 1. Make the necessary changes to your connector. 2. Build the connector using the following command: ./gradlew :airbyte-integrations:connectors:source-custom-name:build This command is run from the root of the Airbyte repository[^1^]. 3. Once the build process is complete, you can tag the image[^1^]. 4. Add it as a custom connector in the Airbyte UI. You can do this by going to Settings → Source → Custom Connector[^1^]. 5. If you’re running Airbyte locally, it will have access to any Docker images available to your local docker daemon[^4^]. 6. You can verify the image was built by running: docker images | head[^4^]. 7. In your browser, visit http://localhost:8000/, log in with the default user airbyte and default password password. Go to Settings (gear icon in lower left corner), go to Sources or Destinations (depending on which connector you are testing), update the version number to use your docker image tag (default is dev), and click Change to save the changes[^2^]. 8. Now when you run a sync with that connector, it will use your local docker image[^2^]. Please note that for real production connectors to be available on Airbyte Cloud, you’d need to publish them on DockerHub[^4^]. [^1^]: Forum Post: How do I locally switch to a modified version of a connector? > Answers [^2^]: Documentation > Contributing-to-airbyte > Resources > Developing Locally > Add a connector under development to Airbyte [^3^]: Documentation > Connector-development > Tutorials > Building a Source Connector: The Hard Way > Our connector: a stock ticker API > Use the connector in the Airbyte UI > 1. Publish the Docker image [^4^]: Github Issue (closed): Optionally build python connector with the local CDK > Original post
<@U06D9DWTXUH>, happy to help! And also, if you fix bugs or improve the connector performance, or otherwise do things useful for other folks, I would love to help you get your improvements back into the shared Slack connector.
Slack is currently a Python CDK connector, not low-code, so that will be a bit involved. You’re right — I’d recommend that you work on it locally, run airbyte-ci locally to verify your changes.
If you wanted to build a custom version of Slack connector for to use in your OSS workspaces, you can just change the name of the connector docker image and version in it’s metadata.yaml and build the image with airbyte-ci connector build --name source-slack . That should do it, more or less. I expect that we have some things in airbyte-ci that are very airbyte-specific, but I’m here to help if you hit any hurdles.
Thanks much <@U069EMNRPA4>! I will definitely hit you up if I run into any roadblocks.
I’m very curious to see what you want to work on in Slack connector. Don’t get me wrong, there are quite a few things we would like to improve 
If you dump a list of improvements that you’re considering on me, perhaps I can send in some cavalry to help with reviews and other support.
Ooooh nice.
Well for the most part it’s working great. My immediate need is to grab messages from private channels. The slack API supports it, and we have a use case where it’s needed.
I’ll keep track of other nice to haves as I see them.
Question for you <@U069EMNRPA4> - this is from the Slack source connector.
Am I correct that if the sync is run every 24 hours, there’s no need for the thread look back window to be more than a day?
A weird question for you! If you’re up for a challenge, would you like to make Slack full refresh (first sync) to be blazing fast? Caveat: that’ll be quite a bit of code, I think we could use the bulk download / export feature for that
Ooooh that sounds enticing. I’m open to it, but i’d need a little help getting my local dev up and running.
For example I don’t see a Dockerfile in the source_slack connector. I used airbyte_ci to build it, but I can’t push it to my docker hub. Can you help me register my dockerhub service account with airbyte_ci?
Do you see the image built locally in your local docker image ls?
> I’m open to it, but i’d need a little help getting my local dev up and running.
EZ, I just did that a couple of months back. And I’m learning connectors myself, so. 
Also disclaimer, we as the extensibility team are looking at whether we can add a mechanism to grab “bulk download” exports for full syncs as part of the CDK. We have it implemented in a couple of connectors (good, I’ll give you examples!), and I’m looking at whether we can generalize that.
Another caveat is that generally we’re pushing all connectors to be in the low-code CDK, aka yaml declarations. Each connector can also be in a hybrid-stage when simple streams are already in YAML, but for more complex streams we still have Python CDK steam code.
> Can you help me register my dockerhub service account with airbyte_ci?
You can provide a DOCKER_HUB_USERNAME and a DOCKER_HUB_PASSWORD via environment variables. I’m not sure however that we support pushing to other registries than our own 
In general, our publish command in reality does <airbyte/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/pipeline.py at master · airbytehq/airbyte · GitHub ton more> than push the image to dockerhub, I wouldn’t recommend it in your case. You probably want to just do a docker push on the image that was built if you’re looking for fast iteration. If that makes you miss something you need for your workflow, let me know though
Thanks <@U047ANT3J84>! I’ll give it a try and let you know how it goes.