Summary
In Airbyte Cloud, the user is trying to fork a Marketplace source connector (DataDog) to make changes due to a specific API endpoint issue. They are encountering an error when trying to import a YAML manifest for modification.
Question
Hey all. In Airbyte Cloud, is there a way to effectively fork a Marketplace source connector to make some changes to it? Our DataDog source connector has started failing because the role of the Application Key we’re using in DataDog is not allowed to query a certain API endpoint for a stream that I don’t actually need. There is just no way in the DataDog New Source UI for me to disable that stream - so I’m stuck with a connector that fails immediately and is not syncing any records for the stream that I want.
I figured it would be an easy fix for me - just fork the connector, remove any reference to the offending stream, then set up a new Source with that connector. I tried downloading the connector’s https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-datadog/source_datadog/manifest.yaml|manifest.yaml and doing just that using the “Import a YAML manifest” option in the Builder, but I’m getting the error Unable to find spec.yaml or spec.json in the package. .
What am I missing here? Any help would be appreciated
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["airbyte-cloud", "forking-connector", "marketplace-source-connector", "datadog", "yaml-manifest", "error"]
Hey Eric 
- just want to first confirm that you have tried to disable the stream at the Connection level. Does it still fail if you go to the Schema tab and disable the offending stream?
Oh nice, thanks for the response. Yes, it has been disabled the whole time (pre- and post-issue).
The offending stream is audit_logs , and I think that for some reason the connector still wants to query their AuditLogs API endpoint whether or not you have that stream enabled. I think this because <https://docs.airbyte.com/integrations/sources/datadog|the docs> say things like “Type your query to filter records when collecting data from Logs and AuditLogs stream” - as if when building the logs stream, it hits both Logs and AuditLogs .
Got it - yeah it likely still hits that endpoint as a part of the spin up process. Are you running Open Source Community Edition? And did you upgrade the connector version or did this just start happening after a policy was put in place on your end?
I’m on Cloud. So, it DOES seem like the connector was upgraded to 0.4.7 in the last 24 hours. I’ve actually submitted a ticket to y’all asking how to isolate what version a connector was when a job was run (it doesn’t seem to be showing up in the logs). If I could see that and the first failing job was on the new 0.4.7, then obviously that would be the smoking gun - though I don’t know how I would revert to an old version of a connector in Cloud.
The thing is, I’ve looked at the code, and it seems like the only change that occurred in the version bump was https://github.com/airbytehq/airbyte/commit/ec3b119102a70867e79e8c8646527b8467d46389|this which doesn’t seem relevant at all.
And there have been no changes in DataDog permissioning on our end. Really clueless on this, and the only solution I can come up with is to run my own fork of the connector
Hey Eric - just shooting you a DM for more details.
As for this specific question on forking Airbyte Connectors, this is definitely doable. The issue today is not every connector is compatible with the builder just yet. We are working on supporting more components only available in YAML and Python CDKs but it will take a little more time.
In the meantime, for DataDog and other connectors that aren’t builder compatible yet, you will have to pull them into a dev environment and repackage as a Docker Image and load it that way.