User wants to download YAML of an existing source connection to modify errors in the standard connection.
Hi is there a way to download the YAML of an existing source connection? There are some errors in the starndard connection of Airbyte, which I would love to change
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Yes, you can modify the configuration of your source connection. However, it’s important to note that Airbyte’s backend currently does not support users editing the schema. The connection configuration contains streams config + streams schema, which can lead to large YAML configuration files for some connectors. The recommendation is to only keep stream configuration in the connection configuration and split stream schema into a separate file that users are prevented from editing (source). If you’re using the Octavia CLI, you can generate a connection template with the octavia generate connection command. After generating the template, you can edit the configuration as needed. Here’s an example of how to generate a connection: bash octavia generate connection --source sources/postgres/configuration.yaml --destination destinations/bigquery/configuration.yaml postgres-to-bigquery After running this command, you can edit the generated configuration file (connections/postgres_to_bigquery/configuration.yaml) as needed (source). Please be aware that when editing the stream configuration, Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior (source). If you’re facing specific errors, it would be helpful to know more details about them to provide a more accurate solution.
I dont know about Octavia, but if you mean you want the yaml source do an existing connector in our catalog, you can find it on GitHub, happy to help you find it. That’s assuming that it’s actually a low code connector — most current connectors have at least some Python in them.
But, if you know of a bug in a connector, I’m happy to guide you through filing an issue and try and fix it and make a PR.
Which connector are you looking at?
<@U069EMNRPA4> thanks for your response. It is in the Source_close_com
the custom_activities_stream has the wrong path to the close api. With the path custom_activity you will only get the custom activity types but not the instances which are actually the custom activities.
WRONG: https://developer.close.com/resources/custom-activities/custom-activity-types/
CORRECT: https://developer.close.com/resources/custom-activities/custom-activity-instances/
The path needs to be
CURRENT VERSION
custom_activities_stream:
$ref: “#/definitions/base_stream”
$parameters:
name: “custom_activities”
path: “custom_activity”
CORRECT VERSION
custom_activities_stream:
$ref: “#/definitions/base_stream”
$parameters:
name: “custom_activities”
path: “activity/custom”
based on that the custom_activities.json needs to be updated as well https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-close-com/source_close_com/schemas/custom_activities.json
<@U069EMNRPA4> Last point in the custom activities and also other streams we have custom fields like this. Is there any way to transform those fields. so that instead of having in the main streams like leads the field
custom_cf_k57pLkOkPa62Frjt3godWxUisurrmIpb8whX8ImJXer it shoould be custom_Meeting Typ
Field Information streams -> Main object streams
activity_custom_fields -> custom_activities
contact_custom_fields -> contacts
lead_custom_fields -> leads
opportunity_custom_fields -> opportunities
In the field information streams you need to search via id and then replace it with the name
Maybe just as additional fields?
I would also pay for this adjustment if needed. Just send me a DM.
Thanks a lot
For your first question here’s how you can do changes:
https://docs.airbyte.com/contributing-to-airbyte/change-cdk-connector
Also, for transformations, it is very possible to add them with<https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte| dbt cloud>, however I this option is just for Airbyte Cloud.
If you have your own setup, you can add your own transformations in your own setup via the YAML files. Here’s the YAML file composition, where you will find a transformations object with 3 possibilities: Addfields, RemoFields and CustomTransformation (for complex transformation which will require you to create a python script)