Adding column using CDK

Connector Development
1

By using CDK can I add a column with a default or constant value to every table?

2

Hi @SP271,
In the context of developing a new connector, you can definitely hardcode a constant value for a specific field on each incoming record that you read.
Could you please share what is your use case for this?

3

Hi @alafanechere
I want to fetch data from multiple sources and push it to same destination for analytics purposes. The problem with this is the data from different sources might get mixed up as all sources are having same table name with no column for differentiating source. So something like sourceID as a column will help in this case. That’s why using airbyte API I want to fetch multiple sources and then insert a new column for each source with a sourceID and then load it into destination.

4

You can overwrite the parse_response function to add that field:


class MyConnectorStream(HttpStream):
    def __init__(self, source_id):
        self.source_id = source_id

    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
        records = response.json()
        for record in records:
            record["sourceId"] = self.source_id
            yield record
5

Is there any equivalent of this if I use a java jdbc source. Or maybe as I am using a postgres source something like extending postgres source in the new java jdbc source that I created using cdc and overriding a method to add a column?

6

You need to check the Postgres Source code, it’s possible yes.

7

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.