Low Code Connector Development and Data Loading

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Query regarding development with low code connectors and data loading into destination schema, involving the use of dbt models to spread raw data into separate columns.


Question

Hello, team! Very quick question, does development with low code connector mean that data will be loaded only on airbyte-internal table not to the destination schema , and that I should add dbt model somehow so that it would spread the raw data into separate columns ? Thanks !!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["low-code-connector", "data-loading", "destination-schema", "dbt-model", "raw-data"]
2

<@U066U84SC0P> thank you for your reply. My appsflyer response looks something like this (below), so it looks like it doesn’t need any field path to specify. Can you explain me how the incremental sync happen if the destination table is empty so the cursor value (like event_time) is also empty, how airbyte handles the incremental syncs in this case? Is it is Incremental append will it always go back to beginning date and write all data every time ?
{
“Ad”: “”,
“IP”: “95.82.64.232”,
“DMA”: “None”,
“City”: “Sadovoe”,
“IDFA”: “”,
“IDFV”: “”,
“IMEI”: “”,
“WIFI”: “true”,
“Ad ID”: “”,
“Adset”: “”,
“State”: “11”,
“App ID”: “kz.berekebank.mobile.app”,
“Region”: “AS”,
“Ad Type”: “”,
“Carrier”: “Beeline KZ”,
“Channel”: “”,
“Partner”: “”,
“Site ID”: “”,
“Adset ID”: “”,
“App Name”: “B-Bank”,
“Campaign”: “”,
“Keywords”: “”,
“Language”: “русский”,
“Operator”: “Beeline KZ”,
“Platform”: “android”,
“Bundle ID”: “kz.berekebank.mobile.app”,
“Android ID”: “”,
“Cost Model”: “”,
“Cost Value”: “”,
“Event Name”: “initial_transfer_internal”,
“Event Time”: “2024-03-11 11:45:57”,
“OS Version”: “9”,
“User Agent”: “Dalvik/2.1.0 (Linux; U; Android 9; SM-J415FN Build/PPR1.180610.011)”,
“App Version”: “2.28.0”,
“Campaign ID”: “”,
“Device Type”: “”,
“Event Value”: “{}”,
“Postal Code”: “c58”,
“SDK Version”: “v6.12.2”,
“Sub Param 1”: “”,
“Sub Param 2”: “”,
“Sub Param 3”: “”,
“Sub Param 4”: “”,
“Sub Param 5”: “”,
“Sub Site ID”: “”,
“AppsFlyer ID”: “1699284846307-3501485777574531744”,
“Country Code”: “KZ”,
“Event Source”: “SDK”,
“Install Time”: “2023-11-06 15:25:34”,
“Media Source”: “”,
“Original URL”: “”,
“Cost Currency”: “”,
“Event Revenue”: “”,
“HTTP Referrer”: “”,
“Advertising ID”: “afe8d841-474c-4eb8-8c9d-1afe39e84455”,
“Is Retargeting”: “false”,
“Customer User ID”: “Q002548635”,
“Event Revenue USD”: “”,
“Reengagement Window”: “”,
“Attribution Lookback”: “”,
“Is Receipt Validated”: “”,
“Attributed Touch Time”: “”,
“Contributor 1 Partner”: “”,
“Contributor 2 Partner”: “”,
“Contributor 3 Partner”: “”,
“Contributor 1 Campaign”: “”,
“Contributor 2 Campaign”: “”,
“Contributor 3 Campaign”: “”,
“Event Revenue Currency”: “USD”,
“Is Primary Attribution”: “true”,
“Contributor 1 Touch Time”: “”,
“Contributor 1 Touch Type”: “”,
“Contributor 2 Touch Time”: “”,
“Contributor 2 Touch Type”: “”,
“Contributor 3 Touch Time”: “”,
“Contributor 3 Touch Type”: “”,
“Attributed Touch Type”: “”,
“Contributor 1 Media Source”: “”,
“Contributor 2 Media Source”: “”,
“Contributor 3 Media Source”: “”,
“Retargeting Conversion Type”: “”
}

3

Low code connectors have basically the same behavior as native ones, so the destination behavior will be controlled by the destination itself, not the source. If you mean that raw responses returned by the API are not being normalized, that can controlled by defining the primary key and field path fields in the builder, but there is a limit to how much you can unnest, for instance:

4

While you may not need to specify the field path, the primary key is still needed for incrementally. In your case it looks like it will be IP and Event Time maybe. With this set, if you choose incremental dedup syncs, Airbyte will only write to the destination the newest instance of data according to the primary key. If you correctly set up your datetime cursor, stream partitions, and look back windows, Airbyte will avoid calling the API for old data too.