Summary
User is experiencing issues with the Salesforce connector in Airbyte, where only a limited number of rows are being transferred to Snowflake. They have attempted to reset the stream without success and are seeking assistance or recommendations to transfer all rows.
Question
I’m using sales force connector but only able to transfer few rows from Salesforce to snowflake using airbyte. Does anyone faced this issue before ? Any help or recommendations on how to obtain all rows ? I even tried reset the stream but did not help much.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['salesforce-connector', 'snowflake', 'row-transfer', 'airbyte', 'stream-reset']
We’re not using Snowflake (almost all our destinations are BigQuery), but I can confirm that we have many Salesforce sources working fine, including objects syncing hundreds of millions of records when backfilling and several million per day in incremental syncs.
So I would look into issues with the API itself (e.g. rate limiting or quota contention), or the destination (but it’s worth noting that source and destination work independently of each other in Airbyte . . . so I wouldn’t expect a significant issue unless again there’s some weird config or quota or resource limit for the authorized Snowflake user.
So here the weird part of the issue is we have total of 69 rows in sales force with no restrictions applied in the salesforce end but only 25 rows are making into our target snowflake db all the time. Not sure , I upgraded the connector of SF and gave it a try , still does not work.
Any chance that the primary key is set to something that is non-unique? (e.g. something other than the object’s Id field)
Do you see unique rows landed in the staging tables in airbyte_internal? (these will be in JSON form, but are stored there before the merge happens into the final table, so may help you look into it)
Justin , I have all unique rows 25 count in the airbyte_internal table.
all are unique rows which are landed. Checking if there is non-unique primary key
Now ,that I’m doing Full refresh | Overwrite option/style of loading ID should not matter
We’re not using Snowflake (almost all our destinations are BigQuery), but I can confirm that we have many Salesforce sources working fine, including objects syncing hundreds of millions of records when backfilling and several million per day in incremental syncs.
So I would look into issues with the API itself (e.g. rate limiting or quota contention), or the destination (but it’s worth noting that source and destination work independently of each other in Airbyte . . . so I wouldn’t expect a significant issue unless again there’s some weird config or quota or resource limit for the authorized Snowflake user.
So here the weird part of the issue is we have total of 69 rows in sales force with no restrictions applied in the salesforce end but only 25 rows are making into our target snowflake db all the time. Not sure , I upgraded the connector of SF and gave it a try , still does not work.
Any chance that the primary key is set to something that is non-unique? (e.g. something other than the object’s Id field)
Do you see unique rows landed in the staging tables in airbyte_internal? (these will be in JSON form, but are stored there before the merge happens into the final table, so may help you look into it)
Justin , I have all unique rows 25 count in the airbyte_internal table.
all are unique rows which are landed. Checking if there is non-unique primary key
Now ,that I’m doing Full refresh | Overwrite option/style of loading ID should not matter
facing a similar issue with a seemingly simple sync – SFDC products table with 26 rows, only getting 16 rows in both the airbyte_internal and final Salesforce schema. I tried doing a full overwrite to no avail. Primary key is set to the table’s ID field
Let me know <@U07EBTFD1MJ> if you can find some solution for it.