Source Salesforce: "QUERY_TIMEOUT"

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu, MacOS
  • Memory / Disk: 8Gb / 500 Gb
  • Deployment: GKE (kubernetes)
  • Airbyte Version: 0.39.34-alpha
  • Source name/version: Salesforce / 1.0.20
  • Destination name/version: BigQuery / 1.2.0
  • Step: during sync
  • Description:
    I am attempting to transfer data in Salesforce.
    When I try to transfer a large object, I get the following error and want the correct transfer to take place.
2022-09-16 05:33:10 e[44msourcee[0m > Syncing stream: Contact 
2022-09-16 05:35:06 e[44msourcee[0m > [{"message":"Your query request was running for too long.","errorCode":"QUERY_TIMEOUT"}]
2022-09-16 05:35:06 e[44msourcee[0m > Cannot receive data for stream 'Contact', error message: 'Your query request was running for too long.'
2022-09-16 05:35:06 e[44msourcee[0m > Read 0 records from Contact stream

Do you have any suggestions? Thanks.

2

This issue has been discussed for some time.
The solution is to simplify SOQL. (e.g., filter by WHERE clause, limit number of items by limit, etc.)

However, this solution is not suitable for Airbyte’s architecture.Because Airbyte is ELT.
If the errorCode is QUERY_TIMEOUT, the solution would be to shift the date condition specified in the WHERE clause of the query in request_params and send the request.

Also, I discovered this problem too late.
Because the worker terminates normally even if the above error occurs. I believe there is a bug in the if statement below and the exception is not just raised.
The first condition of the if statement causes all error codes to return instead of raise.

3

Hi @Nakachi-S, thanks for bringing this to our attention! I have created a GitHub issue connected to your forum post and triaged it, you can find it and comment more here:
https://github.com/airbytehq/airbyte/issues/17503

1 Like