- Is this your first time deploying Airbyte?: Yes
- Deployment: Kubernetes (EKS)
- Airbyte Version: 0.45.20
- Source name/version: Salesforce 2.0.11
- Destination name/version: Redshift 0.4.5
- Step: Sync / Reset Data
- Description: When I sync data to the Redshift Destination, it will hang indefinitely and not ever be marked as completed. Here are two examples of the end of the logs where it hangs (it’s always one of this lines: “Finished syncing <source>” or “Executing finalization of tables.”
Example 1:
2023-05-31 21:18:11 destination > INFO i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$1):164 Preparing staging area in destination completed for schema jira stream board_issues
2023-05-31 21:18:11 destination > INFO i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$1):166 Executing finalization of tables.
2023-05-31 21:18:12 source > Syncing stream: board_issues
2023-05-31 21:18:12 source > Read 4 records from board_issues stream
2023-05-31 21:18:12 source > Finished syncing board_issues
2023-05-31 21:18:12 source > SourceJira runtimes:
Syncing stream board_issues 0:00:00.514304
Syncing stream boards 0:00:00.430819
2023-05-31 21:18:12 source > Finished syncing SourceJira
Example 2:
2023-05-31 21:02:11 source > SourceStripe runtimes:
Syncing stream accounts 0:00:00.133225
Syncing stream charges 0:00:00.204658
Syncing stream customers 0:00:00.215795
2023-05-31 21:02:11 source > Finished syncing SourceStripe
2023-05-31 21:02:18 destination > INFO i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$1):164 Preparing staging area in destination completed for schema stripe_test_data stream customers
2023-05-31 21:02:18 destination > INFO i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$1):166 Executing finalization of tables.
If I restart the worker pod, the sync will fail and then invariably succeed in the retry, then hang again on the next sync. This happens with all source connectors I have tried (Salesforce, Jira, Stripe) but does not happen with other destinations I have tried (mostly S3).
We are deploying Airbyte with the Helm chart and an RDS Postgres DB, everything else is pretty standard in the chart. We are currently connecting to redshift through a proxy while we set up VPC peering.