Normalization Performance Issue in Aurora RDS

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is experiencing delays in normalization and DBT transformation steps while using Airbyte with Aurora RDS, noting that the process takes 2 to 3 minutes despite a quick replication streaming completion.


Question

Hello everyone,
I am facing an issue while normalization. My source and destinations are different endpoints in Aurora RDS.

  1. I can achieve my replication streaming completion on within 30 seconds
  2. but starting basic normalization and DBT transformation steps take 2 to 3 minutes each.
  3. I observed the logs, Concurrency: 8 threads (target='prod') taking 2 mins for each.
  4. My EC2 instance and RDS are in the same VPC.
2024-01-16 08:20:57 normalization >            [--event-buffer-size EVENT_BUFFER_SIZE]
2024-01-16 08:20:57 normalization >   --event-buffer-size EVENT_BUFFER_SIZE
2024-01-16 08:20:57 platform > 
2024-01-16 08:20:57 normalization > DBT >=1.0.0 detected; using 10K event buffer size
2024-01-16 08:20:57 platform > 
2024-01-16 08:20:59 normalization > Running with dbt=1.0.0
2024-01-16 08:20:59 normalization > Partial parse save file not found. Starting full parse.
2024-01-16 08:21:01 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
- models.airbyte_utils.generated.airbyte_tables
- models.airbyte_utils.generated.airbyte_views
2024-01-16 08:21:01 normalization > Found 105 models, 0 tests, 0 snapshots, 0 analyses, 599 macros, 0 operations, 0 seed files, 21 sources, 0 exposures, 0 metrics
2024-01-16 08:23:15 normalization > Concurrency: 8 threads (target='prod')
2024-01-16 08:23:16 normalization > 1 of 63 START incremental model _airbyte_public.```
Can someone please help with this? Why does this consume 2 to 3 mins in this step?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1705394108945039) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['normalization', 'aurora-rds', 'dbt-transformation', 'ec2-instance', 'vpc', 'event-buffer-size']
</sub>