Issue with Normalization and DBT Transformation Steps in Aurora RDS Replication

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing an issue where basic normalization and DBT transformation steps in Aurora RDS replication are taking 2 to 3 minutes each, despite achieving replication streaming completion within 30 seconds. The logs show a warning about unused configuration paths in the dbt_project.yml file and mention a concurrency of 8 threads taking 2 minutes for each step.


Question

Hello everyone,
I am facing an issue while normalization. My source and destinations are different endpoints in Aurora RDS.

  1. I can achieve my replication streaming completion on within 30 seconds
  2. but starting basic normalization and DBT transformation steps take 2 to 3 minutes each.
  3. I observed the logs, Concurrency: 8 threads (target='prod') taking 2 mins for each.
  4. My EC2 instance and RDS are in the same VPC.
2024-01-16 08:20:57 normalization >            [--event-buffer-size EVENT_BUFFER_SIZE]
2024-01-16 08:20:57 normalization >   --event-buffer-size EVENT_BUFFER_SIZE
2024-01-16 08:20:57 platform > 
2024-01-16 08:20:57 normalization > DBT >=1.0.0 detected; using 10K event buffer size
2024-01-16 08:20:57 platform > 
2024-01-16 08:20:59 normalization > Running with dbt=1.0.0
2024-01-16 08:20:59 normalization > Partial parse save file not found. Starting full parse.
2024-01-16 08:21:01 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
- models.airbyte_utils.generated.airbyte_tables
- models.airbyte_utils.generated.airbyte_views
2024-01-16 08:21:01 normalization > Found 105 models, 0 tests, 0 snapshots, 0 analyses, 599 macros, 0 operations, 0 seed files, 21 sources, 0 exposures, 0 metrics
2024-01-16 08:23:15 normalization > Concurrency: 8 threads (target='prod')
2024-01-16 08:23:16 normalization > 1 of 63 START incremental model _airbyte_public.```
Can someone please help with this? Why does this consume 2 to 3 mins in this step?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1705394108945039) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["normalization", "dbt-transformation", "aurora-rds", "replication", "concurrency", "dbt-project-yml"]
</sub>