Hubspot Normalization Failure

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Linux
  • Deployment: Kubernetes
  • Airbyte Version: 0.39.26
  • Source name/version: Hubspot 0.1.72
  • Destination name/version: Redshift 0.3.44
  • Step: Sync
  • Description: Normalization Failing
2022-07-06 09:24:48 normalization > 09:24:41.295666 [info ] [MainThread]: 
2022-07-06 09:24:48 normalization > 09:24:41.296094 [info ] [MainThread]: Completed with 2 errors and 0 warnings:
2022-07-06 09:24:48 normalization > 09:24:41.296421 [info ] [MainThread]: 
2022-07-06 09:24:48 normalization > 09:24:41.296672 [error] [MainThread]: Database Error in model hubspot_export_engagements_emails_scd (models/generated/airbyte_incremental/scd/xxxxxxxxxxxxxxxxxxxxx/hubspot_export_engagements_emails_scd.sql)
2022-07-06 09:24:48 normalization > 09:24:41.296944 [error] [MainThread]:   Invalid input
2022-07-06 09:24:48 normalization > 09:24:41.297180 [error] [MainThread]:   DETAIL:  
2022-07-06 09:24:48 normalization > 09:24:41.297650 [error] [MainThread]:     -----------------------------------------------
2022-07-06 09:24:48 normalization > 09:24:41.298328 [error] [MainThread]:     error:  Invalid input
2022-07-06 09:24:48 normalization > 09:24:41.299000 [error] [MainThread]:     code:      8001
2022-07-06 09:24:48 normalization > 09:24:41.299517 [error] [MainThread]:     context:   SUPER value exceeds export size.
2022-07-06 09:24:48 normalization > 09:24:41.300039 [error] [MainThread]:     query:     7078589
2022-07-06 09:24:48 normalization > 09:24:41.300564 [error] [MainThread]:     location:  partiql_export.cpp:9
2022-07-06 09:24:48 normalization > 09:24:41.301131 [error] [MainThread]:     process:   query0_122_7078589 [pid=29308]
2022-07-06 09:24:48 normalization > 09:24:41.301647 [error] [MainThread]:     -----------------------------------------------
2022-07-06 09:24:48 normalization > 09:24:41.302225 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/xxxxxxxxxxxxxxxxxxxxx/hubspot_export_engagements_emails_scd.sql
2022-07-06 09:24:48 normalization > 09:24:41.302942 [info ] [MainThread]: 
2022-07-06 09:24:48 normalization > 09:24:41.303498 [error] [MainThread]: Database Error in model hubspot_export_engagements_scd (models/generated/airbyte_incremental/scd/xxxxxxxxxxxxxxxxxxxxx/hubspot_export_engagements_scd.sql)
2022-07-06 09:24:48 normalization > 09:24:41.304108 [error] [MainThread]:   Invalid input
2022-07-06 09:24:48 normalization > 09:24:41.304634 [error] [MainThread]:   DETAIL:  
2022-07-06 09:24:48 normalization > 09:24:41.305147 [error] [MainThread]:     -----------------------------------------------
2022-07-06 09:24:48 normalization > 09:24:41.305583 [error] [MainThread]:     error:  Invalid input
2022-07-06 09:24:48 normalization > 09:24:41.305989 [error] [MainThread]:     code:      8001
2022-07-06 09:24:48 normalization > 09:24:41.306397 [error] [MainThread]:     context:   SUPER value exceeds export size.
2022-07-06 09:24:48 normalization > 09:24:41.306916 [error] [MainThread]:     query:     7078635
2022-07-06 09:24:48 normalization > 09:24:41.307384 [error] [MainThread]:     location:  partiql_export.cpp:9
2022-07-06 09:24:48 normalization > 09:24:41.307915 [error] [MainThread]:     process:   query0_121_7078635 [pid=1734]
2022-07-06 09:24:48 normalization > 09:24:41.308457 [error] [MainThread]:     -----------------------------------------------
2022-07-06 09:24:48 normalization > 09:24:41.308957 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/xxxxxxxxxxxxxxxxxxxxx/hubspot_export_engagements_scd.sql
2

Hey, I think the error is because the Redshift SUPER type can’t store more than 1MB records. You probably have a big record that triggers this error.
Could you please try to upgrade Airbyte and your redshift connector to their latest version and let me know if the error persists?

3

Upgraded Redshift Connector to the latest version, issue persists.
Not sure how useful upgrading Airbyte would be if the issue comes from Redshift itself.
Any workarounds?

4

Not sure how useful upgrading Airbyte would be if the issue comes from Redshift itself.

The normalization process is part of Airbyte’s core platform, so if you want the latest normalization for redshift, you need to have the latest Airbyte version.
Please let me know if the error persists after upgrading to the latest version and I’ll open an issue on GitHub.

5

Tried with Airbyte v0.39.35-alpha, issue persists.

6

I created an issue on our repo for this error. Please subscribe to receive updates.

1 Like