MySQL to Snowflake Fails Normalization (Doesn't build RAW table)

danieldiamond · November 1, 2022, 12:34pm

Updates: that seems to be unrelated. After a few successful syncs it’s now randomly failing every subsequent incremental sync

I wonder if this issue is related

github.com/airbytehq/airbyte

MySQL to Snowflake incremental loading fails

opened 05:12PM - 03 Oct 22 UTC

marcosmarxm

type/bug temporal connectors/source/mysql zendesk team/db-dw-sources

This Github issue is synchronized with Zendesk: **Ticket ID:** [#2471](https://…airbyte7538.zendesk.com/agent/tickets/2471) **Priority:** normal **Group:** Community Assistance Engineer **Assignee:** Nataly Merezhuk **Original ticket description:** <blockquote><div class="zd-comment" dir="auto"><ul style="margin: 0 0 0 10px; padding: 0 0 0 20px" dir="auto"> <li style="padding-bottom: 10px">Is this your first time deploying Airbyte?: No</li> <li style="padding-bottom: 10px">OS Version / Instance: EC2 Linux AMI 2</li> <li style="padding-bottom: 10px">Memory / Disk: M6a.xlarge instance - 4 vCPUs, 16 GiB memory, 60 Gb storage</li> <li style="padding-bottom: 10px">Deployment: Docker</li> <li style="padding-bottom: 10px">Airbyte Version: 0.40.9 (latest as of writing)</li> <li style="padding-bottom: 10px">Source name/version: MySQL 0.6.14 (latest as of writing) connected through SSH tunnel</li> <li style="padding-bottom: 10px">Destination name/version: Snowflake 0.4.38 (latest as of writing) internal staging</li> <li style="padding-bottom: 10px">Step: During incremental sync</li> <li style="padding-bottom: 10px">Description:</li> </ul> <p dir="auto">Hi.</p> <p dir="auto">I am trying to sync moodle data sitting in Amazon Aurora to Snowflake. I was able to do so successfully in a previous Airbyte version 0.39.19-alpha.</p> <p dir="auto">Syncing 7.21Gb of data (about 40 tables) is successful on the initial run.<br> All subsequent syncs are failing. See image below.</p> <p dir="auto"></p><div><a href="https://aws1.discourse-cdn.com/business7/uploads/airbyte/original/2X/a/ab87d0b2c21fd7de2036abb527efbaf07b725eb4.png" title="historical-success-incremental-fail" rel="noopener nofollow ugc noreferrer" style="color: #006699; font-weight: bold; text-decoration: none"><img src="https://aws1.discourse-cdn.com/business7/uploads/airbyte/optimized/2X/a/ab87d0b2c21fd7de2036abb527efbaf07b725eb4_2_690x435.png" alt="historical-success-incremental-fail" width="auto" height="auto" style="max-width: 100%"><div style="display: none"> <span>historical-success-incremental-fail</span><span>1493×943 67.8 KB</span> </div></a></div><p dir="auto"></p> <p dir="auto">Here is the log for the successful initial sync:<br> <a href="https://discuss.airbyte.io/uploads/short-url/2Tud15TexlxX7Xb9V4VGJggJVsn.txt" style="color: #006699; font-weight: bold; text-decoration: none" rel="noreferrer">success-logs.txt</a> (7.0 MB)<br> Errors/warnings in the log that may be of interest:</p> <ul style="margin: 0 0 0 10px; padding: 0 0 0 20px" dir="auto"> <li style="padding-bottom: 10px">JSON schema validation failed</li> <li style="padding-bottom: 10px">Signalling close because record’s binlog file : mysql-bin-changelog.002503 , position : 75169383 is after target file : mysql-bin-changelog.002503 , target position : 75125447</li> <li style="padding-bottom: 10px">The main thread is exiting while children non-daemon threads from a connector are still active. Ideally, this should not happen</li> </ul> <p dir="auto">Here is the log for the first failed sync:<br> <a href="https://discuss.airbyte.io/uploads/short-url/7JmUomae5tbeMucpn97dAMO3Zq0.txt" style="color: #006699; font-weight: bold; text-decoration: none" rel="noreferrer">failure-logs.txt</a> (5.3 MB)<br> Errors/warnings in the log that may be of interest:</p> <ul style="margin: 0 0 0 10px; padding: 0 0 0 20px" dir="auto"> <li style="padding-bottom: 10px">2022-09-28 19:47:53 - Additional Failure Information: ScheduleActivityTaskCommandAttributes.Input exceeds size limit.</li> <li style="padding-bottom: 10px">Repeat of the errors/warnings found in successful.logs</li> </ul> <p dir="auto">Even with the warnings, the initial sync is successful. However, subsequent syncs are failing. I believe there is enough CPU and RAM for this sync. The error:</p> <pre style="max-width: 694px; word-wrap: break-word"><code style="background-color: #f9f9f9; display: block; overflow: auto; padding: 2px 5px">ScheduleActivityTaskCommandAttributes.Input exceeds size limit. </code></pre> <p dir="auto">seems to be related to temporal, but it makes no sense that the initial sync ran without an issue.</p> <p dir="auto">Please could someone assist me on this issue. Thank you.</p> [<a href="https://discuss.airbyte.io/t/mysql-to-snowflake-incremental-loading-fails/2740/1" rel="noreferrer">Discourse post</a>]</div></blockquote>

Topic		Replies	Views
Normalization process fails for the MySQL source and Snowflake destination Connector Questions & Issues normalization	2	206	March 21, 2023
Normalization failed for mysql source/dest Connector Questions & Issues normalization	3	449	July 14, 2022
Multiple mysql sources streamed to the same snowflake db are failing after the first successful sync Connector Questions & Issues connectors	5	367	July 14, 2022
MySQL 1.0.1 (GA) Source Normalization Failure Connector Questions & Issues source-mysql , normalization	8	667	October 12, 2022
MySQL to Snowflake CDC Sync Fails: Race condition, raw table not found but still being written to snowflake Connector Questions & Issues source-mysql , destination-snowflake , data-loading , cdc	9	621	July 14, 2022

MySQL to Snowflake Fails Normalization (Doesn't build RAW table)

Related topics