Is Sync mode Incremental - Deduped History supported for mysql destination connector

Hi, Earlier I used to see Sync mode Incremental - Deduped History available in previous version of mysql destination connector. Now its not visible in latest version of airbyte. I have the following versions of mysql and normalized mysql images.


As per the documentation, its supported.

Can some one please advise?

What source connector are you using?

I have tried all sources including MySQL, still Incremental-De duped History is not available under sync mode. Can you check from your end by creating a pipeline from MySQL to MySQL and see whether it is displayed or not ?

@rnncredit what version were you before of the connector and Airbyte? The Append+Dedup never was enabled in MySQL before, probably it’s an error in our docs.

Well I do not have exact information about version but it used to show up. The same happens with Oracle destination connector.

I have enabled normalization and it does not show up incremental-deduped history. But the docs say below

can you check from your end for the oracle db as well?
In general, when does these modes show up ?
I built a custom connector and mentioned the following in the discover method:
supported_sync_modes=[
SyncMode.full_refresh,SyncMode.incremental],
supported_destination_sync_modes=[DestinationSyncMode.append,
DestinationSyncMode.overwrite, DestinationSyncMode.append_dedup]

@marcosmarxm You were right seems de-dupe was not enabled in any earlier versions of MYSQL.
But even basic normalization with full refresh|overwrite mode is failing in latest code base.

Attached log message, can you please check from your end as well whether basic normalization working or not?

2022-04-19 02:19:36 normalization > Running: transform-config --config destination_config.json --integration-type mysql --out /data/113/0/normalize
2022-04-19 02:19:37 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.mysql: 'mysql'>, out='/data/113/0/normalize')
2022-04-19 02:19:37 normalization > transform_mysql
2022-04-19 02:19:37 normalization > Running: transform-catalog --integration-type mysql --profile-config-dir /data/113/0/normalize --catalog destination_catalog.json --out /data/113/0/normalize/models/generated/ --json-column _airbyte_data
2022-04-19 02:19:37 normalization > Processing destination_catalog.json...
2022-04-19 02:19:37 normalization >   Generating airbyte_ctes/discometa/gl_periods_ab1.sql from gl_periods
2022-04-19 02:19:37 normalization >   Generating airbyte_ctes/discometa/gl_periods_ab2.sql from gl_periods
2022-04-19 02:19:37 normalization >   Generating airbyte_ctes/discometa/gl_periods_ab3.sql from gl_periods
2022-04-19 02:19:37 normalization >   Generating airbyte_tables/discometa/gl_periods.sql from gl_periods
2022-04-19 02:19:37 normalization > detected no config file for ssh, assuming ssh is off.

io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:61) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:58) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	... 3 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:160) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:46) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
2022-04-19 02:19:39 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-04-19 02:19:39 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating...
2022-04-19 02:19:39 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=9183f927-dbdb-3d4f-ba95-8b61c3b0bb3c, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:233) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:72) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]

I have the following images:

Please upload the complete file log. Sometimes normalization failed because a previous error happened.

Attached logs. Please advise.
logs-114.txt (33.7 KB)

@marcosmarxm I think the same issue was reported few days back.

1 Like

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.