Summary
The user is encountering an error while normalizing MySQL data in Airbyte. The normalization process failed with a WorkerException. The error seems to be related to the normalization process in the Python code.
Question
hi all,
I am getting an error while normalizing MySQL.
2023-12-20 11:52:24 INFO i.a.w.g.DefaultNormalizationWorker(run):69 - Normalization failed
2023-12-20 11:52:24 normalization > Traceback (most recent call last):
2023-12-20 11:52:24 normalization > File “/usr/local/bin/transform-config”, line 5, in <module>
2023-12-20 11:52:24 normalization > from normalization.transform_config.transform import main
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/normalization/init.py”, line 2, in <module>
2023-12-20 11:52:24 INFO i.a.w.g.DefaultNormalizationWorker(run):73 - Normalization catch block exception—> {}.
io.airbyte.workers.exception.WorkerException: Normalization process wasn’t successful
at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:204) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:72) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:27) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
2023-12-20 11:52:24 normalization > from normalization.transform_catalog.transform import TransformCatalog
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/init.py”, line 1, in <module>
2023-12-20 11:52:24 normalization > from normalization.transform_catalog.transform import TransformCatalog
2023-12-20 11:52:24 INFO i.a.w.g.DefaultNormalizationWorker(run):84 - Normalization executed in 0 seconds.
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py”, line 12, in <module>
2023-12-20 11:52:24 normalization > from normalization.transform_catalog.catalog_processor import CatalogProcessor
2023-12-20 11:52:24 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):162 - Completing future exceptionally…
io.airbyte.workers.exception.WorkerException: Normalization Failed.
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:94) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:27) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/catalog_processor.py”, line 12, in <module>
2023-12-20 11:52:24 normalization > from airbyte_cdk.models import DestinationSyncMode, SyncMode
2023-12-20 11:52:24 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling…
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/init.py”, line 5, in <module>
2023-12-20 11:52:24 normalization > from .connector import AirbyteSpec, Connector
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py”, line 14, in <module>
2023-12-20 11:52:24 normalization > from airbyte_cdk.models import AirbyteConnectionStatus, ConnectorSpecification
2023-12-20 11:52:24 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating…
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/models/init.py”, line 8, in <module>
2023-12-20 11:52:24 normalization > from .airbyte_protocol import (
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/models/airbyte_protocol.py”, line 5, in <module>
2023-12-20 11:52:24 normalization > from airbyte_protocol.models import *
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_protocol/init.py”, line 25, in <module>
2023-12-20 11:52:24 normalization > from .models import (
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_protocol/models/init.py”, line 3, in <module>
2023-12-20 11:52:24 normalization > from .airbyte_protocol import *
2023-12-20 11:52:24 normalization > File “/usr/local/lib/python3.9/site-packages/airbyte_protocol/models/airbyte_protocol.py”, line 9, in <module>
2023-12-20 11:52:24 normalization > from pydantic import AnyUrl, BaseModel, Extra, Field
2023-12-20 11:52:24 normalization > File “pydantic/init.py”, line 2, in init pydantic.init
2023-12-20 11:52:24 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=c7b6ceaf-3faa-3d6c-afd3-bc54bb3cd52e, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:310) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:80) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:136) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:108) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:305) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
… 12 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:130) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:108) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:305) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
… 12 more
please help
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.