File not found error: Normalization profile

Any idea what the cause of this error could be? I cannot even reset the connection…

Airbyte Version: 0.40.0-alpha
Destination: Postgres (0.3.21)

2022-08-11 14:49:16 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:91) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:26) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
2022-08-11 14:49:16 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-08-11 14:49:16 e[32mINFOe[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating...
2022-08-11 14:49:16 e[42mnormalizatione[0m > Traceback (most recent call last):
2022-08-11 14:49:16 e[33mWARNe[m i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=7172a6ef-183a-370a-bcea-b07408f61b3d, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:310) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:75) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:103) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:305) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	... 12 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:129) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:103) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:305) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	... 12 more
Caused by: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:91) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:26) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.40.0-alpha.jar:?]
	... 1 more
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/bin/transform-config", line 8, in <module>
2022-08-11 14:49:16 e[42mnormalizatione[0m >     sys.exit(main())
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 364, in main
2022-08-11 14:49:16 e[42mnormalizatione[0m >     TransformConfig().run(args)
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 23, in run
2022-08-11 14:49:16 e[42mnormalizatione[0m >     transformed_config = self.transform(integration_type, original_config)
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 52, in transform
2022-08-11 14:49:16 e[42mnormalizatione[0m >     transformed_integration_config = {
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 181, in transform_postgres
2022-08-11 14:49:16 e[42mnormalizatione[0m >     dbt_config["sslmode"] = ssl_mode.get("mode")
2022-08-11 14:49:16 e[42mnormalizatione[0m > AttributeError: 'str' object has no attribute 'get'
2022-08-11 14:49:16 e[42mnormalizatione[0m > Traceback (most recent call last):
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/bin/transform-catalog", line 8, in <module>
2022-08-11 14:49:16 e[42mnormalizatione[0m >     sys.exit(main())
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 104, in main
2022-08-11 14:49:16 e[42mnormalizatione[0m >     TransformCatalog().run(args)
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 35, in run
2022-08-11 14:49:16 e[42mnormalizatione[0m >     self.parse(args)
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 46, in parse
2022-08-11 14:49:16 e[42mnormalizatione[0m >     profiles_yml = read_profiles_yml(parsed_args.profile_config_dir)
2022-08-11 14:49:16 e[42mnormalizatione[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 75, in read_profiles_yml
2022-08-11 14:49:16 e[42mnormalizatione[0m >     with open(os.path.join(profile_dir, "profiles.yml"), "r") as file:
2022-08-11 14:49:16 e[42mnormalizatione[0m > FileNotFoundError: [Errno 2] No such file or directory: '/data/407/2/normalize/profiles.yml'

Hey could you help with more details like

  1. Deployment: Docker/k8s
  2. Which sync it is source and destination
  3. Airbyte version

Also share the complete logs

1 Like

I had a similar issue!

  1. Deployment: Docker desktop
  2. Version 11 Postgres source to a version 13.7 Azure hosted Postgres destination
  3. Airbyte version 0.40.0-alpha

airbyte_normalization_failure_postgres.txt (1.3 MB)

Good news, my issue may have been related to Postgres user permission conflict. It looks like the data moved fine once I cleaned up the permissions.