Error logs when syncing BigQuery table to pgsql

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Logs show table truncation errors during synchronization from BigQuery to pgsql. User is seeking assistance in identifying the reason for the unsuccessful synchronization.


Question

Hi, I’d like to ask about the logs I encountered when syncing a BigQuery table to pgsql. It seems like the table synchronization wasn’t successful. Does anyone know the reason for this? I’m using Airbyte 0.50.50.

2024-10-22 08:04:13 normalization > Truncating data_analytics_user_wallets_records_raw_changelog (#49) to data_analytics_user__records_raw_changelog (#43)
2024-10-22 08:04:13 normalization > Truncating data_analytics_user_wallets_records_raw_latest (#46) to data_analytics_user__ts_records_raw_latest (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_audiences_creators (#47) to timed_events_launch__gn_audiences_creators (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_audiences_creators_audit_logs (#58) to timed_events_launch__s_creators_audit_logs (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_audiences_stablecoin_claim_records (#63) to timed_events_launch__blecoin_claim_records (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_audiences_transactions (#51) to timed_events_launch__udiences_transactions (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_creators_audit_logs (#48) to timed_events_launch__n_creators_audit_logs (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_creators_stablecoin_claim_records (#62) to timed_events_launch__blecoin_claim_records (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_creators_transactions (#50) to timed_events_launch__creators_transactions (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_audiences_stablecoin_claim_records_26f (#67) to timed_events_launch__oin_claim_records_26f (#43)
2024-10-22 08:04:13 normalization > Truncating timed_events_launch_campaign_creators_stablecoin_claim_records_880 (#66) to timed_events_launch__oin_claim_records_880 (#43)```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1729584878665419) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["logs", "syncing", "bigquery-table", "pgsql", "airbyte-0.50.50", "truncation-error"]
</sub>
2

Why do you think that synchronization wasn’t successful? The logs show only that names were shortened.

3

Oh, yes, I just checked, and indeed there is a truncated table.
But how can we determine the final synchronized table name in the Airbyte web? Do we have to rely solely on the logs to find out?

4

It’s some kind of behavior provided by normalization. You should see streams names in connection configuration.
Do you use normalization/transformations? Maybe you can turn it off.
I didn’t have much opportunity to use this feature because it was removed in newer Airbyte versions.

5

I have encountered another error. I’m not sure if you have come across it as well.

io.airbyte.api.client.invoker.generated.ApiException: getWorkspace call failed with: 404 - {"message":"Internal Server Error: Could not find configuration for STANDARD_WORKSPACE: b41258aa-68d8-4331-9473-20f63c1a68a2.","exceptionClassName":"io.airbyte.commons.server.errors.IdNotFoundKnownException","exceptionStack":["io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find configuration for STANDARD_WORKSPACE: b41258aa-68d8-4331-9473-20f63c1a68a2.","\tat io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:32)","\tat io.airbyte.server.apis.WorkspaceApiController.getWorkspace(WorkspaceApiController.java:133)","\tat io.airbyte.server.apis.$WorkspaceApiController$Definition$Exec.dispatch(Unknown Source)","\tat io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371)","\tat io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594)","\tat io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)","\tat io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)","\tat io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)","\tat io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)","\tat reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)","\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)","\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)","\tat io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62)","\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)","\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)","\tat io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)","\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)","\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)","\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)","\tat java.base/java.lang.Thread.run(Thread.java:1583)","Caused by: io.airbyte.config.persistence.ConfigNotFoundException: config type: STANDARD_WORKSPACE id: b41258aa-68d8-4331-9473-20f63c1a68a2","\tat io.airbyte.config.persistence.ConfigRepository.getStandardWorkspaceNoSecrets(ConfigRepository.java:263)","\tat io.airbyte.commons.server.handlers.WorkspacesHandler.getWorkspace(WorkspacesHandler.java:343)","\tat io.airbyte.server.apis.WorkspaceApiController.lambda$getWorkspace$4(WorkspaceApiController.java:133)","\tat io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28)","\t... 19 more"],"rootCauseExceptionClassName":"java.lang.Class","rootCauseExceptionStack":["io.airbyte.config.persistence.ConfigNotFoundException: config type: STANDARD_WORKSPACE id: b41258aa-68d8-4331-9473-20f63c1a68a2","\tat io.airbyte.config.persistence.ConfigRepository.getStandardWorkspaceNoSecrets(ConfigRepository.java:263)","\tat io.airbyte.commons.server.handlers.WorkspacesHandler.getWorkspace(WorkspacesHandler.java:343)","\tat io.airbyte.server.apis.WorkspaceApiController.lambda$getWorkspace$4(WorkspaceApiController.java:133)","\tat io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28)","\tat io.airbyte.server.apis.WorkspaceApiController.getWorkspace(WorkspaceApiController.java:133)","\tat io.airbyte.server.apis.$WorkspaceApiController$Definition$Exec.dispatch(Unknown Source)","\tat io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371)","\tat io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594)","\tat io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)","\tat io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)","\tat io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)","\tat io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)","\tat reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)","\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)","\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)","\tat io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62)","\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)","\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)","\tat io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)","\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)","\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)","\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)","\tat java.base/java.lang.Thread.run(Thread.java:1583)"]}
        at io.airbyte.api.client.generated.WorkspaceApi.getApiException(WorkspaceApi.java:83) ~[io.airbyte-airbyte-api-0.50.54.jar:?]
        at io.airbyte.api.client.generated.WorkspaceApi.getWorkspaceWithHttpInfo(WorkspaceApi.java:438) ~[io.airbyte-airbyte-api-0.50.54.jar:?]
        at io.airbyte.api.client.generated.WorkspaceApi.getWorkspace(WorkspaceApi.java:416) ~[io.airbyte-airbyte-api-0.50.54.jar:?]
        at io.airbyte.workers.config.AnalyticsTrackingBeanFactory.workspaceFetcher$lambda$1(AnalyticsTrackingBeanFactory.kt:36) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.analytics.TrackingIdentityFetcher.apply(TrackingClient.kt:354) ~[io.airbyte-airbyte-analytics-0.50.54.jar:?]
        at io.airbyte.analytics.SegmentTrackingClient.track(TrackingClient.kt:158) ~[io.airbyte-airbyte-analytics-0.50.54.jar:?]
        at io.airbyte.workers.internal.bookkeeping.ParallelStreamStatsTracker.trackStateCountMetrics(ParallelStreamStatsTracker.kt:321) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.internal.bookkeeping.ParallelStreamStatsTracker.validateStateChecksum(ParallelStreamStatsTracker.kt:251) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.internal.bookkeeping.ParallelStreamStatsTracker.validateStateChecksum$default(ParallelStreamStatsTracker.kt:189) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.internal.bookkeeping.ParallelStreamStatsTracker.updateSourceStatesStats(ParallelStreamStatsTracker.kt:115) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.internal.syncpersistence.SyncPersistenceImpl.updateSourceStatesStats(SyncPersistence.kt:403) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.internal.bookkeeping.AirbyteMessageTracker.acceptFromSource(AirbyteMessageTracker.kt:45) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.general.ReplicationWorkerHelper.internalProcessMessageFromSource(ReplicationWorkerHelper.kt:343) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.general.ReplicationWorkerHelper.processMessageFromSource(ReplicationWorkerHelper.kt:418) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.processMessage(BufferedReplicationWorker.java:422) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:252) ~[io.airbyte-airbyte-commons-worker-0.50.54.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]```
6

have you created and removed any workspaces?
does this path work in your browser &lt;your-airbyte-host&gt;/workspaces/b41258aa-68d8-4331-9473-20f63c1a68a2/connections ?

7

for new unrelated errors/issues/questions, I recommend starting new threads, so they are more visible on channel and other people can answer in them as well