Summary
The user is facing an issue with a connector sending data from BigQuery to a webservice in batches. The error message indicates a missing catalog stream for a specific stream. The issue occurs in the production environment but not in staging. The user suspects the problem might be in the source connector of BigQuery.
Question
hello,
I am facing a problem with a connector i create. The connection it consuming data from BigQuery and it send it on a webservice by batch of 100 éléments. It works on our staging’s environnement but not in production. I feel that it’s not a problem in the connector I created, but in the source connector of bigquery. I tried to remove the connection and recreate it but nothing make it works. The error is Missing catalog stream for the stream (namespace: event_user_updated, name: null
"failureOrigin" : "source",
"internalMessage" : "Missing catalog stream for the stream (namespace: event_user_updated, name: null",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 3,
"jobId" : 10649,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Missing catalog stream for the stream (namespace: event_user_updated, name: null\n\tat io.airbyte.workers.internal.BasicAirbyteMessageValidator.validate(BasicAirbyteMessageValidator.java:63)\n\tat io.airbyte.workers.internal.VersionedAirbyteStreamFactory.toAirbyteMessage(VersionedAirbyteStreamFactory.java:327)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:541)\n\tat java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1950)\n\tat java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)\n\tat java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)\n\tat java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)\n\tat java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)\n\tat java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.isFinished(DefaultAirbyteDestination.java:212)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.destinationIsFinished(BufferedReplicationWorker.java:512)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:469)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:219)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1729256510559
}, {
"failureOrigin" : "destination",
"internalMessage" : "Destination process message delivery failed",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 3,
"jobId" : 10649,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process message delivery failed\n\tat io.airbyte.workers.general.BufferedReplicationWorker.writeToDestination(BufferedReplicationWorker.java:455)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithTimeout$5(BufferedReplicationWorker.java:247)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.\n\tat com.google.common.base.Preconditions.checkState(Preconditions.java:515)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:141)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.writeToDestination(BufferedReplicationWorker.java:444)\n\t... 5 more\n",
"timestamp" : 1729256520449
}, {
"failureOrigin" : "source",
"internalMessage" : "Source process read attempt failed",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 3,
"jobId" : 10649,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process read attempt failed\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:379)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:226)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.\n\tat com.google.common.base.Preconditions.checkState(Preconditions.java:515)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:141)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:359)\n\t... 5 more\n",
"timestamp" : 1729256520533
} ]```
Any help or advice is welcome. Thanks you :slightly_smiling_face:
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1729256973979569) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["bigquery-source-connector", "data-batch", "missing-catalog-stream", "production-environment"]
</sub>