Writing multiple streams to a unique destination table in BigQuery

Summary

The user is asking if it’s possible to write multiple streams to a single destination table in BigQuery to avoid creating multiple raw tables with the same DDL. They are considering creating a new connector for BigQuery using low-code CDK and the bigquery_storage library. The user has created the source but is unsure about controlling the date range introduced in the source.


Question

Hi Community !

Does anyone know if it’s possible to write multiple streams to an unique destination table ?
For example, in BigQuery source, I want to write each stream (one per day) to the same table, because I don’t want multiple raw tables when they have the same DDL…

If not, I’m thinking on creating a new connector using low-code CDK, specific for BigQuery (GA4 specific) that uses bigquery_storage library (so fast comparing with classic google.cloud.bigquery classic library) and define a common destination in any way, because all streams has the same schema.

I’m starting to do this and I have created the source (AbstractSource) now, but I have a doubt:

• How can I control the date range introduced in source ? I don’t understand really well the relationship between date range introduced and the streams to discover.
◦ I need to make requests individually by day (or session, with Google BigQuery ReadSession).
https://cloud.google.com/bigquery/docs/reference/storage

Any suggestion ?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["multiple-streams", "unique-destination-table", "bigquery", "connector", "low-code", "cdk", "bigquery-storage", "date-range", "google-bigquery-readsession"]