Summary
The user is asking if it’s possible to write multiple streams to a single destination table in BigQuery to avoid creating multiple raw tables with the same DDL. They are considering creating a new connector for BigQuery using low-code CDK and the bigquery_storage library. The user has created the source but is unsure about controlling the date range introduced in the source.
Question
Hi Community !
Does anyone know if it’s possible to write multiple streams to an unique destination table ?
For example, in BigQuery source, I want to write each stream (one per day) to the same table, because I don’t want multiple raw tables when they have the same DDL…
If not, I’m thinking on creating a new connector using low-code CDK, specific for BigQuery (GA4 specific) that uses bigquery_storage library (so fast comparing with classic google.cloud.bigquery classic library) and define a common destination in any way, because all streams has the same schema.
I’m starting to do this and I have created the source (AbstractSource) now, but I have a doubt:
• How can I control the date range introduced in source ? I don’t understand really well the relationship between date range introduced and the streams to discover.
◦ I need to make requests individually by day (or session, with Google BigQuery ReadSession).
https://cloud.google.com/bigquery/docs/reference/storage
Any suggestion ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.