Connection from BigQuery to RedShift is hanging, is this normal?
Is it normal that a connection from BigQuery to RedShift is hanging
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Do you have any idea about many tables Bigquery have?
<@U02TQLBLDU4> I have around 10 schemas and each schema has less than 5 tables
I’m already exporting data from Google Analytics 4 into BigQuery using what they call a BigQuery link as shown in the screenshot, the resulting table in BigQuery looks like shown as well.
I’m not sure if I can do the same with Airbyte into RedShift
<@U04UU36NK0F> Something to note is that the GA4 export uses sharded tables, which group together in the UI but will be at least one table per day (e.g. events_20240419) plus an intraday table, and then sometimes user tables and such depending on the options selected on the link. (The number next to the table name in BigQuery tells you how many table shards exist—which will increase every day.)
It’s also worth noting that the GA4 schema has a large number of nested fields. I’m not sure how Airbyte pulls this data, but in systems that loop over all of them it can slow things down at scale.
As for Redshift, no—the only option is BigQuery for raw data. But you could use Airbyte or other CDC tools to relay that data to Redshift.