Hello, I have multiple postgres sources which I want to sync to same Bigquery Destination Namespace and same table names. I am appending data to the same table names from all those sources. I see that some tmp tables are getting created in between and are then getting deleted by the connection. As the name for the tmp tables from different sources will be same as the table names in those connections are same, I was just wondering can there be an issue while concurrently running these connections. Like one connection deletes tmp tables for another connection as the names will be same. Or can there be any other concurrency issue because the table names are same?
Hello @SP271,
As you described, there will indeed be concurrency issues. I would suggest you use different datasets (namespaces) for each of your connections and create a view that gathers the records from all of these datasets on BigQuery.
Hello @alafanechere Is there any other way to avoid it instead of creating different datasets? Like maybe using BigQuery(Denormalized) or any other way on your mind? There are too many connections that we have to make that will indeed make too many datasets.
Is there any other way to avoid it instead of creating different datasets?
The only workaround I can currently suggest is to make sure a single sync is running anytime to avoid concurrency issues. You can set the MAX_SYNC_WORKERS value to 1.
I raised this feedback to our product team and I hope that we will soon be able to tackle this limitiation.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.