Summary
When syncing a large table from Postgres to BigQuery using Airbyte, encountering ‘Result set exceeded maxResultBuffer limit’ error. Seeking advice on resolving this issue.
Question
Hello everyone,
I’m using airbyte to push data from postgres to big query
Initially I enebaled WAL(postgres) to track deletion of the records but the amount of logs generated was so huge I was not able to bare the server size expansion, hence I have decided to go ahead based on cursor that is based on timestamp and serially generated id.
The issue I’m facing right now is on syncing big table, I have table of size 500gb which needs to be synced to big query
I’m getting following error
Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: Result set exceeded maxResultBuffer limit. Received: 161061290; Current limit: 161061273
Has anyone faced this error? Any advice on how to get it solved?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.