Summary
User is experiencing a RuntimeException related to maxResultBuffer limit while syncing a 500GB table from Postgres to BigQuery using Airbyte. The user switched from WAL to cursor-based syncing due to log size issues.
Question
Hello everyone,
I’m using airbyte to push data from postgres to big query
Initially I enebaled WAL(postgres) to track deletion of the records but the amount of logs generated was so huge I was not able to bare the server size expansion, hence I have decided to go ahead based on cursor that is based on timestamp and serially generated id.
The issue I’m facing right now is on syncing big table, I have table of size 500gb which needs to be synced to big query
I’m getting following error
Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: Result set exceeded maxResultBuffer limit. Received: 161061290; Current limit: 161061273
Has anyone faced this error? Any advice on how to get it solved?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.