Summary
Error occurs when syncing a large table with over 50 million rows from Oracle to local Postgres using Airbyte with Docker. The error message suggests unexpected data beyond EOF in a relation block, potentially related to buggy kernels.
Question
Hi Everyone, I keep getting this error when trying to sync large table over 50 Million rows, all I did is changing the cpu and memory request to 4 cpu and 4g memory.
I’m using airbyte with docker and the source database is oracle and destination is local postgres, postgres also running on docker.
2741 Hint: This has been seen to occur with buggy kernels; consider updating your system.
2742 Where: COPY public_raw__stream_fdebts_primium_micro_daily, line 234
2743 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.lambda$insertRecordsInternal$0(PostgresSqlOperations.java:92)
2744 at io.airbyte.cdk.db.jdbc.DefaultJdbcDatabase.execute(DefaultJdbcDatabase.kt:30)
2745 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.insertRecordsInternal(PostgresSqlOperations.java:81)
2746 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.insertRecordsInternalV2(PostgresSqlOperations.java:53)
2747 at io.airbyte.cdk.integrations.destination.jdbc.JdbcSqlOperations.insertRecords(JdbcSqlOperations.kt:231)
2748 at io.airbyte.cdk.integrations.destination.jdbc.JdbcBufferedConsumerFactory.recordWriterFunction$lambda$5(JdbcBufferedConsumerFactory.kt:302)
2749 at io.airbyte.cdk.integrations.destination.jdbc.JdbcInsertFlushFunction.flush(JdbcInsertFlushFunction.kt:19)
2750 at io.airbyte.cdk.integrations.destination.async.FlushWorkers.flush$lambda$6(FlushWorkers.kt:172)
2751 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2752 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2753 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2754 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2755 at java.base/java.lang.Thread.run(Thread.java:1583)
2756 Caused by: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 26789 of relation base/224213/233904
2757 Hint: This has been seen to occur with buggy kernels; consider updating your system.
2758 Where: COPY public_raw__stream_fdebts_primium_micro_daily, line 234
2759 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
2760 at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1299)
2761 at org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImpl.java:1095)
2762 at org.postgresql.core.v3.CopyInImpl.endCopy(CopyInImpl.java:49)
2763 at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:185)
2764 at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:160)
2765 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.lambda$insertRecordsInternal$0(PostgresSqlOperations.java:90)```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1716711511086769) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["error", "syncing-large-table", "oracle", "postgres", "airbyte", "docker", "bug", "relation-block", "buggy-kernels"]
</sub>