Summary
User is seeking guidance on how to increase the record chunk size from 5000 while streaming data from Zendesk to BigQuery using Airbyte on Kubernetes. They have already attempted to locate a variable for this adjustment without success.
Question
Hey team,
I setup stream from Zendesk to BigQuery with Airbyte on K8S, I need to stream a large data, however I saw in log it’s 5000 record per chunk. How I can increase it? I tried to find a variable to change, but I cannot find it. I can scale resource for job larger.
2024-03-11 04:18:13 replication-orchestrator > Attempt 0 to update stream status running null:ticket_audits
2024-03-11 04:19:10 replication-orchestrator > Records read: 5000 (24 MB)
2024-03-11 04:20:05 replication-orchestrator > Records read: 10000 (51 MB)
2024-03-11 04:20:58 replication-orchestrator > Records read: 15000 (79 MB)
2024-03-11 04:21:49 replication-orchestrator > Records read: 20000 (102 MB)
2024-03-11 04:22:45 replication-orchestrator > Records read: 25000 (126 MB)
2024-03-11 04:23:37 replication-orchestrator > Records read: 30000 (150 MB)
2024-03-11 04:24:34 replication-orchestrator > Records read: 35000 (177 MB)
2024-03-11 04:25:32 replication-orchestrator > Records read: 40000 (206 MB)
2024-03-11 04:26:27 replication-orchestrator > Records read: 45000 (234 MB)
2024-03-11 04:27:25 replication-orchestrator > Records read: 50000 (261 MB)
2024-03-11 04:28:20 replication-orchestrator > Records read: 55000 (288 MB)
2024-03-11 04:29:15 replication-orchestrator > Records read: 60000 (314 MB)
2024-03-11 04:30:09 replication-orchestrator > Records read: 65000 (340 MB)
2024-03-11 04:31:01 replication-orchestrator > Records read: 70000 (365 MB)
2024-03-11 04:31:53 replication-orchestrator > Records read: 75000 (389 MB)
2024-03-11 04:32:49 replication-orchestrator > Records read: 80000 (417 MB)
2024-03-11 04:33:47 replication-orchestrator > Records read: 85000 (442 MB)
2024-03-11 04:34:36 replication-orchestrator > Records read: 90000 (466 MB)
2024-03-11 04:35:25 replication-orchestrator > Records read: 95000 (488 MB)```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1710132040601019) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['zendesk', 'bigquery', 'airbyte', 'kubernetes', 'chunk-size', 'data-streaming']
</sub>