Summary
The user wants to increase the record chunk size for streaming data from Zendesk to BigQuery using Airbyte on Kubernetes. They are looking to scale the resource for the job to handle larger data volumes.
Question
Hey team,
I setup stream from Zendesk to BigQuery with Airbyte on K8S, I need to stream a large data, however I saw in log it’s 5000 record per chunk. How I can increase it? I tried to find a variable to change, but I cannot find it. I can scale resource for job larger.
2024-03-11 04:18:13 replication-orchestrator > Attempt 0 to update stream status running null:ticket_audits
2024-03-11 04:19:10 replication-orchestrator > Records read: 5000 (24 MB)
2024-03-11 04:20:05 replication-orchestrator > Records read: 10000 (51 MB)
2024-03-11 04:20:58 replication-orchestrator > Records read: 15000 (79 MB)
2024-03-11 04:21:49 replication-orchestrator > Records read: 20000 (102 MB)
2024-03-11 04:22:45 replication-orchestrator > Records read: 25000 (126 MB)
2024-03-11 04:23:37 replication-orchestrator > Records read: 30000 (150 MB)
2024-03-11 04:24:34 replication-orchestrator > Records read: 35000 (177 MB)
2024-03-11 04:25:32 replication-orchestrator > Records read: 40000 (206 MB)
2024-03-11 04:26:27 replication-orchestrator > Records read: 45000 (234 MB)
2024-03-11 04:27:25 replication-orchestrator > Records read: 50000 (261 MB)
2024-03-11 04:28:20 replication-orchestrator > Records read: 55000 (288 MB)
2024-03-11 04:29:15 replication-orchestrator > Records read: 60000 (314 MB)
2024-03-11 04:30:09 replication-orchestrator > Records read: 65000 (340 MB)
2024-03-11 04:31:01 replication-orchestrator > Records read: 70000 (365 MB)
2024-03-11 04:31:53 replication-orchestrator > Records read: 75000 (389 MB)
2024-03-11 04:32:49 replication-orchestrator > Records read: 80000 (417 MB)
2024-03-11 04:33:47 replication-orchestrator > Records read: 85000 (442 MB)
2024-03-11 04:34:36 replication-orchestrator > Records read: 90000 (466 MB)
2024-03-11 04:35:25 replication-orchestrator > Records read: 95000 (488 MB)```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1710132040601019) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["streaming", "zendesk", "bigquery", "airbyte", "k8s", "record-chunk-size", "scaling-resource"]
</sub>