Summary
User is experiencing issues with fetching data from MS SQL using CDC method in Airbyte while trying to extract data from the last 6 days. The system works fine for the last 6-8 hours but fails for longer periods, and user seeks guidance on error logs and the implications of changing queue size in MS SQL connector.
Question
Hi ,
I am bit stuck on Airbyte , Can anyone please help me !!
Source :- MS SQL (Airbyte Connector Version :- 4.1.16)
Destination :- S3 (Airbyte Destination Version :- 1.4.0)
Airbyte Version :- 1.1.1
Deployed on Kubernetes (AWS) , Using CDC method to fetch data from MS SQL & dump it to S3 Bucket in JSONL Zipped Format , I need to Extract data from MS SQL for Last 6 days So I changed the LSN (Log Sequence Number) & set it to that Date & Airbyte is unable to fetch the data & it kept throwing similar Logs , I waited for 30-40 Min but no difference
When I changed the LSN to fetch data for Last 6-8 Hours it worked Fine , Can anyone guide me on this ?
2024-12-08 15:30:00 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT0.00000021S after 999 previous call(s) which were not logged.
Also Did any one know about Size of the queue (Advanced) what is that Parameter in MS SQL Airbyte Connector & , If i increase or decrease the Size of the Queue what impact it will have ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.