Issue with Custom Connector Builder for Incremental Streams

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Custom connector built using Builder is experiencing long run times and synchronizing large amounts of data unexpectedly. Looking for guidance on debugging the issue related to Incremental Streams configuration.


Question

Hi, I built a custom connector using Builder, but sometimes it works successfully, and sometimes it keeps running and synchronising data for more than 28GB and running for more than 10hours. Usually, it finishes within 30 minutes. The connector runs every 3 hours, and I am not expecting this amount of data to be updated quickly. Has anyone experienced an issue in Custom Builder for Incremental Streams? How should I debug that? Logs are not much helping, it just keep adding this log
Attempting to read from queue namespace: {{stream_name}}
How can I check or debug what is happening if anything is wrong in the builder stream configuration?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-connector", "builder", "incremental-streams", "debugging", "logs"]
2

Hmm, I don’t know enough to give you a confident answer yet. However, you can refer to the Airbyte documentation for more information on how to use the connector builder and ensure that all streams are included in your connectors. This might help you identify any potential issues with your current setup.