Summary
When dealing with multiple companies in custom connectors, should you create separate connectors for each company or have a single connector with parent and child streams? Consider trade-offs between separation and automation.
Question
Hi guys,
I have a question regarding best practices for custom connectors. I’ve set up a custom source connector for Microsoft BC that fetches data from a “Company”. I have multiple companies and we plan to add two new companies this year. Each company follows the same pattern, and there is only one parameter in the Base URL that changes.
What do you think:
Should I create a source connector for each company? This would lead to better separation, easier debugging, scheduling, and monitoring, but it would require creating multiple tables. Also, when I add a new company, I would have to duplicate a connector and adjust my DBT in the transformation stage.
Or, should I create a unique source connector where a parent stream fetches the list of companies and then runs child streams for each company, uploading data to the same table with an added field for the company? This would automate the process from EL to the Transform stage when a new company is created, but there would be no separation, which could lead to complex debugging.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connectors", "multiple-companies", "source-connector", "parent-stream", "child-stream", "separation", "automation", "debugging"]
Based on the information provided in the extracts, both approaches have their pros and cons, and the choice largely depends on your specific use case and requirements. Creating a separate source connector for each company would indeed provide better separation, easier debugging, scheduling, and monitoring. However, as you mentioned, it would require creating multiple tables and duplicating the connector each time a new company is added. This approach aligns with the concept of a connector in Airbyte, which defines the functionality of how to access an API or a database, including the way requests need to be structured and how to extract records from responses source. On the other hand, creating a unique source connector where a parent stream fetches the list of companies and then runs child streams for each company would automate the process from EL to the Transform stage when a new company is created. This approach seems to align with the concept of a “configured source” in Airbyte, which involves configuring a connector to actually extract records source. However, as you mentioned, this approach could lead to complex debugging due to the lack of separation. It’s also worth noting that Airbyte connectors can be written in any language, and you can reach out to the Airbyte team if you need help developing connectors in other languages source. In conclusion, the choice between these two approaches depends on your specific needs and constraints. You might want to consider factors such as the complexity of your data, the number of companies you plan to add in the future, and your capacity for maintaining multiple connectors versus a single, more complex connector.
Hey Xavier,
I would do one connector per company -> easier to debug and easier in case of customization
feel free to upvote this (standard Airbyte BC connector): https://github.com/airbytehq/airbyte/discussions/34382