Custom MySQL Source Connector Output Formatting in Java

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Building a custom MySQL source connector in Java to improve performance over existing Python implementation. Facing challenges with output formatting, specifically needing columns with metadata and aggregated record data in a JSON_SCR column.


Question

Hi everyone,

I’m currently working on building a custom MySQL source connector in Java to improve performance over my existing Python implementation, which is proving to be too slow for handling large volumes of data.

The main challenge I’m facing is with output formatting. I need the output to consist of columns with metadata, including information such as the source identifier and a unique load hash. Additionally, all the record data of the source should be aggregated into a JSON_SCR column. This would change the Airbyte message.

Could anyone provide guidance on how to achieve this within the Java connector framework? I’ve already implemented the connector in Python, so I have a solid understanding of the logic required, but I’m struggling with the Java specifics for output formatting.
Any advice, code snippets, or resources you could share would be greatly appreciated.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-mysql-source-connector", "java", "output-formatting", "metadata", "json-column", "performance"]
2

Hey <@U05TL3G9HSS>, thank you for a detailed post!

I doubt folks in this channel will be able to help. It would be a good idea to write a detailed github issue and tag @natikgadzhi so I can route it accordingly.

If your source-mysql change would benefit others and bring better performance, we would be happy to guide you through it. If it’s a one-off specific to your usecase, hopefully we can at list give you some hints.

3

One note is that we are actively moving the Java source CDK to Kotlin, and most sources are in progress or moved.