I’m trying to build a database (SQL query based) connector using Python. All Python built connectors I’ve looked at are API/HTTP request based. Is there an example out there of a true database source built using Python? Specifically seeing how to handle the source.read method and whether or not streams need to be used (IE for processing data in chunks to give the nice “Records read: n” that Java based database connectors output).
Hi, @luke! Take a look at Source Faker and its read method, it does look like streams need to be used. I found another question from someone building a non API/HTTP Python connector here, it focuses on incremental sync if you need that later on. Let me know if this helps!
Luke,
The read function that the template generates gives a pretty straightforward example.
For each record and for each stream, you just emit AirbyteMessage. I think it goes by 1000s but it will generate “1000 records read” message in the log by default.
I just loop through all of the records and emit AirbyteMessage per record per stream. Works pretty fast. You can also utilize logger.info to print out more stats about your data.
I am surprised myself that most of the connectors are using API/HTTP method which to me was more complicated than implementing generic python source connector.
Good luck.