Summary
Creating a custom connector in Python CDK to request an API, wait for report generation, and download it while following best practices. Inquiring about inheriting from existing classes like HttpStream in Airbyte Python CDK.
Question
Hello everyone.
I need to create a custom connector using the python CDK, my source requires me to request the api first, wait for a report to be generated and when it’s done download it. How can I implement this code following the best practices? Is it possible to inherit something from existing classes in airbyte python cdk such as HttpStream?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connector", "python-cdk", "api-request", "report-generation", "download", "best-practices", "inheritance", "httpstream"]
Hai,
What we did is:
Make a new python file that imports requests.
Write a function to get_request and post_request. With url, params and header as arguments in the functions.
You can import the file into your main.py file to get the data.
response = get_request(url, params, header)
You can get the url from your spec/config.
Make the header with auth where you can get a token from the spec/config.
Then you can fill the token and url in when setting up the connection in the airbyte webUI
or hardcode this.
In your main code you could do your first api request.
do a wait or maybe you can do a request every so often to check if the report is done generating and then download the report.
For downloading the report you could add the function in the new python file with the get_request and post_request functions.
Give arguments like url, download path. and call the function in you main.py