Building Airbyte Connector with GraphQL Endpoint and Multiple Requests

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is inquiring about building an Airbyte Connector with a GraphQL endpoint and multiple requests to access and download CSV documents for transportation.


Question

Hello, everyone.

I’m designated to build an Airbyte Connector, but I’m not sure if it’s possible due to the requirements. I’d like to know of you if it’s possible to build this connector only using the low-code builder, or if it’ll be necessary to use the CDK. Here follows the requirements:

• The endpoint uses GraphQL languages to access data.
• I have to make three requests to achieve my data:
◦ 1: A request to start the doc processing
◦ 2: A request to get the ID’s of processed documents. This requests return’s also a status of the document which can be PENDING, COMPLETED, etc. I need to wait all documents to be COMPLETED to move to the next step.
◦ 3: A request that uses the ID’s to get the URL’s to download the documents.
• The downloaded document is the main goal here. It’is a CSV to be downloaded and transformed to be transported to destination.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte-connector", "graphql-endpoint", "multiple-requests", "csv-document", "low-code-builder", "cdk"]
2

Take a look at source-sendgrid. It’s a hybrid connector that uses lowcode for most streams, but one particular stream (Contacts) is built in Python cdk, it requests an export, waits for it to be available, then downloads and parses the CSV file of the report.

That’s pretty close to what you want, except for the graphql request part.

3

https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview#what-connectors-can-i-build-using-the-low-code-framework
You can find useful list of question in section What connectors can I build using the low-code framework?
> If the answer to all questions is yes, you can use the low-code framework to build a connector for the source. If not, use the <https://docs.airbyte.com/connector-development/cdk-python/|Python CDK>.
For me, it looks like CDK is a way to go