Building Custom Connector for API with Specific Parameters Order

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is trying to build a custom connector for an API that requires specific parameter order, including a URL endpoint parameter. They are facing issues with Airbyte’s Custom Connector UI not maintaining parameter order and converting special characters incorrectly. The user is using the Python connector option but encountering an error related to the schema when running the connection.


Question

I’m trying to build a custom connector for this API

https://developer.chrobinson.com/api-reference#tag/Events

to extract only the “event” documents.

This API has some peculiarities, like the parameters need to be sent in a very specific order, and one of these parameters is a URL to an endpoint, this is an example:
https://server_api.com/http/QueryCall_QA?endPoint=https://sandbox-
http://api.navisphere.com/v2/events?skip=10&take=10&httpMethod=GET|api.navisphere.com/v2/events?skip=10&take=10&httpMethod=GET

The Airbyte Custom Connector UI can’t guarantee the correct order of parameters when pagination or dates are passed. Also for this endpoint parameter “https://sandbox-
http://api.navisphere.com/v2/events|api.navisphere.com/v2/events?” (yes, the question mark at the end is required) the Airbyte converts to HTML characters like “http://sandbox-.
http://api.navisphere.com|api.navisphere.com/v2/events?” and the API rejects these special characters. So the Python version is the only way to manipulate the request into the format the API accepts.

I’m using the CDK, selected the “Python connector” option, and started from the template. But I’m getting the error when the connection runs:

2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.integrations.base.destination.typing_deduping.CatalogParser.toStreamConfig(CatalogParser.java:107) ~[airbyte-cdk-typing-deduping-0.16.4.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.integrations.base.destination.typing_deduping.CatalogParser.parseCatalog(CatalogParser.java:44) ~[airbyte-cdk-typing-deduping-0.16.4.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.parseCatalog(BigQueryDestination.java:433) ~[io.airbyte.airbyte-integrations.connectors-destination-bigquery-0.50.45.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.getSerializedMessageConsumer(BigQueryDestination.java:236) ~[io.airbyte.airbyte-integrations.connectors-destination-bigquery-0.50.45.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:189) ~[airbyte-cdk-core-0.16.4.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.16.4.jar:?]
2024-05-21 19:39:58 [43mdestination[0m > 	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:468) ~[io.airbyte.airbyte-integrations.connectors-destination-bigquery-0.50.45.jar:?]
2024-05-21 19:39:58 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromDstRunnable$4):323 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@117369b4[type=TRACE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=<null>,trace=io.airbyte.protocol.models.AirbyteTraceMessage@1fb495ea[type=ERROR,emittedAt=1.716320398356E12,error=io.airbyte.protocol.models.AirbyteErrorTraceMessage@1b8b02c4[message=Something went wrong in the connector. See the logs for more details.,internalMessage=java.lang.IllegalArgumentException: Top-level schema must be an object,stackTrace=java.lang.IllegalArgumentException: Top-level schema must be an object
	at io.airbyte.integrations.base.destination.typing_deduping.CatalogParser.toStreamConfig(CatalogParser.java:107)
	at io.airbyte.integrations.base.destination.typing_deduping.CatalogParser.parseCatalog(CatalogParser.java:44)
	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.parseCatalog(BigQueryDestination.java:433)
	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.getSerializedMessageConsumer(BigQueryDestination.java:236)
	at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:189)
	at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
	at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:468)
,failureType=system_error,additionalProperties={}],estimate=<null>,additionalProperties={}],control=<null>,additionalProperties={}]```

I tried to change the schema used, tried to implement schemaless, tried to change the code, without success.
Can someone help me, please?
CC: <@U070V50N0BT> <@U06SV3WK399>

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1716383478461479) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["custom-connector", "api", "specific-parameters", "python-connector", "parameter-order", "url-endpoint", "special-characters", "schema-error"]
</sub>