500 Error on 'Finish & Sync' with custom source connector built with low-code builder UI

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Encountering a 500 error with a custom source connector built using a low-code builder UI, resulting in a NullPointerException related to JSON handling in the airbyte-server-container output.


Question

:red_circle: Title: 500 Error on “Finish & Sync” with a custom source connector built with low-code builder UI (see picture)

:question: What I’m doing: Testing a custom source connector (<https://app.makosoftware.de/docs/api/|Makoflow API>) built with the Builder Web UI running locally via abctl install

:eyes: Error Message from airbyte-server-container output from crictl logs container-id``

java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.fieldNames()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null
	at io.airbyte.commons.server.validation.CatalogValidator.fieldCount(CatalogValidator.kt:39) ~[io.airbyte-airbyte-commons-server-0.61.0.jar:?]
	at io.airbyte.commons.server.handlers.ConnectionsHandler.validateCatalogSize(ConnectionsHandler.java:1172) ~[io.airbyte-airbyte-commons-server-0.61.0.jar:?]
	at io.airbyte.commons.server.handlers.ConnectionsHandler.createConnection(ConnectionsHandler.java:507) ~[io.airbyte-airbyte-commons-server-0.61.0.jar:?]
	at io.airbyte.commons.server.handlers.WebBackendConnectionsHandler.webBackendCreateConnection(WebBackendConnectionsHandler.java:564) ~[io.airbyte-airbyte-commons-server-0.61.0.jar:?]
	at io.airbyte.server.apis.WebBackendApiController.lambda$webBackendCreateConnection$1(WebBackendApiController.java:88) ~[io.airbyte-airbyte-server-0.61.0.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28) ~[io.airbyte-airbyte-server-0.61.0.jar:?]
	at io.airbyte.server.apis.WebBackendApiController.webBackendCreateConnection(WebBackendApiController.java:86) ~[io.airbyte-airbyte-server-0.61.0.jar:?]
	at io.airbyte.server.apis.$WebBackendApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.61.0.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4232) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:271) ~[micronaut-router-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488) ~[micronaut-http-server-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.RouteExecutor.lambda$callRoute$6(RouteExecutor.java:465) ~[micronaut-http-server-4.4.8.jar:4.4.8]
	at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87) ~[micronaut-core-4.4.8.jar:4.4.8]
	at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211) ~[micronaut-core-4.4.8.jar:4.4.8]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]```
See thread for source connector YAML

:green_heart: *Call to action: Has anyone experienced this 500 error (null value instead of json) output from airbyte-server-container when setting up a new custom source connector?*

(could it be related to the fact that little or no data is available at some of the source API endpoints ?) ... IDK

:pray: any comments, thoughts, suggstions welcome! THANK YOU!!

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1716747525392389) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["500-error", "custom-source-connector", "low-code-builder", "null-pointer-exception", "json-handling", "airbyte-server-container"]
</sub>
2

Source connector yaml here! :wink:

3

Validate catalog size hints at something being weird with the lost of streams / their columns that is returned. Weird. Airbyte Cloud I assume?

4

Ah, not Airbyte cloud. Interesting.

5

Try that in cloud? Just to verify on current new version.

6

Just configured the same connector with a Postgres destination in Airbyte Cloud and got the same 500 error

7

what’s validate catalog size? I’ve run spec discover read and check using the source connector cli source-makoflow as well (I developed the connector in VS Code originally and then imported the YAML to the Builder for ez-r deployment

8

What if you try to only keep one stream in check key of the yaml?

9

Hey <@U069EMNRPA4> Any thoughts here? I am really keen on Airbyte for our API source integrations but when setting up another custom low-code CDK connector, I’m getting this same 500 error: “An unexpected error occurred. Please report this if the issue persists. (HTTP 500)”

Should I report this as a bug on GitHub?

This new custom connector manifest.yaml is in the snippet.

10

Here are the screenshots for steps 3 & 4 of connector setup.

11

Just the line that spotted my attention in the stacktrace. I’m not entirely sure what’s broken and pretty slammed this week — won’t be able to reply quickly.

12

I reduced the number of streams in check to just one buildings and I received the same 500 error on ‘Set up connection’

13

You can see that in the Builder UI the buildings stream successfully retrieves the test data available currently:

14

by the way - I was able to get one of these custom connectors working on airbyte cloud (no 500 error) when I added the schema inline the connector manifest.yaml. I guess the default is JSON schema loader and so it was throwing a null pointer when looking for the json?