Uncaught exception when configuring custom source connector

aarsco · May 25, 2022, 7:03pm

Is this your first time deploying Airbyte?: No
OS Version / Instance: Ubuntu
Memory / Disk: you can use something like 16Gb / 100 Gb
Deployment: Docker
Airbyte Version: 0.39.1-alpha
Source name/version: Custom
Destination name/version: Snowflake
Step: Source Configuration
Description:

I built a custom source connector that has been working fine for quite some time on an older version of Airbyte (0.35.X I believe). However, when trying to upgrade to the current Airbyte version, I can no longer get the source configured as a jsonSchemaNode exception keeps popping up.

Internal Server Error: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.isObject()" because "jsonSchemaNode" is null

I can run check, discover, and read locally without problem

python main.py discover --config secrets/config.json

I can’t make much from the server log so any direction as to where to start looking would be much appreciated:

2022-05-25 18:23:44 ERROR i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.isObject()" because "jsonSchemaNode" is null
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:152) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:168) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:177) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchema(JsonSchemas.java:99) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaWithCollector(JsonSchemas.java:116) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.commons.json.JsonSchemas.collectJsonPathsThatMeetCondition(JsonSchemas.java:129) ~[io.airbyte-airbyte-commons-0.39.1-alpha.jar:?]
	at io.airbyte.config.persistence.split_secrets.SecretsHelpers.getSortedSecretPaths(SecretsHelpers.java:173) ~[io.airbyte.airbyte-config-persistence-0.39.1-alpha.jar:?]
	at io.airbyte.config.persistence.split_secrets.SecretsHelpers.internalSplitAndUpdateConfig(SecretsHelpers.java:233) ~[io.airbyte.airbyte-config-persistence-0.39.1-alpha.jar:?]
	at io.airbyte.config.persistence.split_secrets.SecretsHelpers.splitConfig(SecretsHelpers.java:65) ~[io.airbyte.airbyte-config-persistence-0.39.1-alpha.jar:?]
	at io.airbyte.config.persistence.SecretsRepositoryWriter.splitSecretConfig(SecretsRepositoryWriter.java:193) ~[io.airbyte.airbyte-config-persistence-0.39.1-alpha.jar:?]
	at io.airbyte.config.persistence.SecretsRepositoryWriter.statefulSplitEphemeralSecrets(SecretsRepositoryWriter.java:185) ~[io.airbyte.airbyte-config-persistence-0.39.1-alpha.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:186) ~[io.airbyte-airbyte-server-0.39.1-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceCheckConnection$77(ConfigurationApi.java:753) ~[io.airbyte-airbyte-server-0.39.1-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:874) ~[io.airbyte-airbyte-server-0.39.1-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.executeSourceCheckConnection(ConfigurationApi.java:753) ~[io.airbyte-airbyte-server-0.39.1-alpha.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:833) [?:?]
2022-05-25 18:23:44 ERROR i.a.s.RequestLogger(filter):110 - REQ 192.168.80.6 POST 500 /api/v1/scheduler/sources/check_connection - {"connectionConfiguration":"REDACTED","sourceDefinitionId":"fb247691-cd6d-4b08-88d9-0456336599f7"}

harshith · May 26, 2022, 5:26am

Can you share complete logs from server and scheduler.

You can dump from docker container logs or
You can go to settings → configurations → download logs

aarsco · May 31, 2022, 10:41pm

Sorry, I missed the notification for you response. here are the logs.
scheduler_logs.log (33.5 KB)
server_logs.log (211.6 KB)

harshith · June 1, 2022, 4:32am

Hey can you share the response for the discover command? Also can upgrade to 0.39.6?

aarsco · June 2, 2022, 10:41pm

I upgraded to 0.39.8, no luck.

since I can’t get the source to configure in airbyte itself, here’s the output of python main.py discover --config secrets/config.json.

{"type": "LOG", "log": {"level": "INFO", "message": "v2 token retrieved"}}
{"type": "CATALOG", "catalog": {"streams": [{"name": "studies", "json_schema": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": {"features": {"type": "object"}, "instanceId": {"type": "integer"}, "isApiAccessAllowed": {"type": "boolean"}, "isEPro": {"type": "boolean"}, "region": {"type": "string"}, "studyId": {"type": "string"}, "title": {"type": "string"}}}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["studyId"]]}, {"name": "queries", "json_schema": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": {"GlobalQueryId": {"type": "string"}, "Study": {"type": "string"}, "Site": {"type": "string"}, "Group": {"type": "string"}, "MedrioID": {"type": "string"}, "SubjectID": {"type": "string"}, "Visit": {"type": "string"}, "VisitRepeatLabel": {"type": ["null", "string"]}, "Form": {"type": "string"}, "FormRepeatLabel": {"type": ["null", "string"]}, "VariableName": {"type": "string"}, "VariableText": {"type": "string"}, "RowNumber": {"type": "string"}, "QueryId": {"type": "string"}, "QueryType": {"type": "string"}, "QueryDesignation": {"type": ["null", "string"]}, "QueryStatus": {"type": "string"}, "QueryDescription": {"type": "string"}, "LastUpdatedTimestamp": {"type": "string", "format": "date-time"}, "OpenTimestamp": {"type": "string", "format": "date-time"}, "OpenedBy": {"type": "string"}, "CloseTimestamp": {"type": "string", "format": "date-time"}, "ClosedBy": {"type": "string"}, "ReasonForClosing": {"type": ["null", "string"]}, "ResponseCount": {"type": "string"}, "CustomerID": {"type": "string"}, "GlobalStudyId": {"type": "string"}}}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["LastUpdatedTimestamp"], "source_defined_primary_key": [["GlobalQueryId"]]}, {"name": "approval_event", "json_schema": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": {"GlobalApprovalEvent_Id": {"type": "string", "format": "uuid"}, "StudyName": {"type": "string"}, "Site": {"type": "string"}, "Group": {"type": "string"}, "MedrioId": {"type": "number"}, "SubjectId": {"type": "string"}, "Visit": {"type": "string"}, "Form": {"type": "string"}, "ApprovalStepName": {"type": "string"}, "StepOrder": {"type": "number"}, "ApprovalStatus": {"type": "string"}, "TimeStamp": {"type": ["null", "string"], "format": "date-time"}, "ApprovalStepCompletion": {"type": "string"}, "Required_Variables_Approved": {"type": "string"}, "IsLatest": {"type": "boolean"}, "UserMessage": {"type": "string"}, "FirstName": {"type": "string"}, "LastName": {"type": "string"}, "UserName": {"type": "string"}, "CustomerID": {"type": "number"}, "GlobalStudyId": {"type": "string", "format": "uuid"}}}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["GlobalApprovalEvent_Id"]]}, {"name": "form_status", "json_schema": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": {"GlobalCollectionPtID": {"type": "string", "format": "uuid"}, "FormLabel": {"type": "string"}, "FormCompletionStatus": {"type": "string"}, "DDEStatus": {"type": "integer"}, "FormDdeStatus": {"type": "string"}, "FormHasDataEntered": {"type": "string"}, "FormFirstSaveEntryDate": {"type": ["null", "string"], "format": "date-time"}, "FormFirstSaveUsername": {"type": "string"}, "FormLastUpdateEntryDate": {"type": ["null", "string"], "format": "date-time"}, "FormLastUpdateUsername": {"type": "string"}, "FormIsLocked": {"type": "string"}, "FormLockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "FormUnlockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "FormDoubleEntryDate": {"type": ["null", "string"], "format": "date-time"}, "FormIsDoubleEntered": {"type": "number"}, "FormDoubleEntryUsername": {"type": "string"}, "TotalQueries": {"type": "number"}, "OpenQueries": {"type": "number"}, "ClosedQueries": {"type": "number"}, "RespondedQueries": {"type": "number"}, "SiteName": {"type": "string"}, "SubjectId": {"type": "string"}, "SubjectMedrioId": {"type": "number"}, "SubjectStatus": {"type": "string"}, "SubjectIsLocked": {"type": "string"}, "SubjectLockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "SubjectUnlockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "SubjectVisitIsLocked": {"type": "string"}, "SubjectVisitLockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "SubjectVisitUnlockTimestamp": {"type": ["null", "string"], "format": "date-time"}, "VisitLabel": {"type": "string"}, "GroupName": {"type": "string"}, "CustomerID": {"type": "number"}, "GlobalStudyId": {"type": "string", "format": "uuid"}}}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["GlobalCollectionPtID"]]}]}}```

aarsco · June 3, 2022, 9:42pm

looks like my issue might be related UI "Unknown error occured" after upgrade · Issue #13015 · airbytehq/airbyte · GitHub

harshith · June 6, 2022, 11:56am

Can you run docker run --rm -v $(pwd)/secrets:/secrets airbyte/custom-source-name check --config secrets/config.jsonand see the response

marcosmarxm · July 13, 2022, 12:00am

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

Topic		Replies	Views
500 Error on 'Finish & Sync' with custom source connector built with low-code builder UI Connector Development connector , question , custom-source-connector , low-code-builder , 500-error	13	83	September 12, 2024
Error when changing connector's version Connector Questions & Issues connectors	5	1095	August 16, 2022
Custom connector not working anymore after updating Airbyte version Connector Questions & Issues connectors	2	412	December 23, 2022
Cannot connect to Snowflake via OAuth, 'Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null' Connector Questions & Issues source-snowflake , destination-snowflake , getting-started , connectors , oauth	6	411	July 14, 2022
Destination S3 Glue - java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.get(String)" Connector Questions & Issues destination-s3 , connectors	5	2321	April 4, 2023

Uncaught exception when configuring custom source connector

Related topics