Destination S3 Glue - java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.get(String)"

Connector Questions & Issues
,
1

  • Is this your first time deploying Airbyte?: No

  • OS Version / Instance: AMI on AWS, MacOS

  • Deployment: Docker compose?

  • Source name/version: Braze

  • Destination name/version: S3-Glue

  • Step: The issue is happening during sync.

  • Description: Both the source and destination have been tested and passed Airbytes testing as well. Upon setting up the connection i get prompted this error:

Failure Origin: destination, Message: Something went wrong in the connector. See the logs for more details.

Here are the last lines of the error log:

,retryable=<null>,timestamp=1679302567978]],commitStateAsap=false]
2023-03-20 08:56:07 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$4):264 - Sync summary length: 11826
2023-03-20 08:56:07 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):307 - Stopping temporal heartbeating...
2023-03-20 08:56:08 INFO i.a.p.j.e.LoggingJobErrorReportingClient(reportJobFailureReason):26 - Report Job Error -> workspaceId: 0998da37-2b21-4082-83a7-7c01af7920a1, dockerImage: airbyte/destination-s3-glue:0.1.2, failureReason: io.airbyte.config.FailureReason@775a3c17[failureOrigin=destination,failureType=system_error,internalMessage=java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.get(String)" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null,externalMessage=Something went wrong in the connector. See the logs for more details.,metadata=io.airbyte.config.Metadata@3faeba92[additionalProperties={attemptNumber=2, jobId=18, connector_command=write, from_trace_message=true}],stacktrace=java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.get(String)" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null
	at io.airbyte.integrations.destination.s3_glue.GlueOperations.transformSchemaRecursive(GlueOperations.java:138)
	at io.airbyte.integrations.destination.s3_glue.GlueOperations.lambda$transformSchema$0(GlueOperations.java:116)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.airbyte.integrations.destination.s3_glue.GlueOperations.transformSchema(GlueOperations.java:117)
	at io.airbyte.integrations.destination.s3_glue.GlueOperations.upsertTable(GlueOperations.java:87)
	at io.airbyte.integrations.destination.s3_glue.S3GlueConsumerFactory.lambda$onCloseFunction$3(S3GlueConsumerFactory.java:170)
	at io.airbyte.integrations.destination.buffered_stream_consumer.OnCloseFunction.accept(OnCloseFunction.java:9)

Please let me know if you need further logs or explanation.

3

Thanks @zevex for the post,

It seems like the internal error is:

java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.get(String)" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null

Which implies that there is some function expecting a JSON value and it’s getting null instead. Any chance you could post the full logs? Perhaps there is a stream from Braze that is returning an empty JSON object which is causing this to crash.

4

Hi @sajarin

Thanks for looking inot this.

I currently cant spin the environment as im also affected by this issue: Cannot reach server. The server may still be starting up - #3 by marcosmarxm

Will update this thread once i get it up and running again.

Best,

5

Hi @zevex,

I am the original requester of this connector and worked with the author on its implementation, so it’s great to hear that someone else is able to benefit from that work.

The error that you are experiencing is happening when the destination connector is trying to process columns of an array type. It assumes that the items property of the array type information in the JSON schema is required, but it is actually optional. I’ve actually got a PR up right now to fix this error which should be merging shortly. Once destination-s3-glue: Fix invalid struct schema syntax by blarghmatey · Pull Request #23950 · airbytehq/airbyte · GitHub merges it should resolve your current issue.

Regards,
Tobias Macey

6

Hi @blarghmatey

Sorry the late reply, i was on vacation.

Thanks for the info. Could you confirm whether it has been merged yet? I can only see its closed.

7

Yes, the change has been merged and released in version 0.1.4