Cannot connect to Snowflake via OAuth, 'Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null'

Connector Questions & Issues
, , , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: MacOS
  • Memory / Disk: you can use something like 32Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.36.9
  • Source name/version: Snowflake
  • Destination name/version: Snowflake
  • Step: Cannot create OAuth source or destination for Snowflake
  • Description: I have created OAuth 2.0 credentials in my Snowflake client, and am able to successfully connect via SnowSQL.

However, when I try to connect via Airbyte, I get the error:

Could not connect with provided configuration. Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null

Why is this? This is the open source version of Airbyte running locally via Docker.

I couldn’t get the server logs to download, but here is what was in the webapp terminal:

airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/b63cfce2-528b-4625-9fa5-0432d8d3ad54/0/logs.log
airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.36.9-alpha
airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-snowflake:0.4.25 exists...
airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-snowflake:0.4.25 was found locally.
airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.w.p.DockerProcessFactory(create):107 - Creating docker job ID: b63cfce2-528b-4625-9fa5-0432d8d3ad54
airbyte-worker      | 2022-05-05 19:58:10 INFO i.a.w.p.DockerProcessFactory(create):162 - Preparing command: docker run --rm --init -i -w /data/b63cfce2-528b-4625-9fa5-0432d8d3ad54/0 --log-driver none --name destination-snowflake-sync-b63cfce2-528b-4625-9fa5-0432d8d3ad54-0-axbmw --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_CONNECTOR_IMAGE=airbyte/destination-snowflake:0.4.25 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.9-alpha -e WORKER_JOB_ID=b63cfce2-528b-4625-9fa5-0432d8d3ad54 airbyte/destination-snowflake:0.4.25 check --config source_config.json
airbyte-worker      | 2022-05-05 19:58:11 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Class path contains multiple SLF4J bindings.
airbyte-worker      | 2022-05-05 19:58:11 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
airbyte-worker      | 2022-05-05 19:58:11 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
airbyte-worker      | 2022-05-05 19:58:11 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
airbyte-worker      | 2022-05-05 19:58:11 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json}
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 INFO i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 INFO i.a.i.b.IntegrationRunner(runInternal):122 - Command: CHECK
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 INFO i.a.i.d.j.c.SwitchingDestination(check):55 - Using destination type: INTERNAL_STAGING
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-05-05 19:58:12 ERROR i.a.i.d.s.SnowflakeInternalStagingDestination(check):52 - Exception while checking connection:
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.snowflake.SnowflakeDatabase.createDataSource(SnowflakeDatabase.java:71) ~[io.airbyte.airbyte-integrations.connectors-destination-snowflake-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.snowflake.SnowflakeDatabase.getDatabase(SnowflakeDatabase.java:173) ~[io.airbyte.airbyte-integrations.connectors-destination-snowflake-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.snowflake.SnowflakeInternalStagingDestination.getDatabase(SnowflakeInternalStagingDestination.java:74) ~[io.airbyte.airbyte-integrations.connectors-destination-snowflake-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.snowflake.SnowflakeInternalStagingDestination.check(SnowflakeInternalStagingDestination.java:44) [io.airbyte.airbyte-integrations.connectors-destination-snowflake-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.jdbc.copy.SwitchingDestination.check(SwitchingDestination.java:56) [io.airbyte.airbyte-integrations.connectors-destination-jdbc-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:138) [io.airbyte.airbyte-integrations.bases-base-java-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:105) [io.airbyte.airbyte-integrations.bases-base-java-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:12 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.snowflake.SnowflakeDestination.main(SnowflakeDestination.java:30) [io.airbyte.airbyte-integrations.connectors-destination-snowflake-0.36.7-alpha.jar:?]
airbyte-worker      | 2022-05-05 19:58:13 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
airbyte-server      | 2022-05-05 19:58:13 INFO i.a.s.RequestLogger(filter):112 - REQ 172.19.0.3 POST 200 /api/v1/scheduler/destinations/check_connection - {"connectionConfiguration":"REDACTED","destinationDefinitionId":"424892c4-daac-4491-b35d-c6688ba547ba"}
airbyte-webapp      | 172.19.0.1 - - [05/May/2022:19:58:13 +0000] "POST /api/v1/scheduler/destinations/check_connection HTTP/1.1" 200 8439 "http://localhost:8000/workspaces/e3b4d4bd-3c0e-408b-8552-56e3e2c49dfb/destination/new-destination" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.147 Safari/537.36" "-"

SCR-20220505-kk6
SCR-20220505-kk6582×929 68.2 KB

SCR-20220505-k14
SCR-20220505-k141181×145 10.6 KB

2

Any ideas on this? This is blocking me from demo’ing this product for my team, and I’m really excited to try it :frowning:

3

The client_id and client_secret are required when using OAuth method.
I strongly recommend to you read the code from https://github.com/airbytehq/airbyte/blob/73c7fad7fce952a8c3ba827ca858e4280bd846f3/airbyte-integrations/connectors/destination-snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDatabase.java#L61-L75 to understand how Airbyte uses those credentials.
My suggestion for your PoC is to use user/password method. Looks the OAuth method need to update the refresh token and other credentials frequently and if you don’t have this configured in your instance the connector won’t work

4

@marcosmarxm Thanks for responding! If the client_id and secret_id are required for Airbyte’s OAUth 2.0 implementation, shouldn’t they have an “*” in the UI? They aren’t required when using the interface. But I will try this and report back!

Re: the need to use the user/password method, are you sure that’s how the OAuth implementation works? I was under the impression that OAuth issues a Refresh Token that never expires, and you can use that Refresh token to get a new Access token once the current Access token expires.

In the code you linked there above (that’s a great resource, I think maybe the docs on using this method could be a bit more fleshed out?) is a getAccessTokenUsingRefreshToken method. I would think that this method handles getting a new Acces Token via the Refresh token? Although it calls a REFRESH_TOKEN_URL variable, and I don’t quite understand how that is set, given the fields asked for in the UI.

I’ll try again with the client_id and client_secret (I tried with them once at got the same error, but maybe my syntax was wrong), and report back.

My company is very interested in moving our workload to Airbyte (including the Cloud service if it has additional features that clear blockers like this issue), and I think a lot of business use Snowflake but have stricter security methods than Username/Passoword (we use RSA keys, for example). I think more auth options and clearer docs for Snowflake would really be to Airbyte’s benefit!

Thanks for your help,
-Bryan

5

@marcosmarxm Thanks for responding! If the client_id and secret_id are required for Airbyte’s OAUth 2.0 implementation, shouldn’t they have an “*” in the UI? They aren’t required when using the interface. But I will try this and report back!

Yes, I’ll raise an issue.

I’m suggesting to you use the Username/Password because is simpler than OAuth for your PoC.

1602×530 36 KB

Thanks for the feedback. We’re always trying to improve our docs.

6

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.