Summary
The user is facing issues with populating the credentials_json field in the Terraform connector for setting up a BigQuery destination. The error message received is ‘Internal Server Error: no JSON input found’. The user has tried different combinations to resolve the issue but without success.
Question
Hi there, I am attempting to use the Terraform connector to setup a BigQuery destination. I am populating the credentials_json field which i think is giving me issues with the output from a google_service_account_key resource.
This is a JSON service account key encoded as a string, when looking at the request to the airbyte api it looks fine, and i can reproduce the error when manually constructing the request with Postman.
The body looks like this (some fields redacted)
"configuration": {
"big_query_client_buffer_size_mb": 15,
"credentials_json": "{\n \"type\": \"service_account\",\n .... \"auth_provider_x509_cert_url\": \"<https://www.googleapis.com/oauth2/v1/certs>\",\n \"client_x509_cert_url\": \"<https://www.googleapis.com/robot/v1/metadata/x509/> \"universe_domain\": \"<http://googleapis.com|googleapis.com>\"\n}\n",
"dataset_id": "...",
"dataset_location": "europe-west2",
"destinationType": "bigquery",
"disable_type_dedupe": false,
"loading_method": {
"credential": {
"credential_type": "HMAC_KEY",
"hmac_key_access_id": "...",
"hmac_key_secret": "..."
},
"gcs_bucket_name": "...",
"gcs_bucket_path": "",
"keep_files_in_gcs-bucket": "Delete all tmp files from GCS",
"method": "GCS Staging"
},
"project_id": "...",
"transformation_priority": "batch"
},
"name": "BigQuery ... Raw",
"workspaceId": "a689fb18-e6a3-4435-8893-6634add7aa94"
}```
and the response i am getting is this
```{
"type": "<https://reference.airbyte.com/reference/errors>",
"title": "unexpected-problem",
"status": 500,
"detail": "An unexpected problem has occurred. If this is an error that needs to be addressed, please submit a pull request or github issue."
}```
In the airbyte server logs i can see a bit more detail
```airbyte-api-server | 2024-02-27 16:13:21 ERROR i.a.a.s.s.DestinationServiceImpl(createDestination):124 - Config api response error for createDestination:
airbyte-api-server | io.micronaut.http.client.exceptions.HttpClientResponseException: Internal Server Error: no JSON input found
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.makeErrorFromRequestBody(DefaultHttpClient.java:2232) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.buildResponse(DefaultHttpClient.java:2194) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.buildResponse(DefaultHttpClient.java:2117) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2092) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2153) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2117) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:49) ~[micronaut-http-client-3.10.1.jar:3.10.1]
airbyte-api-server | at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:216) ~[micronaut-http-netty-3.10.1.jar:3.10.1]
airbyte-api-server | at io.micronaut.http.netty.stream.HttpStreamsClientHandler.channelRead(HttpStreamsClientHandler.java:180) ~[micronaut-http-netty-3.10.1.jar:3.10.1]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
airbyte-api-server | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]```
I have tried various different combinations to try and resolve this, the error messages are really giving me much help and have tried searching through docs / github issues etc. to try and find what i am doing wrong here without much luck, asking here as a last resort. any help is greatly appreciated, thanks
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1709051049771799) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["terraform-connector", "bigquery-destination", "credentials-json", "internal-server-error"]
</sub>