Airbyte using fleet workload identity overwrites GOOGLE_APPLICATION_CREDENTIALS inside connector

Connector Questions & Issues
1

Hello while trying to add destination with bigquery I cannot properly enter credentials json for service account

Here is the link with my solution to use fleet workload identity with helm release below you can see the image with my drawbacks.

This situation kept me thinking if it’s possible to use external account with small rewrite inside connector?

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: 0.32.4
  • Memory / Disk: e2-standard-2
  • Deployment: k8s
  • Airbyte Version: 0.32.4
  • Destination name/version: airbyte/destination-bigquery 1.1.11
2

image
image960×1236 108 KB

And here is example of using my deployment and trying to add source snowflake

3

Hey could you share the logs from the server and the logs while adding the source?

4

Startup logs:
startup.log (23.0 KB)

Test 1 destination bigquery
failure1.log (32.7 KB)

Test 2 source snowflake
failure2.log (12.9 KB)

Error message in both examples in UI

The connection tests failed.
Internal Server Error: java.io.IOException: Error reading credentials from stream, 'type' value 'external_account' not recognized. Expecting 'service_account'.

Response from request
http://34.149.218.216/api/v1/scheduler/sources/check_connection

{"message":"Internal Server Error: java.io.IOException: Error reading credentials from stream, 'type' value 'external_account' not recognized. Expecting 'service_account'.","exceptionClassName":"java.lang.RuntimeException","exceptionStack":["java.lang.RuntimeException: java.io.IOException: Error reading credentials from stream, 'type' value 'external_account' not recognized. Expecting 'service_account'.","\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:43)","\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:21)","\tat io.airbyte.config.helpers.GcsLogs.getOrCreateGcsClient(GcsLogs.java:125)","\tat io.airbyte.config.helpers.GcsLogs.tailCloudLog(GcsLogs.java:72)","\tat io.airbyte.config.helpers.LogClientSingleton.getJobLogFile(LogClientSingleton.java:113)","\tat io.airbyte.server.converters.JobConverter.getLogRead(JobConverter.java:205)","\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:225)","\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:212)","\tat io.airbyte.server.handlers.SchedulerHandler.reportConnectionStatus(SchedulerHandler.java:333)","\tat io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:150)","\tat io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceCheckConnection$77(ConfigurationApi.java:742)","\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:873)","\tat io.airbyte.server.apis.ConfigurationApi.executeSourceCheckConnection(ConfigurationApi.java:742)","\tat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)","\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)","\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)","\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)","Caused by: java.io.IOException: Error reading credentials from stream, 'type' value 'external_account' not recognized. Expecting 'service_account'.","\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:551)","\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:520)","\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:40)","\t... 62 more"],"rootCauseExceptionClassName":"java.lang.Class","rootCauseExceptionStack":["java.io.IOException: Error reading credentials from stream, 'type' value 'external_account' not recognized. Expecting 'service_account'.","\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:551)","\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:520)","\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:40)","\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:21)","\tat io.airbyte.config.helpers.GcsLogs.getOrCreateGcsClient(GcsLogs.java:125)","\tat io.airbyte.config.helpers.GcsLogs.tailCloudLog(GcsLogs.java:72)","\tat io.airbyte.config.helpers.LogClientSingleton.getJobLogFile(LogClientSingleton.java:113)","\tat io.airbyte.server.converters.JobConverter.getLogRead(JobConverter.java:205)","\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:225)","\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:212)","\tat io.airbyte.server.handlers.SchedulerHandler.reportConnectionStatus(SchedulerHandler.java:333)","\tat io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:150)","\tat io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceCheckConnection$77(ConfigurationApi.java:742)","\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:873)","\tat io.airbyte.server.apis.ConfigurationApi.executeSourceCheckConnection(ConfigurationApi.java:742)","\tat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)","\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)","\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)","\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)"]}
5

Gotcha. Thank you. We can discuss on the github issue. Also did you get a chance to try the suggestion over the github issue?

6

What do you mean by suggestion over the github issue ?

Should I create github issue and link this discuss there?

7

There is this github issue right https://github.com/airbytehq/airbyte/issues/13081?

8

Yes, kinda because this issue is about how to use GKE workload identity and in my messages I posted my solution for this using fleet workload identity but this deployment comes with bug where I cannot add any connector due to this message from screen.

If it’s okay we can merge these two issues.

9

Gotcha I don’t think there is a direct solution but one thing would be is to make a custom connector and ensure that doesn’t pick from env GOOGLE_APPLICATION_CREDENTIALS.