- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Debian 11
- Memory / Disk: 8GB / 80GB
- Deployment: Docker
- Airbyte Version: 0.40.26
- Source name/version: Salesforce 1.0.27
- Destination name/version: BigQuery 1.2.9
- Step: The issue is happening during the schema fetching
- Description: Salesforce connector throws and error when fetching the schema:
Failed to fetch schema. Please try again
Error: non-json response
For the /discover_schema request. There’s no issues, when I’m setting the same connection on my laptop (Ubuntu, Airbyte and Salesforce connector versions are the same). Can’t really find any clues in the server log. GCP configuration: VM instance with Load balancer and IAP proxy.
I’m thinking about some blocked port or proxy issue, but it’s hard to tell, since we’ve already set up the Hubspot connection without such problem.
server-logs (1).txt (23.0 KB)
Found this in the logs:
airbyte-server | 2022-12-22 12:54:54 INFO i.a.s.RequestLogger(filter):112 - REQ 192.168.0.5 POST 200 /api/v1/sources/discover_schema - {"sourceId":"eb207a5d-c37c-484d-a59e-293fde025113","disable_cache":true}
airbyte-server | Dec 22, 2022 12:54:54 PM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
airbyte-server | SEVERE: An I/O error has occurred while writing a response message entity to the container output stream.
airbyte-server | org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
airbyte-server | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
airbyte-server | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
airbyte-server | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:638)
airbyte-server | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:371)
airbyte-server | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:361)
airbyte-server | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
airbyte-server | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
airbyte-server | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
airbyte-server | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
airbyte-server | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
airbyte-server | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
airbyte-server | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
airbyte-server | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
airbyte-server | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
airbyte-server | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
airbyte-server | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
airbyte-server | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
airbyte-server | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
airbyte-server | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
airbyte-server | at org.eclipse.jetty.server.Server.handle(Server.java:501)
airbyte-server | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
airbyte-server | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
airbyte-server | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
airbyte-server | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
airbyte-server | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
airbyte-server | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
airbyte-server | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
airbyte-server | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
airbyte-server | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
airbyte-server | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
airbyte-server | at java.base/java.lang.Thread.run(Thread.java:1589)
airbyte-server | Caused by: org.eclipse.jetty.io.EofException
airbyte-server | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
airbyte-server | at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
airbyte-server | at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
airbyte-server | at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
airbyte-server | at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)
airbyte-server | at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
airbyte-server | at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
airbyte-server | at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)
airbyte-server | at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)
airbyte-server | at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)
airbyte-server | at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)
airbyte-server | at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
airbyte-server | at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:833)
airbyte-server | at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:301)
airbyte-server | at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:200)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)
airbyte-server | at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2171)
airbyte-server | at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeFieldName(UTF8JsonGenerator.java:244)
airbyte-server | at com.fasterxml.jackson.databind.node.ObjectNode.serialize(ObjectNode.java:327)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:39)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:20)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
airbyte-server | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
airbyte-server | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
airbyte-server | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
airbyte-server | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
airbyte-server | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
airbyte-server | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
airbyte-server | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
airbyte-server | at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1518)
airbyte-server | at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1007)
airbyte-server | at org.glassfish.jersey.jackson.internal.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:647)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
airbyte-server | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
airbyte-server | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
airbyte-server | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
airbyte-server | ... 45 more
airbyte-server | Caused by: java.io.IOException: Broken pipe
airbyte-server | at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
airbyte-server | at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66)
airbyte-server | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:226)
airbyte-server | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:157)
airbyte-server | at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:574)
airbyte-server | at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:642)
airbyte-server | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)
airbyte-server | ... 91 more
airbyte-server |
Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:
- It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
- Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
- Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
- We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.
Thank you for your time and attention.
Best,
The Community Assistance Team
1 Like
Hey @valentine, I’m looking into this and found someone having a similar problem. In their case though it was due to the connector image architecture.
https://discuss.airbyte.io/t/error-with-api-v1-source-definitions-create-when-running-on-gcp/1271
I think this has to be related to the GCP setup since the connection is working correctly on your local machine. I’m looking into what might cause issues like this on GCP, hope to have some ideas for you next week!
1 Like
Hi @natalyjazzviolin, I’ve done some additional debugging and it revealed a few aspects. Maybe, it will help.
- The original server 502 error was caused by the GCP load balancer backend service timeout settings. It was set 30 sec, thus the connection was dropped after 30 sec. After changing it to 560 sec the problem persisted, it’s just takes 560 sec now to timeout anyway.
- New errors popping up in the log indicating some problems while connecting to the SF API endpoints.
HTTPSConnectionPool(host='jaxxinc-dev-ed.my.salesforce.com', port=443): Max retries exceeded with url:
Airbyte instance has a full access to the public internet. Could it be something on the Salesforce side?
Ok, I thinks this issue is resolved. The latest connection issues are related to sandbox account, so it works now with production one.
To conclude, the main problem was related to GCP load balancer backend service settings timeout.
Thanks you so much for all the details and for following up with the solution! This is very helpful and I’ll definitely refer future users to this post. Happy holidays!