Summary
The user is facing an issue with missing records in Snowflake destination while using API Connector Builder with a paginated stream. They have configured PageNumber and PageSize, but only about half of the expected records are being synced. The user is also looking to append the values used in outgoing paginated requests to the response body for debugging purposes.
Question
Hi all, I’m troubleshooting a situation using the API Connector Builder where I’ve configured a paginated stream which uses both PageNumber and PageSize. I’ve set the page size to 1000 records and am starting with page 0. The sync runs and completes “successfully” without errors. The destination is Snowflake, and when I check the resulting table and I only find about half records that I’m expecting (~300k of the ~700k). I am using the Full Sync | Override sync mode.
I’ve verified the GET params and number of pages expected using Postman. There are warnings in the job log, but no errors (warnings added in reply). What am I missing here?
Also, is it possible to append the values used in the outgoing paginated requests (i.e. Page Number and Page Size values) to the response body using a transformation? This would be extremely helpful for debugging my situation where records are being silently dropped.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["api-connector-builder", "paginated-stream", "snowflake-destination", "missing-records", "debugging", "transformation"]
Job Warnings:
2024-01-31 20:04:12 destination > Ideally, this situation should not happen...
2024-01-31 20:04:12 destination > Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
2024-01-31 20:04:12 destination > The main thread is: main (RUNNABLE)
2024-01-31 20:04:12 destination > Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
2024-01-31 20:04:12 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.dumpThread(IntegrationRunner.java:396)
2024-01-31 20:04:12 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.stopOrphanedThreads(IntegrationRunner.java:370)
2024-01-31 20:04:12 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:192)
2024-01-31 20:04:12 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
2024-01-31 20:04:12 destination > at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:102)
2024-01-31 20:04:12 destination > at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationRunner.main(SnowflakeDestinationRunner.java:20)
2024-01-31 20:04:12 destination > 2024-01-31 20:04:12 WARN i.a.c.i.b.IntegrationRunner(stopOrphanedThreads):378 - Active non-daemon thread: pool-10-thread-68 (RUNNABLE)
2024-01-31 20:04:12 destination > Thread stacktrace: java.base@17.0.8.1/sun.nio.ch.Net.poll(Native Method)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:186)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:290)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/java.net.Socket$SocketInputStream.read(Socket.java:966)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.RestRequest.execute(RestRequest.java:222)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.RestRequest.execute(RestRequest.java:67)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.core.HttpUtil.executeRequestInternal(HttpUtil.java:751)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.core.HttpUtil.executeRequest(HttpUtil.java:684)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.core.HttpUtil.executeGeneralRequest(HttpUtil.java:601)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.sendBatch(TelemetryClient.java:352)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.lambda$sendBatchAsync$0(TelemetryClient.java:282)
2024-01-31 20:04:12 destination > at app//net.snowflake.client.jdbc.telemetry.TelemetryClient$$Lambda$321/0x00007fd1c4434440.call(Unknown Source)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-01-31 20:04:12 destination > at java.base@17.0.8.1/java.lang.Thread.run(Thread.java:833)```