Error when incremental sync from postgres to redshift

Truc_Nguyen · June 8, 2022, 5:22am

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: ec2 linux
Memory / Disk: 50gb
Deployment: Docker
Airbyte Version: 0.39.7-alpha
Source: Postgres (0.4.21)
Destination: Redshift (0.3.37)
Step: The issue is happening during sync during creating query for table
Description: This problem appear when I run incremental sync after the initial load from db. I seem like there is error when airbyte generate query for source table

2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(queryTableIncremental):268 - Queueing query for table: notify_logs
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(queryTableIncremental):268 - Queueing query for table: notify_requests
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(queryTableIncremental):268 - Queueing query for table: notify_tasks
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(queryTableIncremental):268 - Queueing query for table: receivers
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(queryTableIncremental):268 - Queueing query for table: senders
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$queryTableIncremental$16):273 - Preparing query for table: notify_endpoints
2022-06-08 04:32:49 e[44msourcee[0m > 2022-06-08 04:32:49 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):130 - Closing database connection pool.
2022-06-08 04:32:50 e[44msourcee[0m > 2022-06-08 04:32:50 e[32mINFOe[m c.z.h.HikariDataSource(close):350 - HikariPool-2 - Shutdown initiated...
2022-06-08 04:32:50 e[44msourcee[0m > 2022-06-08 04:32:50 e[32mINFOe[m c.z.h.HikariDataSource(close):352 - HikariPool-2 - Shutdown completed.
2022-06-08 04:32:50 e[44msourcee[0m > 2022-06-08 04:32:50 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):132 - Closed database connection pool.
2022-06-08 04:32:50 e[44msourcee[0m > 2022-06-08 04:32:50 e[1;31mERRORe[m i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 - Something went wrong in the connector. See the logs for more details.
2022-06-08 04:32:50 e[44msourcee[0m > java.lang.IllegalArgumentException: TIMESTAMP_WITH_TIMEZONE is not supported.
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:86) ~[io.airbyte.airbyte-db-lib-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:26) ~[io.airbyte.airbyte-db-lib-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$16(AbstractJdbcSource.java:281) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:62) ~[io.airbyte.airbyte-db-lib-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$17(AbstractJdbcSource.java:271) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:37) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.source.relationaldb.StateDecoratingIterator.computeNext(StateDecoratingIterator.java:48) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.source.relationaldb.StateDecoratingIterator.computeNext(StateDecoratingIterator.java:18) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:63) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$2(IntegrationRunner.java:176) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:232) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:175) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.lambda$runInternal$0(IntegrationRunner.java:156) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:54) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:38) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:156) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:107) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[44msourcee[0m > 	at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:410) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.39.5-alpha.jar:?]
2022-06-08 04:32:50 e[43mdestinatione[0m > 2022-06-08 04:32:50 e[32mINFOe[m c.z.h.p.PoolBase(getAndSetNetworkTimeout):536 - HikariPool-1 - Driver does not support get/set network timeout for connections. ([Amazon][JDBC](10220) Driver does not support this optional feature.)
2022-06-08 04:32:50 e[32mINFOe[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):347 - Source has no more messages, closing connection.

ibraraslam · June 8, 2022, 8:43am

Exact same issue for incremental sync on using timestamp with time zone as courser.

Is this your first time deploying Airbyte?: No
OS Version / Instance: ec2 linux
Memory / Disk: 50gb
Deployment: Docker
Airbyte Version: 0.39.13-alpha
Source: Postgres (0.4.21)
Destination: S3 (0.3.6)
Step: The issue has been happening since June 6. incremental sync works if the courser is changed to integer.
Description: This problem appear when I run incremental sync after the initial load from db. postgres error logs has info.

2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[33mWARNe[m i.a.i.b.IntegrationRunner(watchForOrphanThreads):241 - The main thread is exiting while children non-daemon threads from a connector are still active.
2022-06-08 07:08:55 e[44msourcee[0m > Ideally, this situation should not happen...
2022-06-08 07:08:55 e[44msourcee[0m > Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
2022-06-08 07:08:55 e[44msourcee[0m > The main thread is: main (RUNNABLE)
2022-06-08 07:08:55 e[44msourcee[0m >  Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.dumpThread(IntegrationRunner.java:276)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:245)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:175)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.lambda$runInternal$0(IntegrationRunner.java:156)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:54)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:38)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:156)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:107)
2022-06-08 07:08:55 e[44msourcee[0m >         at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:410)
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[33mWARNe[m i.a.i.b.IntegrationRunner(watchForOrphanThreads):255 - Active non-daemon thread: pool-2-thread-1 (TIMED_WAITING)
2022-06-08 07:08:55 e[44msourcee[0m >  Thread stacktrace: java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[33mWARNe[m i.a.i.b.IntegrationRunner(watchForOrphanThreads):255 - Active non-daemon thread: NioSocketAcceptor-2 (RUNNABLE)
2022-06-08 07:08:55 e[44msourcee[0m >  Thread stacktrace: java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:146)
2022-06-08 07:08:55 e[44msourcee[0m >         at app//org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:299)
2022-06-08 07:08:55 e[44msourcee[0m >         at app//org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:466)
2022-06-08 07:08:55 e[44msourcee[0m >         at app//org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2022-06-08 07:08:55 e[44msourcee[0m >         at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):130 - Closing database connection pool.
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[32mINFOe[m c.z.h.HikariDataSource(close):350 - HikariPool-2 - Shutdown initiated...
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[32mINFOe[m c.z.h.HikariDataSource(close):352 - HikariPool-2 - Shutdown completed.
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):132 - Closed database connection pool.
2022-06-08 07:08:55 e[44msourcee[0m > 2022-06-08 07:08:55 e[1;31mERRORe[m i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 - Something went wrong in the connector. See the logs for more details.
2022-06-08 07:08:55 e[44msourcee[0m > java.lang.IllegalArgumentException: TIMESTAMP_WITH_TIMEZONE is not supported.
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:86) ~[io.airbyte.airbyte-db-db-lib-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:26) ~[io.airbyte.airbyte-db-db-lib-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$16(AbstractJdbcSource.java:281) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:62) ~[io.airbyte.airbyte-db-db-lib-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$17(AbstractJdbcSource.java:271) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:37) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.source.relationaldb.StateDecoratingIterator.computeNext(StateDecoratingIterator.java:48) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.source.relationaldb.StateDecoratingIterator.computeNext(StateDecoratingIterator.java:18) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:63) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38) ~[io.airbyte-airbyte-commons-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$2(IntegrationRunner.java:176) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:232) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:175) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.lambda$runInternal$0(IntegrationRunner.java:156) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:54) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.sentry.AirbyteSentry.executeWithTracing(AirbyteSentry.java:38) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:156) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:107) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[44msourcee[0m > 	at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:410) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.39.13-alpha.jar:?]
2022-06-08 07:08:55 e[32mINFOe[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):347 - Source has no more messages, closing connection.

alafanechere · June 8, 2022, 9:34am

Hey @Truc_Nguyen,
Our connector team made recent efforts to support the TIMESTAMP_WITH_TIMEZONE type in the postgres connector (version >= 0.4.19).
Could you please recreate a new connection and retry after making sure you are indeed using 0.4.21?
I’m wondering if the problem could come from the fact your initial load was made with version <= 0.4.19 and the incremental load on 0.4.21.

ibraraslam · June 8, 2022, 10:24am

Hey @alafanechere ,
I redeployed the latest Airbyte and made sure the both source and destination are up-to-date.
But the issue persists. Could this be an invalid formate issue? From “Data Types in Records” docs, the formate for timestamp_with_timezone is 2021-11-22T01:23:45+00:00 but the default Postgres formate is “2021-07-09 04:10:29.676346 +00:00”

Is this your first time deploying Airbyte?: No
OS Version / Instance: ec2 linux
Memory / Disk: 50gb
Deployment: Docker
Airbyte Version: 0.39.13-alpha
Source: Postgres (0.4.21)
Destination: S3 (0.3.6)

alafanechere · June 8, 2022, 1:09pm

I think I nailed it down to the right location in our codebase:
The error is raised when the incremental SQL query on the source is built. The connector is not able to use a TIMESTAMP_WITH_TIMEZONE column as a cursor.
I opened this issue to escalate it to our engineering team. Please subscribe to the issue to receive updates and feel free to share your observations.

grishick · June 14, 2022, 12:08am

We have deployed a fix about an hour ago. The fix adds support for TIMESTAMP_WITH_TIMEZONE and addresses the problem with unparseable dates. For OSS deployments - use Source Postgres v 0.4.22

marcosmarxm · July 13, 2022, 12:00am

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

Topic		Replies	Views
Postgres Sync Fails After Upgrading to 0.39.8-alpha Connector Questions & Issues source-postgres	5	422	July 14, 2022
Incremental Models Failing Connector Questions & Issues destination-snowflake , data-loading	3	538	September 16, 2022
Incremental sync failing for Postgres source connector on M1 Mac Docker deployment Connector Questions & Issues source-postgres , connectors , cdc	7	1678	May 11, 2022
Redshift connection lost, job hanging Connector Questions & Issues source-redshift	3	354	July 14, 2022
Error Sync Data Google Analytic -> Redshift Connector Questions & Issues source-google-analytics , destination-redshift	6	403	July 14, 2022

Error when incremental sync from postgres to redshift

Related topics