Connector sync scheduling stopped for no reason

Hello,

I don’t understand why the sync scheduling stopped for a connector while sync is enabled
Do you know why ?

Example with the Salesloft connector below


Sorry the long delay in replying Yanni. Harshith Mullapudi can you take a look Monday on this issue? Try to identify how the scheduler get the value from the database and launch the next sync please

Sure will take a look. Also is it possible to share Scheduler logs here

What’s is the best way get the scheduler logs ?
I can’t download files from the interface :sweat_smile:

I will get the docker service logs on the vm otherwise

ok @harshith i found part of the scheduler logs (via docker logs airbyte-scheduler)

I can see an error

2022-04-22 07:36:36 ERROR i.a.s.a.JobSubmitter(run):81 - Job Submitter Error
org.jooq.exception.DataAccessException: Error getting connection from data source org.apache.commons.dbcp2.BasicDataSource@5c37accd
	at org.jooq_3.13.4.POSTGRES.debug(Unknown Source) ~[?:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:86) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:647) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:334) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:354) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetch(DefaultDSLContext.java:892) ~[jooq-3.13.4.jar:?]
	at io.airbyte.scheduler.persistence.DefaultJobPersistence.lambda$getNextJob$21(DefaultJobPersistence.java:410) ~[io.airbyte.airbyte-scheduler-persistence-0.35.51-alpha.jar:?]
	at io.airbyte.db.Database.query(Database.java:27) ~[io.airbyte.airbyte-db-lib-0.35.51-alpha.jar:?]
	at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:23) ~[io.airbyte.airbyte-db-lib-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.persistence.DefaultJobPersistence.getNextJob(DefaultJobPersistence.java:409) ~[io.airbyte.airbyte-scheduler-persistence-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.app.JobSubmitter.run(JobSubmitter.java:74) ~[io.airbyte.airbyte-scheduler-app-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.app.SchedulerApp.lambda$start$1(SchedulerApp.java:167) ~[io.airbyte.airbyte-scheduler-app-0.35.51-alpha.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:313) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.makeConnection(Driver.java:466) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.connect(Driver.java:265) ~[postgresql-42.3.1.jar:42.3.1]
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83) ~[jooq-3.13.4.jar:?]
	... 16 more
Caused by: java.net.SocketTimeoutException: Connect timed out
	at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
	at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
	at java.net.Socket.connect(Socket.java:633) ~[?:?]
	at org.postgresql.core.PGStream.createSocket(PGStream.java:238) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:100) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:215) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.makeConnection(Driver.java:466) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.connect(Driver.java:265) ~[postgresql-42.3.1.jar:42.3.1]
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83) ~[jooq-3.13.4.jar:?]
	... 16 more
2022-04-22 07:36:49 ERROR i.a.s.a.JobSubmitter(run):81 - Job Submitter Error
org.jooq.exception.DataAccessException: Error getting connection from data source org.apache.commons.dbcp2.BasicDataSource@5c37accd
	at org.jooq_3.13.4.POSTGRES.debug(Unknown Source) ~[?:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:86) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:647) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:334) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:354) ~[jooq-3.13.4.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetch(DefaultDSLContext.java:892) ~[jooq-3.13.4.jar:?]
	at io.airbyte.scheduler.persistence.DefaultJobPersistence.lambda$getNextJob$21(DefaultJobPersistence.java:410) ~[io.airbyte.airbyte-scheduler-persistence-0.35.51-alpha.jar:?]
	at io.airbyte.db.Database.query(Database.java:27) ~[io.airbyte.airbyte-db-lib-0.35.51-alpha.jar:?]
	at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:23) ~[io.airbyte.airbyte-db-lib-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.persistence.DefaultJobPersistence.getNextJob(DefaultJobPersistence.java:409) ~[io.airbyte.airbyte-scheduler-persistence-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.app.JobSubmitter.run(JobSubmitter.java:74) ~[io.airbyte.airbyte-scheduler-app-0.35.51-alpha.jar:?]
	at io.airbyte.scheduler.app.SchedulerApp.lambda$start$1(SchedulerApp.java:167) ~[io.airbyte.airbyte-scheduler-app-0.35.51-alpha.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.postgresql.util.PSQLException: Connection to 35.187.19.193:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:301) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.makeConnection(Driver.java:466) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.connect(Driver.java:265) ~[postgresql-42.3.1.jar:42.3.1]
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83) ~[jooq-3.13.4.jar:?]
	... 16 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549) ~[?:?]
	at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
	at java.net.Socket.connect(Socket.java:633) ~[?:?]
	at org.postgresql.core.PGStream.createSocket(PGStream.java:238) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:100) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:215) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.makeConnection(Driver.java:466) ~[postgresql-42.3.1.jar:42.3.1]
	at org.postgresql.Driver.connect(Driver.java:265) ~[postgresql-42.3.1.jar:42.3.1]
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753) ~[commons-dbcp2-2.7.0.jar:2.7.0]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83) ~[jooq-3.13.4.jar:?]

scheduler.log (122.8 KB)


I forgot to mention but i deployed airflow on a compte engine + cloud sql

  • Machine type: e2-standard-2

  • CloudSQL : PostgreSQL 13 | vCPUs = 1 | Memory= 1.7 GB | SSD storage = 10 GB

Hey I think it was not able to connect to the database do you mind restarting the Airbyte and also check parallelly on why the database is not reachable

1 Like

now it’s working now because i click on sync now
but i’m wondering why it stopped completely and don’t retry several times after it got the error :sweat_smile:

I feel it’s because the scheduler was not able to reach the database it was not having any information

ok fair enough
I was really expecting a retry on this one but ok good to know