DNS issue? - Unable to resolve host airbyte-temporal

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Rocky Linux 8.5
  • Memory / Disk: you can use something like 4Gb / 250GB
  • Deployment: Docker (podman)
  • Airbyte Version: 0.40.29
  • Source name/version: N/A
  • Destination name/version: N/A
  • Step: airbyte-server logging “UNAVAILABLE: Unable to resolve host airbyte-temporal”

When trying to get airbyte up and running (using podman-compose), it looks like there is an issue resolving the names of other containers.

The airbyte-server logs are reporting these

WARNING: [Channel<7>: (airbyte-temporal:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal
        at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
        at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
        at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
        at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.UnknownHostException: airbyte-temporal
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:952)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1658)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1524)
        at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
        at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
        ... 6 more
}

airbyte-temporal logs are:

Waiting for PostgreSQL to startup.
Waiting for PostgreSQL to startup.
nc: bad address 'db'

airbyte-proxy:

2023/02/13 14:34:37 [emerg] 11#11: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
nginx: [emerg] host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
Adding password for user airbyte
nginx version: nginx/1.23.3
2023/02/13 14:34:37 [emerg] 11#11: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
nginx: [emerg] host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
Adding password for user airbyte
nginx version: nginx/1.23.3
2023/02/13 14:34:38 [emerg] 11#11: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
nginx: [emerg] host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
Adding password for user airbyte
nginx version: nginx/1.23.3

airbyte-bootloader:

2023-02-13 14:24:07 ESC[1;31mERRORESC[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
        at org.jooq_3.13.4.POSTGRES.debug(Unknown Source) ~[?:?]
        at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:86) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:647) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:334) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:502) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:471) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:485) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.AbstractResultQuery.fetchOne(AbstractResultQuery.java:654) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.SelectImpl.fetchOne(SelectImpl.java:2884) ~[jooq-3.13.4.jar:?]
        at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4372) ~[jooq-3.13.4.jar:?]
        at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
        at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
        at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
        at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
        at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
        at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:199) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
        at io.airbyte.bootloader.Bootloader.load(Bootloader.java:101) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
        at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30010ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?]
        at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83) ~[jooq-3.13.4.jar:?]
        ... 16 more
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:331) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.0.jar:42.5.0]
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.net.UnknownHostException: db
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:560) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.net.Socket.connect(Socket.java:666) ~[?:?]
        at org.postgresql.core.PGStream.createSocket(PGStream.java:241) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:109) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.0.jar:42.5.0]
        at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.0.jar:42.5.0]
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?]
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Any ideas on what I can do to resolve this? I’ve tried 0.40.29 - 0.40.30 with no change and the exact same issues.

2

I think I found the issue. Will post results here for anyone having a similar problem once I confirm.

3

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

4

I was able to resolve the issue I had with name resolution by following the steps in this article. Basically - podman wasn’t using the newer netavark network stack.

However, I’m still not able to get into the airbyte web interface and airbyte-webapp logs are showing this:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/02/13 19:12:59 [emerg] 1#1: the closing bracket in "API_URL" variable is missing in /etc/nginx/conf.d/default.conf:29
nginx: [emerg] the closing bracket in "API_URL" variable is missing in /etc/nginx/conf.d/default.conf:29

Edit: attaching logs. Note that the “API_URL” line doesn’t appear in the log file for some reason, I may not have redirected the output to the file with the correct syntax.

airbyte-server.log (331.5 KB)

5

airbyte-worker.log (3.4 MB)
airbyte-webapp.log (269.7 KB)

6

Hey Drew, sorry for the delay with this. Are you still facing this problem? Try upgrading to v41.0 (long awaited).

7

hey can you give an example of what you did, or share the config file for how you adjusted it? im breaking my head over this and am on podman also

8

Hi, I ended up uninstalling podman and installing docker.