- Is this your first time deploying Airbyte?: Yes
- OS Version / Instance: Ubuntu
- Memory / Disk: 32gb
- Deployment: Docker Swarm
- Airbyte Version: v0.40.18
- Step: Deployment over Portainer
I copied the docker-compose and .env from git and added the temporal/dynamicconfig/development.yaml
- Description: While deploying airbyte over portainer i get this erro message in airbyte-proxy:
host not found in upstream “airbyte-server:8001” in /etc/nginx/conf.d/default.conf:2
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/11/15 14:10:46 [emerg] 1#1: host not found in upstream "airbyte-server:8001" in /etc/nginx/conf.d/default.conf:2
nginx: [emerg] host not found in upstream "airbyte-server:8001" in /etc/nginx/conf.d/default.conf:2
and in airbyte_webapp
[emerg] 12#12: host not found in upstream “airbyte-webapp” in /etc/nginx/nginx.conf:15
starting nginx...
nginx version: nginx/1.23.2
2022/11/15 14:10:35 [emerg] 12#12: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:15
nginx: [emerg] host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:15
Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:
- It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
- Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
- Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
- We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.
Thank you for your time and attention.
Best,
The Community Assistance Team
Hey @aschmidt, let me look into this and get back to you! We’re experiencing a high volume of messages, so thanks for your patience in advance!
I now added the following nginx config:
upstream api-server {
server airbyte-server:8001;
}
server {
listen 80;
listen [::]:80;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
add_header Content-Security-Policy "script-src * 'unsafe-inline'; worker-src self blob:;";
location / {
root /usr/share/nginx/html;
try_files $uri $uri/ /index.html;
sub_filter </head>
'</head><script language="javascript"\>
window.TRACKING_STRATEGY = "$TRACKING_STRATEGY";
window.FULLSTORY = "$FULLSTORY";
window.AIRBYTE_VERSION = "$AIRBYTE_VERSION";
window.API_URL = "$API_URL";
window.IS_DEMO = "$IS_DEMO";
</script>';
sub_filter_once on;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location /api/ {
fastcgi_read_timeout 1h;
proxy_read_timeout 1h;
client_max_body_size 200M;
proxy_pass http://api-server/api/;
}
}
to the webapp as follow:
configs:
- source: airbyte_default.conf
target: /etc/nginx/conf.d/default.conf
this changes my error in webapp to the following:
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.d/20-envsubst-on-templates.sh: line 33: can't create /etc/nginx/conf.d/default.conf: Read-only file system
The error message in proxy stays the same:
BASIC_AUTH_USERNAME is set, requiring auth for user '""'
Adding password for user ""
starting nginx...
nginx version: nginx/1.23.2
2022/11/23 12:48:06 [emerg] 12#12: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:15
nginx: [emerg] host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:15
Should the nginx config be enough or do i need a running nginx container in my swarm?
Ok i added nginx as service in my swarm and now im back to my initial error.
This is my current Compose:
version: "3.7"
#https://github.com/compose-spec/compose-spec/blob/master/spec.md#using-extensions-as-fragments
x-logging: &default-logging
options:
max-size: "100m"
max-file: "5"
driver: json-file
services:
web:
image: nginx
configs:
- source: default.conf.template
target: /etc/nginx/templates
ports:
- "8080:80"
networks:
- airbyte_internal
# hook in case we need to add init behavior
# every root service (no depends_on) should depend on init
init:
image: airbyte/init:${VERSION}
logging: *default-logging
container_name: init
command: /bin/sh -c "./scripts/create_mount_directories.sh /local_parent ${HACK_LOCAL_ROOT_PARENT} ${LOCAL_ROOT}"
environment:
- LOCAL_ROOT=${LOCAL_ROOT}
- HACK_LOCAL_ROOT_PARENT=${HACK_LOCAL_ROOT_PARENT}
volumes:
- ${HACK_LOCAL_ROOT_PARENT}:/local_parent
restart: on-failure
bootloader:
image: airbyte/bootloader:${VERSION}
logging: *default-logging
container_name: airbyte-bootloader
environment:
- AIRBYTE_VERSION=${VERSION}
- CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
- CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
- CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_URL=${DATABASE_URL}
- DATABASE_USER=${DATABASE_USER}
- LOG_LEVEL=${LOG_LEVEL}
- RUN_DATABASE_MIGRATION_ON_STARTUP=${RUN_DATABASE_MIGRATION_ON_STARTUP}
networks:
- airbyte_internal
restart: on-failure
db:
image: airbyte/db:${VERSION}
logging: *default-logging
container_name: airbyte-db
restart: unless-stopped
environment:
- CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
- CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
- CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_URL=${DATABASE_URL}
- DATABASE_USER=${DATABASE_USER}
- POSTGRES_PASSWORD=${DATABASE_PASSWORD}
- POSTGRES_USER=${DATABASE_USER}
volumes:
- db:/var/lib/postgresql/data
networks:
- airbyte_internal
worker:
image: airbyte/worker:${VERSION}
logging: *default-logging
container_name: airbyte-worker
restart: unless-stopped
environment:
- AIRBYTE_VERSION=${VERSION}
- AUTO_DISABLE_FAILING_CONNECTIONS=${AUTO_DISABLE_FAILING_CONNECTIONS}
- CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
- CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
- CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
- CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
- CONFIG_ROOT=${CONFIG_ROOT}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_URL=${DATABASE_URL}
- DATABASE_USER=${DATABASE_USER}
- DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
- INTERNAL_API_HOST=${INTERNAL_API_HOST}
- JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
- JOB_MAIN_CONTAINER_CPU_LIMIT=${JOB_MAIN_CONTAINER_CPU_LIMIT}
- JOB_MAIN_CONTAINER_CPU_REQUEST=${JOB_MAIN_CONTAINER_CPU_REQUEST}
- JOB_MAIN_CONTAINER_MEMORY_LIMIT=${JOB_MAIN_CONTAINER_MEMORY_LIMIT}
- JOB_MAIN_CONTAINER_MEMORY_REQUEST=${JOB_MAIN_CONTAINER_MEMORY_REQUEST}
- LOCAL_DOCKER_MOUNT=${LOCAL_DOCKER_MOUNT}
- LOCAL_ROOT=${LOCAL_ROOT}
- LOG_LEVEL=${LOG_LEVEL}
- LOG_CONNECTOR_MESSAGES=${LOG_CONNECTOR_MESSAGES}
- MAX_CHECK_WORKERS=${MAX_CHECK_WORKERS}
- MAX_DISCOVER_WORKERS=${MAX_DISCOVER_WORKERS}
- MAX_SPEC_WORKERS=${MAX_SPEC_WORKERS}
- MAX_SYNC_WORKERS=${MAX_SYNC_WORKERS}
- NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=${NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT}
- NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=${NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST}
- NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT=${NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT}
- NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST=${NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST}
- SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
- SYNC_JOB_MAX_ATTEMPTS=${SYNC_JOB_MAX_ATTEMPTS}
- SYNC_JOB_MAX_TIMEOUT_DAYS=${SYNC_JOB_MAX_TIMEOUT_DAYS}
- TEMPORAL_HOST=${TEMPORAL_HOST}
- TRACKING_STRATEGY=${TRACKING_STRATEGY}
- WEBAPP_URL=${WEBAPP_URL}
- WORKSPACE_DOCKER_MOUNT=${WORKSPACE_DOCKER_MOUNT}
- WORKSPACE_ROOT=${WORKSPACE_ROOT}
- METRIC_CLIENT=${METRIC_CLIENT}
- OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}
- JOB_ERROR_REPORTING_STRATEGY=${JOB_ERROR_REPORTING_STRATEGY}
- JOB_ERROR_REPORTING_SENTRY_DSN=${JOB_ERROR_REPORTING_SENTRY_DSN}
- ACTIVITY_MAX_ATTEMPT=${ACTIVITY_MAX_ATTEMPT}
- ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=${ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS}
- ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=${ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS}
- WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=${WORKFLOW_FAILURE_RESTART_DELAY_SECONDS}
- USE_STREAM_CAPABLE_STATE=${USE_STREAM_CAPABLE_STATE}
- MICRONAUT_ENVIRONMENTS=${WORKERS_MICRONAUT_ENVIRONMENTS}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- workspace:${WORKSPACE_ROOT}
- ${LOCAL_ROOT}:${LOCAL_ROOT}
ports:
- 9000
networks:
- airbyte_internal
server:
image: airbyte/server:${VERSION}
logging: *default-logging
container_name: airbyte-server
restart: unless-stopped
environment:
- AIRBYTE_ROLE=${AIRBYTE_ROLE:-}
- AIRBYTE_VERSION=${VERSION}
- CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
- CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
- CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
- CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
- CONFIG_ROOT=${CONFIG_ROOT}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_URL=${DATABASE_URL}
- DATABASE_USER=${DATABASE_USER}
- JOB_MAIN_CONTAINER_CPU_LIMIT=${JOB_MAIN_CONTAINER_CPU_LIMIT}
- JOB_MAIN_CONTAINER_CPU_REQUEST=${JOB_MAIN_CONTAINER_CPU_REQUEST}
- JOB_MAIN_CONTAINER_MEMORY_LIMIT=${JOB_MAIN_CONTAINER_MEMORY_LIMIT}
- JOB_MAIN_CONTAINER_MEMORY_REQUEST=${JOB_MAIN_CONTAINER_MEMORY_REQUEST}
- JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
- LOG_LEVEL=${LOG_LEVEL}
- NEW_SCHEDULER=${NEW_SCHEDULER}
- SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
- TEMPORAL_HOST=${TEMPORAL_HOST}
- TRACKING_STRATEGY=${TRACKING_STRATEGY}
- JOB_ERROR_REPORTING_STRATEGY=${JOB_ERROR_REPORTING_STRATEGY}
- JOB_ERROR_REPORTING_SENTRY_DSN=${JOB_ERROR_REPORTING_SENTRY_DSN}
- WEBAPP_URL=${WEBAPP_URL}
- WORKER_ENVIRONMENT=${WORKER_ENVIRONMENT}
- WORKSPACE_ROOT=${WORKSPACE_ROOT}
- GITHUB_STORE_BRANCH=${GITHUB_STORE_BRANCH}
ports:
- 8001
volumes:
- workspace:${WORKSPACE_ROOT}
- data:${CONFIG_ROOT}
- ${LOCAL_ROOT}:${LOCAL_ROOT}
networks:
- airbyte_internal
webapp:
image: airbyte/webapp:${VERSION}
logging: *default-logging
container_name: airbyte-webapp
restart: unless-stopped
ports:
- 80
environment:
- AIRBYTE_ROLE=${AIRBYTE_ROLE:-}
- AIRBYTE_VERSION=${VERSION}
- API_URL=${API_URL:-}
- INTERNAL_API_HOST=${INTERNAL_API_HOST}
#- CONNECTOR_BUILDER_API_HOST=${CONNECTOR_BUILDER_API_HOST} #FIXME: Uncomment this block when enabling airbyte-connector-builder
- OPENREPLAY=${OPENREPLAY:-}
- PAPERCUPS_STORYTIME=${PAPERCUPS_STORYTIME:-}
- TRACKING_STRATEGY=${TRACKING_STRATEGY}
networks:
- airbyte_internal
airbyte-temporal:
image: airbyte/temporal:${VERSION}
logging: *default-logging
container_name: airbyte-temporal
restart: unless-stopped
environment:
- DB=postgresql
- DB_PORT=${DATABASE_PORT}
- DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development.yaml
- LOG_LEVEL=${LOG_LEVEL}
- POSTGRES_PWD=${DATABASE_PASSWORD}
- POSTGRES_SEEDS=${DATABASE_HOST}
- POSTGRES_USER=${DATABASE_USER}
configs:
- source: airbyte_development.yaml
target: /etc/temporal/config/dynamicconfig/development.yaml
networks:
- airbyte_internal
airbyte-cron:
image: airbyte/cron:${VERSION}
logging: *default-logging
container_name: airbyte-cron
restart: unless-stopped
environment:
- AIRBYTE_VERSION=${VERSION}
- CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_URL=${DATABASE_URL}
- DATABASE_USER=${DATABASE_USER}
- DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
- LOG_LEVEL=${LOG_LEVEL}
- REMOTE_CONNECTOR_CATALOG_URL=${REMOTE_CONNECTOR_CATALOG_URL}
- TEMPORAL_HISTORY_RETENTION_IN_DAYS=${TEMPORAL_HISTORY_RETENTION_IN_DAYS}
- UPDATE_DEFINITIONS_CRON_ENABLED=${UPDATE_DEFINITIONS_CRON_ENABLED}
- WORKSPACE_ROOT=${WORKSPACE_ROOT}
- MICRONAUT_ENVIRONMENTS=${CRON_MICRONAUT_ENVIRONMENTS}
volumes:
- workspace:${WORKSPACE_ROOT}
networks:
- airbyte_internal
#airbyte-connector-builder-server: #FIXME: Uncomment this block when enabling airbyte-connector-builder
#image: airbyte/connector-builder-server:${VERSION}
# logging: *default-logging
#container_name: airbyte-connector-builder-server
#restart: unless-stopped
#ports:
# - 8003
#environment:
# - AIRBYTE_VERSION=${VERSION}
# networks:
# - airbyte_internal
airbyte-proxy:
image: airbyte/proxy:${VERSION}
container_name: airbyte-proxy
ports:
- 8004:8004
- 8001:8001
# - 8003:8003
environment:
- BASIC_AUTH_USERNAME=${BASIC_AUTH_USERNAME}
- BASIC_AUTH_PASSWORD=${BASIC_AUTH_PASSWORD}
networks:
- airbyte_internal
- airbyte_public
depends_on:
- webapp
- server
volumes:
workspace:
name: ${WORKSPACE_DOCKER_MOUNT}
# the data volume is only needed for backward compatibility; when users upgrade
# from an old Airbyte version that relies on file-based configs, the server needs
# to read this volume to copy their configs to the database
data:
name: ${DATA_DOCKER_MOUNT}
db:
name: ${DB_DOCKER_MOUNT}
networks:
airbyte_public:
airbyte_internal:
configs:
airbyte_development.yaml:
external: true
default.conf.template:
external: true
And this my default.config.template:
upstream api-server {
server $INTERNAL_API_HOST;
}
upstream connector-builder-server {
server $CONNECTOR_BUILDER_API_HOST;
}
server {
listen 80;
listen [::]:80;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
add_header Content-Security-Policy "script-src * 'unsafe-inline'; worker-src self blob:;";
location / {
root /usr/share/nginx/html;
try_files $uri $uri/ /index.html;
sub_filter </head>
'</head><script language="javascript"\>
window.TRACKING_STRATEGY = "$TRACKING_STRATEGY";
window.AIRBYTE_VERSION = "$AIRBYTE_VERSION";
window.API_URL = "$API_URL";
window.CONNECTOR_BUILDER_API_URL = "$CONNECTOR_BUILDER_API_URL";
</script>';
sub_filter_once on;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location /api/ {
fastcgi_read_timeout 1h;
proxy_read_timeout 1h;
client_max_body_size 200M;
proxy_pass http://api-server/api/;
}
location /connector-builder-api/ {
fastcgi_read_timeout 1h;
proxy_read_timeout 1h;
client_max_body_size 200M;
proxy_pass http://connector-builder-server/;
}
}
Deleting the container-name properties and renaming server to airbyte-server fixed the issue, a running nginx service is not needed
For a more detailed fix refer to: can't create /etc/nginx/conf.d/default.conf: Read-only file system · Issue #19787 · airbytehq/airbyte · GitHub
Thanks so much for your patience - I was out for Thanksgiving. That is wonderful to hear, thank you so much for following up - I am sure this will be really helpful to other users!