Connection Error after Upgrading to abctl in Self-Hosted Docker Installation

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

After upgrading to abctl in a self-hosted docker installation, encountering a connection error with a Postgres source. Error message: io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10196ms (total=0, active=0, idle=0, waiting=0). Seeking help to resolve this.


Question

Hello! After having no issues for >1 month with a self-hosted docker compose-based installation, we upgraded to abctl today. We had to double our machine size due to CPU limitations, but are now getting io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10196ms (total=0, active=0, idle=0, waiting=0) (for a Postgres source). Has anyone seen / resolved this?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["connection-error", "abctl", "docker", "postgres-source", "connection-timeout"]
2

OK! It now failed with a different error around the replication slot. I’ve dropped/recreated the slot and am running a full data refresh. Let’s see how it goes!

3

Interesting: the machine, (EC2 t3a.xlarge, nothing running on it except abctl with one Postgres->Redshift connector) has become unresponsive (can’t ssh, no metrics collected for 10 minutes). Before getting to that state it seemed to be syncing some data, but is now unreachable. I’ll restart and try again.

4

<@U01MMSDJGC9> The machine became unresponsive again after a reboot. I happened to have top open sorted by memory usage and am including a screenshot. Notably the load average is ~32, the machine is out of memory, several java processes are using considerable/all virtual memory. Any recommendations?

5

While it restarts, I’m curious—are we using the wrong machine size entirely for a production use case? It seems like it’s overprovisioned based on the docs, but I’m seeing a lot of signals that perhaps we’re off a bit in terms of specs?

6

Hello Adam, do you have more information about the version you were running previously? What machine are you using before/now?

7

Hi <@U01MMSDJGC9>! Thank you for reaching out! I was previously on 0.63.8 installed via the run-ab-platform.sh path <https://docs.airbyte.com/deploying-airbyte/on-aws-ec2|described here>. We were on an EC2 t3a.large, but with the upgrade to abctl, we’ve moved to t3a.xlarge based on other CPU limit errors we were seeing.

8

If it helps, <Issues · airbytehq/airbyte · GitHub issue> has more of a stack trace and context on the current versions of software we’re running.

9

Adam, can you consider run abctl with the flag --low-resource-mode it will allow to have more syncs in parallel.

10

how many connections do you have in your instance?

11

<@U01MMSDJGC9> We’ve currently got a single Postgres->Redshift connection. Our plan is to migrate another 5-10 from another provider, but are blocked on that until we can figure out why this connector has stopped. I just ran it with the --low-resource-mode flag and got this:

  ERROR   Failed to install airbyte/airbyte Helm Chart                                                                                                                                  
  ERROR   Unable to install Airbyte locally                                                                                                                                             
  ERROR   unable to install airbyte chart: unable to install helm: failed to create patch: The order in patch list:
          [map[name:JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST valueFrom:map[configMapKeyRef:map[key:JOB_MAIN_CONTAINER_CPU_REQUEST name:airbyte-abctl-airbyte-env]]] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST valueFrom:map[configMapKeyRef:map[key:JOB_MAIN_CONTAINER_MEMORY_REQUEST name:airbyte-abctl-airbyte-env]]] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:SECRET_PERSISTENCE value:<nil>] map[name:JOB_RESOURCE_VARIANT_OVERRIDE value:lowresource]]
           doesn't match $setElementOrder list:
          [map[name:API_AUTHORIZATION_ENABLED] map[name:LOG_LEVEL] map[name:LOG4J_CONFIGURATION_FILE] map[name:AIRBYTE_API_HOST] map[name:AIRBYTE_VERSION] map[name:AIRBYTE_EDITION] map[name:AIRBYTE_URL] map[name:CONFIG_ROOT] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:MICRONAUT_ENVIRONMENTS] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TRACKING_STRATEGY] map[name:WORKER_ENVIRONMENT] map[name:WORKSPACE_ROOT] map[name:WEBAPP_URL] map[name:TEMPORAL_HOST] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION] map[name:JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION] map[name:KEYCLOAK_INTERNAL_HOST] map[name:CONNECTOR_BUILDER_SERVER_API_HOST] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:WORKLOAD_LAUNCHER_ENABLED] map[name:WORKLOAD_API_SERVER_ENABLED] map[name:AB_INSTANCE_ADMIN_PASSWORD] map[name:AB_INSTANCE_ADMIN_CLIENT_ID] map[name:AB_INSTANCE_ADMIN_CLIENT_SECRET] map[name:AB_JWT_SIGNATURE_SECRET] map[name:AB_COOKIE_SECURE] map[name:AB_COOKIE_SAME_SITE] map[name:SECRET_PERSISTENCE] map[name:S3_PATH_STYLE_ACCESS] map[name:STORAGE_TYPE] map[name:STORAGE_BUCKET_ACTIVITY_PAYLOAD] map[name:STORAGE_BUCKET_LOG] map[name:STORAGE_BUCKET_STATE] map[name:STORAGE_BUCKET_WORKLOAD_OUTPUT] map[name:AWS_ACCESS_KEY_ID] map[name:AWS_SECRET_ACCESS_KEY] map[name:MINIO_ENDPOINT] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:AIRBYTE_INSTALLATION_ID] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:JOB_RESOURCE_VARIANT_OVERRIDE] map[name:POSTGRES_TLS_ENABLED] map[name:POSTGRES_TLS_DISABLE_HOST_VERIFICATION] map[name:SQL_TLS_ENABLED] map[name:SQL_TLS_DISABLE_HOST_VERIFICATION]]
           && failed to create patch: The order in patch list:
          [map[name:JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST valueFrom:map[configMapKeyRef:map[key:JOB_MAIN_CONTAINER_CPU_REQUEST name:airbyte-abctl-airbyte-env]]] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT valueFrom:map[configMapKeyRef:map[key:JOB_MAIN_CONTAINER_CPU_LIMIT name:airbyte-abctl-airbyte-env]]] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST valueFrom:map[configMapKeyRef:map[key:JOB_MAIN_CONTAINER_MEMORY_REQUEST name:airbyte-abctl-airbyte-env]]] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:SECRET_PERSISTENCE value:<nil>] map[name:CHECK_JOB_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:DISCOVER_JOB_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:SIDECAR_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:SIDECAR_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:SIDECAR_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:SIDECAR_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:SPEC_JOB_MAIN_CONTAINER_CPU_LIMIT value:0] map[name:SPEC_JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:SPEC_JOB_MAIN_CONTAINER_MEMORY_LIMIT value:0] map[name:SPEC_JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0]]
           doesn't match $setElementOrder list:
          [map[name:AIRBYTE_VERSION] map[name:DATA_PLANE_ID] map[name:PUB_SUB_ENABLED] map[name:PUB_SUB_TOPIC_NAME] map[name:CONFIG_ROOT] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:LOG_LEVEL] map[name:LOG4J_CONFIGURATION_FILE] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TRACKING_STRATEGY] map[name:WORKSPACE_DOCKER_MOUNT] map[name:WORKSPACE_ROOT] map[name:LOCAL_ROOT] map[name:WEBAPP_URL] map[name:WORKLOAD_API_HOST] map[name:TEMPORAL_HOST] map[name:TEMPORAL_WORKER_PORTS] map[name:JOB_KUBE_NAMESPACE] map[name:JOB_KUBE_SERVICEACCOUNT] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:KUBERNETES_CLIENT_MAX_IDLE_CONNECTIONS] map[name:WORKLOAD_API_BEARER_TOKEN] map[name:WORKLOAD_API_BEARER_TOKEN_SECRET_NAME] map[name:WORKLOAD_API_BEARER_TOKEN_SECRET_KEY] map[name:INTERNAL_API_HOST] map[name:CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION] map[name:JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION] map[name:METRIC_CLIENT] map[name:OTEL_COLLECTOR_ENDPOINT] map[name:ACTIVITY_MAX_ATTEMPT] map[name:ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS] map[name:ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS] map[name:WORKFLOW_FAILURE_RESTART_DELAY_SECONDS] map[name:SHOULD_RUN_NOTIFY_WORKFLOWS] map[name:MICRONAUT_ENVIRONMENTS] map[name:WORKLOAD_LAUNCHER_PARALLELISM] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:SECRET_PERSISTENCE] map[name:S3_PATH_STYLE_ACCESS] map[name:STORAGE_TYPE] map[name:STORAGE_BUCKET_ACTIVITY_PAYLOAD] map[name:STORAGE_BUCKET_LOG] map[name:STORAGE_BUCKET_STATE] map[name:STORAGE_BUCKET_WORKLOAD_OUTPUT] map[name:AWS_ACCESS_KEY_ID] map[name:AWS_SECRET_ACCESS_KEY] map[name:MINIO_ENDPOINT] map[name:AIRBYTE_INSTALLATION_ID] map[name:CHECK_JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:DISCOVER_JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:JOB_MAIN_CONTAINER_MEMORY_REQUEST] map[name:SIDECAR_MAIN_CONTAINER_CPU_LIMIT] map[name:SIDECAR_MAIN_CONTAINER_CPU_REQUEST] map[name:SIDECAR_MAIN_CONTAINER_MEMORY_LIMIT] map[name:SIDECAR_MAIN_CONTAINER_MEMORY_REQUEST] map[name:SPEC_JOB_MAIN_CONTAINER_CPU_LIMIT] map[name:SPEC_JOB_MAIN_CONTAINER_CPU_REQUEST] map[name:SPEC_JOB_MAIN_CONTAINER_MEMORY_LIMIT] map[name:SPEC_JOB_MAIN_CONTAINER_MEMORY_REQUEST]]```
12

I’m not sure why you can’t just reinstall with low resource. Please run abctl local uninstall and than abctl local install --low-resource-mode :slightly_smiling_face:

13

Thank you for the advice so far!