Connection setup in Airbyte UI is slow and gets stuck at ‘Please wait a little bit more’ message.
I set up a connection in airbyte UI but it took a lot of time and only "Please wait a little bit more…" <@U04SE163WC8>
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Based on the information provided in the extracts, it’s not clear why your connection setup in the Airbyte UI is taking a long time. However, it’s important to verify a few things: 1. Make sure your source and destination databases are accessible from the machine running Airbyte. This depends on your networking setup. You can verify this using the check connection tool in the UI (source). 2. If you’re using Terraform to set up Airbyte connectors, ensure that Terraform has completed its tasks. You should see your source and destination connectors, as well as the connection between them, set up and ready to go in the Airbyte UI (source). 3. If you’re setting up a MySQL source in Airbyte, after filling in the configuration fields and clicking on ‘Set up source’, Airbyte will test the connection. If everything goes well, you should see a successful message (source). If the issue persists, it might be a good idea to seek help from the Airbyte community on their forum or Slack channel.
may the airbyte open source be slow when I had over 20 sources / connector builders <@U04SE163WC8>
A message from http://kapa.ai|kapa.ai
Yes, it’s possible that you might encounter scaling issues with Airbyte when running over a thousand connections. As a reference point, the typical Airbyte user has 5 - 20 connectors and 10 - 100 connections configured. Almost all of these connections are scheduled, either hourly or daily, resulting in at most 100 concurrent jobs. If you have a larger workload, you might need to consider scaling your Airbyte deployment (source). However, the performance of each connector also depends on the connector configuration and the API you’re trying to replicate. Changing the worker configuration in the .env file will only increase/decrease the number of parallel syncs you’ll run (source). If you’re experiencing performance issues, it might be worth checking if there are any specific limitations with the connectors you’re using. For example, the Notion connector is restricted by Notion’s request limits (source). If you’re still facing issues, consider reaching out to the Airbyte community on their forum or Slack channel for further assistance.
<@U04SE163WC8> 2024-01-08 11:01:45 platform > Docker volume job log path: /tmp/workspace/6f13b127-e383-4593-ba09-b7a888496e2f/0/logs.log
2024-01-08 11:01:45 platform > Executing worker wrapper. Airbyte version: 0.50.41
2024-01-08 11:01:45 platform > Attempt 0 to save workflow id for cancellation
2024-01-08 11:01:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: ‘2.0’
2024-01-08 11:01:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: ‘2.0’
2024-01-08 11:01:45 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: ‘0.1’
2024-01-08 11:01:45 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: ‘0.1’
2024-01-08 11:01:45 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ‘’
2024-01-08 11:01:45 platform > Checking if airbyte/source-declarative-manifest:0.57.0 exists…
2024-01-08 11:01:45 platform > airbyte/source-declarative-manifest:0.57.0 was found locally.
2024-01-08 11:01:45 platform > Creating docker container = source-declarative-manifest-discover-6f13b127-e383-4593-ba09-b7a888496e2f-0-ztiee with resources io.airbyte.config.ResourceRequirements@1f0ed758[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-01-08 11:01:45 platform > Preparing command: docker run --rm --init -i -w /data/6f13b127-e383-4593-ba09-b7a888496e2f/0 --log-driver none --name source-declarative-manifest-discover-6f13b127-e383-4593-ba09-b7a888496e2f-0-ztiee --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-declarative-manifest:0.57.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.41 -e WORKER_JOB_ID=6f13b127-e383-4593-ba09-b7a888496e2f airbyte/source-declarative-manifest:0.57.0 discover --config source_config.json
2024-01-08 11:01:45 platform > Reading messages from protocol version 0.2.0
Based on the log you’ve provided, it seems like Airbyte is attempting to run a discovery operation on a source connector (source-declarative-manifest:0.57.0). Here’s a breakdown of what’s happening: 1. Airbyte is preparing to run a Docker container with the name source-declarative-manifest-discover-6f13b127-e383-4593-ba09-b7a888496e2f-0-ztiee. This container will be used to perform the discovery operation on the source connector. 2. The Docker container is being run with several environment variables, such as DEPLOYMENT_MODE=OSS and WORKER_CONNECTOR_IMAGE=airbyte/source-declarative-manifest:0.57.0. These variables are used to configure the behavior of the connector. 3. The command docker run --rm --init -i -w /data/6f13b127-e383-4593-ba09-b7a888496e2f/0 --log-driver none --name source-declarative-manifest-discover-6f13b127-e383-4593-ba09-b7a888496e2f-0-ztiee --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-declarative-manifest:0.57.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=<http://host.docker.internal:4317> -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.41 -e WORKER_JOB_ID=6f13b127-e383-4593-ba09-b7a888496e2f airbyte/source-declarative-manifest:0.57.0 discover --config source_config.json is being executed to start the discovery operation. 4. The log ends with “Reading messages from protocol version 0.2.0”, which indicates that Airbyte is starting to process the data from the source connector. If you’re experiencing issues with this operation, it might be worth checking the logs for any error messages or warnings. You can access the logs in the Airbyte UI, or directly from the Docker container (source). If you’re still having trouble, consider reaching out to the Airbyte community on their forum or Slack channel for further assistance.
explain error Discovering schema failed <@U04SE163WC8> common.error
The error message “Discovering schema failed” typically indicates that Airbyte encountered an issue while trying to retrieve the schema from your data source. This could be due to a variety of reasons such as network issues, incorrect configuration, or limitations with the specific connector you’re using. Unfortunately, the provided extracts do not contain specific information about the “common.error” you mentioned. This error message is quite generic and could be caused by a variety of issues. Here are a few general troubleshooting steps you can take: 1. Check your source configuration: Make sure that all the details like host, port, credentials, etc., are correct. 2. Check the logs: Airbyte logs can provide more detailed information about what went wrong. You can access these logs via the Airbyte UI or directly from the Docker container (source). 3. Check the connector documentation: Some connectors have specific requirements or limitations that might cause issues with schema discovery. If you’re still having trouble, consider reaching out to the Airbyte community on their forum or Slack channel for further assistance. They might be able to provide more specific guidance based on the exact error message and your setup.