Summary
The user is encountering an error while trying to clone the Airbyte repository. The error message indicates an issue with the RPC connection and an unexpected disconnect while reading sideband packet.
Question
Cloning into ‘airbyte’…
remote: Enumerating objects: 648425, done.
remote: Counting objects: 100% (2576/2576), done.
remote: Compressing objects: 100% (1250/1250), done.
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
error: 1921 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
I installed AirByte and am getting this error, why?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["error", "cloning", "airbyte-repository", "rpc-failed", "http2"]
Hmm, I don’t know enough to give you a confident answer yet. The error you’re encountering seems to be related to git and not specifically to Airbyte. It might be due to network issues, git configuration, or server problems. However, if you’re trying to set up Airbyte, you can follow these steps as mentioned in the tutorial: 1. Make sure you have Docker, Docker compose, and git set up on your machine. 2. Open a terminal, and go to a location where you want to download Airbyte. 3. Run the command: git clone <https://github.com/airbytehq/airbyte.git> 4. Go into the cloned airbyte repo by running cd airbyte. 5. Then run: docker-compose up or docker compose up if you are on the newest version of the Docker CLI. This will create and start the Airbyte containers. After it’s done, you can access Airbyte at http://localhost:8000/. If the problem persists, it might be helpful to look for solutions or ask for help in the Airbyte Github Discussions or Issues pages.
<@U04SE163WC8> Is there any way to help me? Instead of connecting to the database created by AirByte, how do I connect to my Postgresql externally?
Yes, you can connect Airbyte to an external Postgres database. Here are the steps you need to follow: 1. First, if you don’t have a running Postgres instance, you can create a new one using Docker with the following command: bash docker run --rm --name airbyte-postgres -e POSTGRES_PASSWORD=password -p 3000:5432 -d postgres 2. Next, you need to configure Airbyte services with this new database by editing the following environment variables declared in the .env file: bash DATABASE_USER=postgres DATABASE_PASSWORD=password DATABASE_HOST=host.docker.internal # refers to localhost of host DATABASE_PORT=3000 DATABASE_DB=postgres 3. If you want to separate the Config Database and the Job Database, you can specify separate parameters like this: bash CONFIG_DATABASE_USER=airbyte_config_db_user CONFIG_DATABASE_PASSWORD=password 4. You also need to redefine the JDBC URL in the DATABASE_URL environment variable to include the correct host, port, and database. If you need to provide extra arguments to the JDBC driver (for example, to handle SSL) you should add it here as well: bash DATABASE_URL=jdbc:<postgresql://host.docker.internal:3000/postgres?ssl=true&sslmode=require> 5. If the config database is separate from the job database, do the same for the CONFIG_DATABASE_URL: bash CONFIG_DATABASE_URL=jdbc:postgresql://<host>:<port>/<database>?<extra-parameters> Please note that currently, Airbyte doesn’t have a migration feature between the internal database and an external one. You need to manually backup the database. For more details, you can refer to the Airbyte Documentation.
<@U04SE163WC8> I don’t get what you mean. Where do I need to edit the information to connect the AirByte log to my external Postgresql?
To connect Airbyte to your external PostgreSQL, you need to edit the charts/airbyte/values.yaml file in your Airbyte deployment. Here are the steps: 1. Disable the default Postgres database (airbyte/db) in the charts/airbyte/values.yaml file: yaml postgresql: enabled: false 2. Enable and configure the external Postgres database in the same values.yaml file: yaml externalDatabase: host: ## Database host user: ## Non-root username for the Airbyte database database: db-airbyte ## Database name port: 5432 ## Database port number 3. For the non-root user’s password which has database access, you may use password, existingSecret or jdbcUrl. We recommend using existingSecret, or injecting sensitive fields from your own external secret store. Each of these parameters is mutually exclusive: yaml externalDatabase: ... password: ## Password for non-root database user existingSecret: ## The name of an existing Kubernetes secret containing the password. existingSecretPasswordKey: ## The Kubernetes secret key containing the password. jdbcUrl: "jdbc:postgresql://<user>:<password>@localhost:5432/db-airbyte" ## Full database JDBC URL. You can also add additional arguments. The optional jdbcUrl field should be entered in the following format: jdbc:<postgresql://localhost:5432/db-airbyte>. We recommend against using this unless you need to add additional extra arguments can be passed to the JDBC driver at this time (e.g. to handle SSL). For more details, you can refer to the Airbyte Documentation.
<@U04SE163WC8> Why can’t I find such a directory in the AirByte source code on Git?
I’m sorry for the confusion. The instructions I provided are for a Kubernetes deployment of Airbyte, which is part of the enterprise edition. If you’re using the open-source version of Airbyte, the process is a bit different. For the open-source version, you need to modify the .env file in the root directory of your Airbyte source code. If you can’t find this file, you can create one. Here’s an example of what the .env file should look like: bash DATABASE_USER=postgres DATABASE_PASSWORD=password DATABASE_HOST=host.docker.internal # refers to localhost of host DATABASE_PORT=3000 DATABASE_DB=postgres Please replace postgres, password, host.docker.internal, 3000, and postgres with your actual PostgreSQL username, password, host, port, and database name respectively. After modifying the .env file, you need to restart your Airbyte instance for the changes to take effect. If you’re using Docker Compose to run Airbyte, you can restart it with the following command: bash docker-compose down && docker-compose up Please note that this will only change the database that Airbyte uses to store its own data (like connector configurations, job history, etc.). It won’t affect the destination databases that Airbyte writes data to during sync jobs. Those are configured separately in the Airbyte UI. For more details, you can refer to the Airbyte Documentation.
DATABASE
DATABASE_USER=postgres
DATABASE_PASSWORD=Test
DATABASE_HOST=34.41.13.81
DATABASE_PORT=5432
DATABASE_DB=logging
translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB} (do not i>
DATABASE_URL=jdbc:postgresql://34.41.13.81:5432/logging
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.40.26.001
Airbyte Internal Config Database, defaults to Job Database if empty. Explicitly left empty to mute docker co>
CONFIG_DATABASE_USER=
CONFIG_DATABASE_PASSWORD=
CONFIG_DATABASE_URL=
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.40.23.002
<@U04SE163WC8> Why do I get this error when I connect to Postgresql fine outside?
ERROR Unable to create SQL database. {“error”: “unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: password authentication failed for user "postgres" pq: password authentication failed for user "postgres"]”, “logging-call-at”: “handler.go:94”}
The error message you’re seeing indicates that the password authentication failed for the user “postgres”. This could be due to a few reasons: 1. The password provided in the .env file is incorrect. Please double-check the DATABASE_PASSWORD value in your .env file. It should match the password for the “postgres” user in your PostgreSQL database. bash DATABASE_PASSWORD=Test 2. The user “postgres” does not exist in your PostgreSQL database, or it does not have the correct permissions. You can check this by logging into your PostgreSQL database and running the following command: sql SELECT usename FROM pg_user; This will list all users in your PostgreSQL database. If “postgres” is not in the list, you will need to create it. 3. The PostgreSQL database is not configured to allow password authentication for the “postgres” user. You can check this in the pg_hba.conf file in your PostgreSQL database. If it’s not set to allow password authentication, you will need to change it. If you’ve checked all of these and you’re still seeing the error, it might be helpful to look at the PostgreSQL server logs for more information. You can usually find these in the data directory of your PostgreSQL installation, in a file called postgresql-[version]-main.log. For more details, you can refer to the Airbyte Documentation.
SHARED
VERSION=0.50.53
When using the airbyte-db via default docker image
CONFIG_ROOT=/data
DATA_DOCKER_MOUNT=airbyte_data
DB_DOCKER_MOUNT=airbyte_db
Workspace storage for running jobs (logs, etc)
WORKSPACE_ROOT=/tmp/workspace
WORKSPACE_DOCKER_MOUNT=airbyte_workspace
Local mount to access local files from filesystem
todo (cgardens) - when we are mount raw directories instead of named volumes, *_DOCKER_MOUNT must
be the same as *_ROOT.
LOCAL_ROOT=/tmp/airbyte_local
LOCAL_DOCKER_MOUNT=/tmp/airbyte_local
todo (cgardens) - hack to handle behavior change in docker compose. *_PARENT directories MUST
already exist on the host filesystem and MUST be parents of *_ROOT.
HACK_LOCAL_ROOT_PARENT=/tmp
Storage type
WORKER_LOGS_STORAGE_TYPE=local
WORKER_STATE_STORAGE_TYPE=local
Proxy Configuration
Set to empty values, e.g. “” to disable basic auth
BASIC_AUTH_USERNAME=airbyte
BASIC_AUTH_PASSWORD=password
BASIC_AUTH_PROXY_TIMEOUT=900
DATABASE
DATABASE_USER=airbyte
DATABASE_PASSWORD=airbyte
DATABASE_HOST=34.41.13.81
DATABASE_PORT=5432
DATABASE_DB=airbyte
translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB} (do not include >
DATABASE_URL=jdbc:postgresql://airbyte:airbyte!#$%@34.41.13.81:5432/logging
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.40.26.001
I want to change the connection instead of airbyte-db to the external Postgresql database. Am I doing something wrong in the above?