Summary
When trying to connect from Source in Builder to Destination BigQuery, an error related to Docker volumes and container creation is encountered.
Question
When connecting from Source in Builder with Destination BigQuery, I get the following error
2024-02-29 02:09:40 platform > Executing worker wrapper. Airbyte version: 0.50.54
2024-02-29 02:09:40 platform > Attempt 0 to save workflow id for cancellation
2024-02-29 02:09:40 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-29 02:09:40 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-29 02:09:40 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-29 02:09:40 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-29 02:09:40 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-29 02:09:40 platform > Checking if airbyte/source-declarative-manifest:0.65.0 exists...
2024-02-29 02:09:40 platform > airbyte/source-declarative-manifest:0.65.0 was found locally.
2024-02-29 02:09:40 platform > Creating docker container = source-declarative-manifest-discover-b198c788-5b33-4411-a736-584cd07aa0e4-0-hqomf with resources io.airbyte.config.ResourceRequirements@729fec5[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-02-29 02:09:40 platform > Preparing command: docker run --rm --init -i -w /data/b198c788-5b33-4411-a736-584cd07aa0e4/0 --log-driver none --name source-declarative-manifest-discover-b198c788-5b33-4411-a736-584cd07aa0e4-0-hqomf --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-declarative-manifest:0.65.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE=dev -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=<http://host.docker.internal:4317> -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.54 -e WORKER_JOB_ID=b198c788-5b33-4411-a736-584cd07aa0e4 airbyte/source-declarative-manifest:0.65.0 discover --config source_config.json
2024-02-29 02:09:40 platform > Reading messages from protocol version 0.2.0```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1709172664682209) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["error", "connecting", "source-in-builder", "destination-bigquery", "docker-volumes", "container-creation"]
</sub>
Hmm, I don’t know enough to give you a confident answer yet. The error message or specific issue you’re encountering with BigQuery as the destination isn’t clearly stated in the log snippet you’ve provided. The log snippet mainly shows the initialization and setup process for a source connector in Airbyte, specifically airbyte/source-declarative-manifest:0.65.0, but it doesn’t detail the error encountered when connecting to BigQuery. For troubleshooting connections with BigQuery or any other destination in Airbyte, consider the following general steps: 1. Check the Airbyte Version: Ensure both your Airbyte instance and the connectors (source and destination) are up to date. Connector issues are often resolved in newer versions. 2. Review the Configuration: Double-check the configuration settings for both the source and destination. For BigQuery, this includes project ID, dataset ID, credentials (JSON key file), etc. 3. Consult the Logs: Detailed logs can provide more insight into what might be going wrong. Look for error messages or warnings that occur after the initialization steps you’ve posted. 4. Refer to Documentation: The Airbyte Documentation is a comprehensive resource for setup and troubleshooting. It might have specific guides or suggestions for the source and BigQuery destination. 5. Search or Post on Community Forums: If the issue persists, consider searching for similar issues on the Airbyte GitHub Issues page or the Airbyte Community Forum. If you don’t find a similar issue, you can post your own, including the details and logs you’ve gathered. Without a specific error message related to BigQuery, it’s challenging to provide a more targeted solution. If you can provide more details about the error you’re encountering or if there are more specific log entries related to the issue, that would be helpful for further troubleshooting.
I am using the latest version of AirByte Open Source and get this error. As for older versions, I still see a normal connection? <@U04SE163WC8>
Based on the information provided in the extracts, it seems you might be encountering issues after upgrading your Airbyte instance. While the extracts don’t directly mention the specific error you’re encountering, there are a couple of relevant points that could help address common upgrade issues or changes that might affect connections. 1. Upgrade Issue with Specific Connectors: If you’re using specific source connectors, such as airbyte/source-freshdesk, there was a mention in a forum post about an upgrade issue related to an IllegalArgumentException: malformed JsonSchema object type. The solution suggested was to ensure you’re using the latest version of the connector and to try upgrading Airbyte again. More details were provided in a GitHub pull request: PR #12325. This might not be your exact scenario, but it highlights the importance of keeping both Airbyte and its connectors up to date. plaintext Hello Dishon, looks in a recent PR we need to update the json lib Airbyte uses and need to update a few specifications. More details <https://github.com/airbytehq/airbyte/pull/12325> 2. General Upgrade Guidance: For users encountering issues after an upgrade, it’s recommended to check the version requirements for specific connectors. For example, the MySQL source connector had a backwards incompatible spec change between versions 0.6.8 and 0.6.9 related to the replication_method configuration parameter. While this might not directly apply to your situation, it underscores the need to review the documentation for any breaking changes or additional steps required after an upgrade. plaintext There is a backwards incompatible spec change between MySQL Source connector versions 0.6.8 and 0.6.9. 3. Troubleshooting and Support: If you’re running into issues that aren’t resolved by ensuring you’re using the latest versions of Airbyte and its connectors, consulting the Airbyte documentation and the Airbyte Forum for common troubleshooting issues can be helpful. Additionally, if you have an Airbyte Enterprise or OSS support package, submitting a support ticket through the Airbyte Help Center is an option. Without more specific details about the error you’re encountering, these are general recommendations based on the information available. If your issue is related to a specific connector or a unique scenario not covered in the extracts, consulting the Airbyte documentation or community forum for more targeted advice might be necessary.
I updated AirByte again to get the latest version. But this is still the case. And in case this error only occurs on some Sources created in the remaining Builder, the other Builders are still normal?
I updated AirByte again to get the latest version. But this is still the case. And in case this error only occurs on some Sources created in the remaining Builder, the other Builders are still normal? <@U04SE163WC8>
Based on the extracts provided, there are a few instances where users have encountered issues after updating Airbyte to a newer version, affecting specific sources or connectors. While the extracts don’t directly address your scenario of errors occurring only on some sources created in the remaining builder while others are normal, they do offer insights into troubleshooting and potential solutions for upgrade-related issues.
-
Updating Airbyte and Connector Versions: One user reported that their issue spontaneously resolved after updating Airbyte to the latest version. This suggests that ensuring both Airbyte and all connectors are up-to-date can sometimes resolve unexpected behaviors.
The issue spontaneously resolved after updating Airbyte to the latest version.
-
Checking for Specific Errors in Logs: Another extract mentions an error related to processing the latest destination definitions list after an upgrade. This indicates that checking the airbyte-server container logs for specific error messages can provide clues to the underlying issue.
2023-07-27 12:56:23 ERROR i.a.s.a.ApiHelper(execute):40 - Unexpected Exception
io.airbyte.commons.server.errors.InternalServerKnownException: Unable to process retrieved latest source definitions list
-
Downgrading as a Temporary Solution: If the issue persists after an upgrade and affects critical workflows, one user shared steps for downgrading Airbyte as a temporary workaround. This involves editing the version in the ./run-ab-platform.sh script and following specific steps to revert to a previous version. However, this approach comes with a warning about potentially needing to reset your configuration, which could involve redoing all sources, destinations, and connections.
Here are the complete steps that I did:
1. Change version on `./run-ab-platform.sh ` to `VERSION=0.50.20` (or any other release versions that you need)
2. If your instance is running, bring it down `docker compose down`
3. Remove your docker-compose and .env files `./run-ab-platform.sh -r`
4. Then bring you instance back up for the required version `./run-ab-platform.sh -b`
Given these insights, here are some steps you can take:
- Ensure Up-to-Date Versions: Verify that both Airbyte and all used connectors are updated to their latest versions.
- Review Logs for Errors: Check the logs, especially the airbyte-server container logs, for any specific error messages that could indicate what’s going wrong with the affected sources.
- Consider Downgrading: If the issue is severely impacting your data integration processes and you suspect the latest version introduced a regression, consider temporarily downgrading to a stable version that worked for you, following the steps mentioned above.
Based on the extracts provided, there are a few instances where users have encountered issues after updating Airbyte to a newer version, affecting specific sources or connectors. While the extracts don’t directly address your scenario of errors occurring only on some sources created in the remaining builder while others are normal, they do offer insights into troubleshooting and potential solutions for upgrade-related issues. 1. Updating Airbyte and Connector Versions: One user reported that their issue spontaneously resolved after updating Airbyte to the latest version. This suggests that ensuring both Airbyte and all connectors are up-to-date can sometimes resolve unexpected behaviors. plaintext The issue spontaneously resolved after updating Airbyte to the latest version. 2. Checking for Specific Errors in Logs: Another extract mentions an error related to processing the latest destination definitions list after an upgrade. This indicates that checking the airbyte-server container logs for specific error messages can provide clues to the underlying issue. shell 2023-07-27 12:56:23 ERROR i.a.s.a.ApiHelper(execute):40 - Unexpected Exception io.airbyte.commons.server.errors.InternalServerKnownException: Unable to process retrieved latest source definitions list 3. Downgrading as a Temporary Solution: If the issue persists after an upgrade and affects critical workflows, one user shared steps for downgrading Airbyte as a temporary workaround. This involves editing the version in the ./run-ab-platform.sh script and following specific steps to revert to a previous version. However, this approach comes with a warning about potentially needing to reset your configuration, which could involve redoing all sources, destinations, and connections. plaintext Here are the complete steps that I did: 1. Change version on `./run-ab-platform.sh ` to `VERSION=0.50.20` (or any other release versions that you need) 2. If your instance is running, bring it down `docker compose down` 3. Remove your docker-compose and .env files `./run-ab-platform.sh -r` 4. Then bring you instance back up for the required version `./run-ab-platform.sh -b` Given these insights, here are some steps you can take: - Ensure Up-to-Date Versions: Verify that both Airbyte and all used connectors are updated to their latest versions. - Review Logs for Errors: Check the logs, especially the airbyte-server container logs, for any specific error messages that could indicate what’s going wrong with the affected sources. - Consider Downgrading: If the issue is severely impacting your data integration processes and you suspect the latest version introduced a regression, consider temporarily downgrading to a stable version that worked for you, following the steps mentioned above. If these steps do not resolve your issue, or if the problem is more specific than what’s covered in the extracts, consulting the Airbyte documentation and seeking support from the Airbyte community forum may provide more targeted assistance.
What should I do if I want to delete all data related to the builder I created? <@U04SE163WC8>
To delete all data related to a builder you created in Airbyte, including sources, destinations, and any associated configurations, you can follow a general approach based on the information provided in the extracts. However, it’s important to note that the extracts do not directly provide a step-by-step guide specifically for deleting a builder and its related data. Instead, they offer insights into managing Airbyte’s data and configurations, which can be adapted to your needs. 1. Direct Database Manipulation: If your builder has created sources, destinations, or other configurations stored in Airbyte’s database, you might consider directly manipulating the database to remove these entries. One extract from a forum post suggests deleting data directly from the database for custom connector removal: plaintext Hey, we store all the data in the database you can delete it from the db directly. But make sure you delete all the sources/destinations related to that connector deleted before removing it from the database. To access the Airbyte database, you can use the following command to exec into the Airbyte database container, assuming you’re using Docker: shell docker exec -it airbyte-db psql -U airbyte Once inside, you can run SQL commands to delete the relevant records. Be cautious with direct database manipulations, as they can lead to data inconsistencies if not done correctly. 2. Resetting the Airbyte Developer Environment: If you’re looking to completely reset your Airbyte instance, removing all configurations including those created by the builder, the following steps can be followed. This approach is more of a “nuclear option” and will remove all data in Airbyte, not just the data related to a specific builder: shell # Make sure you are in your local `airbyte-platform` repository # Delete the datastore volumes in docker VERSION=dev docker compose down -v # Remove the data on disk rm -rf /tmp/dev_root rm -rf /tmp/airbyte_local # Rebuild the project SUB_BUILD=PLATFORM ./gradlew clean build VERSION=dev docker compose up -V This approach is useful if you’re developing locally and want to start fresh, but it’s not suitable for selectively removing data related to a specific builder in a production environment. Given the potential complexity and risks associated with direct database manipulation, it’s crucial to have a clear understanding of the database schema and to take necessary precautions, such as backing up the database before making any changes. If you’re not comfortable with these steps or if your scenario involves a production environment, seeking further assistance from the Airbyte community forum may be advisable.
What should I do when I want to delete all data related to the builder that I created in Postgresql? <@U04SE163WC8>
Based on the information provided in the extracts, there isn’t a direct reference to a specific process for deleting all data related to a “builder” you created in PostgreSQL through Airbyte. However, we can infer a general approach from the guidance given for managing custom connectors and data within Airbyte’s ecosystem. Here’s a structured approach based on the available information: 1. Ensure All Related Sources/Destinations Are Deleted: Before directly manipulating the database to remove a builder and its related data, ensure that all sources and destinations created by the builder are deleted from the Airbyte UI. This step is crucial to avoid any dependencies that might cause issues. 2. Direct Database Manipulation: To delete data directly from the database, you would typically access the Airbyte database and run SQL commands to remove the relevant records. While the extracts do not provide specific SQL commands for this operation, the approach would involve identifying and deleting records related to your builder in the Airbyte database. Here’s a general guideline on how you might proceed, based on the advice for deleting a custom connector: - Access the Airbyte Database: If you’re using Docker, you can access the Airbyte database with the following command (this command is not directly from the extracts, but it’s a common way to access PostgreSQL databases in Docker containers): shell docker exec -it airbyte-db psql -U airbyte - Identify and Delete Related Records: Once inside the database, you would need to identify the tables and records related to your builder. This might involve tables that store information about sources, destinations, and any configurations or settings specific to the builder. The exact SQL commands would depend on your database schema and the data you need to remove. Here’s a hypothetical example of deleting a source related to your builder (note: this is a generic example, not specific SQL from the extracts): sql DELETE FROM sources WHERE name = 'YourBuilderSourceName'; Important: Directly manipulating the database can lead to data inconsistencies or loss if not done carefully. Always back up your database before making direct changes. 3. Consult Documentation or Community for Specific Guidance: Since direct database manipulation requires a good understanding of Airbyte’s database schema and the potential risks involved, consulting the Airbyte documentation or seeking advice from the Airbyte community forum is advisable for specific guidance tailored to your scenario. Remember, the steps above are inferred from general advice within the provided extracts and are not directly outlined instructions for deleting builder-related data in PostgreSQL. Proceed with caution and consider seeking additional support if needed.
What should I do if I delete something like that but still save it in other tables? Is there a way to completely delete the Builder-related keys I’ve deleted? <@U04SE163WC8>
Given the information from the extracts, there isn’t a direct reference to a comprehensive method for ensuring that all builder-related keys and data are completely deleted from every table in the Airbyte database, especially when considering dependencies and references across multiple tables. However, based on general database management practices and the insights provided, here’s a structured approach to ensure thorough deletion:
-
Identify All Related Data and Dependencies: Before deletion, it’s crucial to identify all tables and records that are related to the builder you’ve deleted. This includes not only the direct entries created by the builder but also any related data that might exist in other tables as a result of those entries. For example, if your builder created sources, destinations, or specific configurations, you’ll need to identify all tables where these entities are referenced.
-
Use Cascading Deletes If Possible: To ensure that deleting a record in one table also deletes related records in other tables, you can use cascading deletes if your database schema supports it. This is not directly mentioned in the extracts, but it’s a common database feature. A cascading delete would look something like this in SQL (note: this is a general example, not specific SQL from the extracts):
DELETE FROM main_table WHERE condition = 'YourCondition' CASCADE;
This command would delete not only the matching records in main_table but also any related records in tables that have foreign key references with a cascade delete rule set up. However, implementing cascading deletes requires that foreign key relationships are properly defined in your database schema with the ON DELETE CASCADE option.
-
Manual Cleanup for Non-Cascading Tables: If cascading deletes are not set up or if you prefer to manually control the deletion process, you’ll need to manually delete related records from each table. This involves running separate DELETE commands for each table where related data might exist, ensuring that you’re not leaving orphaned records.
Here’s a hypothetical example of manually deleting related records (again, this is a generic example, not specific SQL from the extracts):
DELETE FROM related_table WHERE builder_key = 'YourBuilderKey';
-
Consult Documentation and Community for Support: Given the complexity of database schema and the potential for data inconsistency or loss, it’s advisable to consult the Airbyte documentation and the Airbyte community forum for support and guidance specific to Airbyte’s database schema and best practices for data deletion.
-
Backup Before Making Changes: Always ensure you have a complete backup of your database before performing deletions, especially when manually deleting data across multiple tables. This provides a safety net in case something goes wrong.
Given the information from the extracts, there isn’t a direct reference to a comprehensive method for ensuring that all builder-related keys and data are completely deleted from every table in the Airbyte database, especially when considering dependencies and references across multiple tables. However, based on general database management practices and the insights provided, here’s a structured approach to ensure thorough deletion: 1. Identify All Related Data and Dependencies: Before deletion, it’s crucial to identify all tables and records that are related to the builder you’ve deleted. This includes not only the direct entries created by the builder but also any related data that might exist in other tables as a result of those entries. For example, if your builder created sources, destinations, or specific configurations, you’ll need to identify all tables where these entities are referenced. 2. Use Cascading Deletes If Possible: To ensure that deleting a record in one table also deletes related records in other tables, you can use cascading deletes if your database schema supports it. This is not directly mentioned in the extracts, but it’s a common database feature. A cascading delete would look something like this in SQL (note: this is a general example, not specific SQL from the extracts): sql DELETE FROM main_table WHERE condition = 'YourCondition' CASCADE; This command would delete not only the matching records in main_table but also any related records in tables that have foreign key references with a cascade delete rule set up. However, implementing cascading deletes requires that foreign key relationships are properly defined in your database schema with the ON DELETE CASCADE option. 3. Manual Cleanup for Non-Cascading Tables: If cascading deletes are not set up or if you prefer to manually control the deletion process, you’ll need to manually delete related records from each table. This involves running separate DELETE commands for each table where related data might exist, ensuring that you’re not leaving orphaned records. Here’s a hypothetical example of manually deleting related records (again, this is a generic example, not specific SQL from the extracts): sql DELETE FROM related_table WHERE builder_key = 'YourBuilderKey'; 4. Consult Documentation and Community for Support: Given the complexity of database schema and the potential for data inconsistency or loss, it’s advisable to consult the Airbyte documentation and the Airbyte community forum for support and guidance specific to Airbyte’s database schema and best practices for data deletion. 5. Backup Before Making Changes: Always ensure you have a complete backup of your database before performing deletions, especially when manually deleting data across multiple tables. This provides a safety net in case something goes wrong. Remember, direct database manipulation should be approached with caution. The steps outlined above are based on general database management practices and should be adapted to your specific database schema and the data structures used by Airbyte.
How to Disabled connection by deleting data in the database logging?
How to Disabled connection by deleting data in the database logging? <@U04SE163WC8>
Based on the information provided in the extracts, there isn’t a direct reference to disabling a connection by deleting data in the database, especially concerning logging data. However, managing connections, including disabling them, typically involves interaction with the Airbyte UI or API rather than directly manipulating logging data in the database. To disable a connection in Airbyte, you would generally follow these steps in the Airbyte UI: 1. Navigate to the Connections section of the Airbyte UI. 2. Find the connection you wish to disable. 3. Click on the … menu button for the connection. 4. Select Disable from the dropdown menu. This process does not involve directly deleting or modifying logging data in the database. Logging data is used for troubleshooting and audit purposes and does not control the operational status of connections. If you’re looking to manage the size of logging data or clean up old logs, the approach would be different and would not directly impact the enabled/disabled status of connections. The extracts do not provide specific instructions for managing or deleting logging data in the database. For tasks like cleaning up logging data or managing database size, it’s important to: - Regularly monitor the size of your logging and database storage. - Implement log rotation or cleanup policies as part of your database maintenance routines. If you’re encountering issues with database size due to logging data, consider consulting the Airbyte documentation or seeking advice from the Airbyte community forum for best practices on database and log management specific to Airbyte. Remember, direct manipulation of the database should be approached with caution and is generally not recommended for routine operations like enabling or disabling connections. Always ensure you have backups before making any changes to the database.
I can’t cancel, what should I do? <@U04SE163WC8>