Failure to import

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 64Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.21

Exporting configuration from production instance and unable to import configuration into a fresh local instance.

airbyte-server | java.nio.file.NoSuchFileException: /tmp/airbyte_archive10113831242964390020/VERSION

but attaching logs nonetheless

failed_import_logs.txt (21.7 KB)

2

Hey @danieldiamond,
I’m not able to reproduce it with our latest version (0.39.28). Could you please upgrade and let me know if you still face this problem?
From which version did you create the export archive?

3

I can try the latest version but I’d be surprised if this is an issue that’s been resolved in the last few releases. Fresh local instance also built with same version as prod

4

I’m also running on a fresh build of the master branch and can’t reproduce :frowning: did you upload the .gz file directly?

5

Yes. I wonder if there are legacy resources or connections or catalogs still stored in the prod DB that are causing it to fail.
I’ll try again with latest version later today.

6

Just tried on latest airbyte version

airbyte-server      | 2022-06-29 04:32:50 ERROR i.a.s.h.ArchiveHandler(importInternal):140 - Import failed
airbyte-server      | org.jooq.exception.DataAccessException: SQL [insert into "public"."connection" ("id", "namespace_definition", "namespace_format", "prefix", "source_id", "destination_id", "name", "catalog", "status", "schedule", "manual", "resource_requirements", "source_catalog_id", "created_at", "updated_at") values (cast(? as uuid), ?::"public"."namespace_definition_type", ?, ?, cast(? as uuid), cast(? as uuid), ?, cast(? as jsonb), ?::"public"."status_type", cast(? as jsonb), ?, cast(? as jsonb), cast(? as uuid), cast(? as timestamp with time zone), cast(? as timestamp with time zone))]; ERROR: insert or update on table "connection" violates foreign key constraint "connection_actor_catalog_id_fk"
airbyte-server      |   Detail: Key (source_catalog_id)=(a6731763-9b3b-41c3-9483-314fa40c4134) is not present in table "actor_catalog".
7

I’ve also tried stripping down the import configuration folder airbyte_config to only one soure, one destination, one sync state etc. and it still fails. back to the java.nio.file.NoSuchFileException: /tmp/airbyte_archive1865273253897889206/VERSION error.

this seems quite alarming as we regularly back up our instance with this export config. if we ever need to import it, it currently seems like that’s not possible :grimacing:

8

searching /tmp/airbyte_archive on the slack workspace brings up a few conversations related to this issue.

  1. Slack → although i’ve tried deleting those ACTOR directories and still have the same issue
  2. Slack → more likely i.e. some rogue files in the DB maybe? i’m not sure
9

Hey Daniel,
Would you mind sharing this export (with fake secrets) in a private message so I can try to reproduce it?
I’ll also try to downgrade my local Airbyte to 0.39.21, run an export and then upgrade to the latest version.
It indeed looks like a schema change in the connection table is the root cause of the error. As a workaround you can try to insert a value in the actor_catalog table with the catalog id that is missing.

10

I got the same error when trying to export from v0.39.28 and import to v0.39.28.

11

I’m glad I’m now able to reproduce!
Here’s what I did:

  • Started from a fresh 0.39.28 instance, after having deleted the whole database I locally had.
  • Created a connection (between Poke API and Postgres)
  • Exported my configurations
  • Deleted my previously created connection, source, destination
  • Import my exported archive

This led to the following error (the same one you had):

2022-07-01T07:22:47.764349930Z org.jooq.exception.DataAccessException: SQL [insert into "public"."connection" ("id", "namespace_definition", "namespace_format", "prefix", "source_id", "destination_id", "name", "catalog", "status", "schedule", "manual", "resource_requirements", "source_catalog_id", "created_at", "updated_at") values (cast(? as uuid), ?::"public"."namespace_definition_type", ?, ?, cast(? as uuid), cast(? as uuid), ?, cast(? as jsonb), ?::"public"."status_type", cast(? as jsonb), ?, cast(? as jsonb), cast(? as uuid), cast(? as timestamp with time zone), cast(? as timestamp with time zone))]; ERROR: insert or update on table "connection" violates foreign key constraint "connection_actor_catalog_id_fk"2022-07-01T07:22:47.764360180Z Detail: Key (source_catalog_id)=(92cc8709-fd35-491c-aac8-0cd2b75d3d3c) is not present in table "actor_catalog".

Nothing gets imported.
I’m opening an issue right now on our repo. I’ll send the link here, please subscribe to it to receive updates.

2 Likes
12

An issue already existed: Import configuration fails with foreign key constraint error · Issue #13498 · airbytehq/airbyte · GitHub
Please track for its resolution on GitHub and comment there if you found a workaround.

1 Like
13

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.