Hi all, I’m trying to upgrade our airbyte deployment from 0.40.5 to 0.44.12, but airbyte boot loader fails migration with the below error. For context, I’ve updated the docker compose file and .env to match new versions (triple checked that everything’s right) and airbyte is running on an ec2 instance, connected to a rds db. Tested the process locally, and works, the only difference being that I was using a docker db rather than our prod db.
023-06-16 08:05:03 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [ec2, cloud]
2023-06-16 08:05:05 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 3079ms. Server Running: http://90fc8a7c223c:9002
2023-06-16 08:05:06 INFO i.a.f.ConfigFileClient(<init>):94 - path /flags does not exist, will return default flag values
2023-06-16 08:05:06 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LOCAL_CONNECTOR_CATALOG_PATH: 'seed/oss_registry.json'
2023-06-16 08:05:06 INFO i.a.b.Bootloader(load):100 - Initializing databases...
2023-06-16 08:05:06 INFO i.a.b.Bootloader(initializeDatabases):199 - Initializing databases...
2023-06-16 08:05:06 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2023-06-16 08:05:06 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2023-06-16 08:05:07 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2023-06-16 08:05:07 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte configs database is initialized
2023-06-16 08:05:07 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2023-06-16 08:05:07 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection...
2023-06-16 08:05:08 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2023-06-16 08:05:08 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte jobs database is initialized
2023-06-16 08:05:08 INFO i.a.b.Bootloader(initializeDatabases):202 - Databases initialized.
2023-06-16 08:05:08 INFO i.a.b.Bootloader(load):103 - Checking migration compatibility...
2023-06-16 08:05:08 INFO i.a.b.Bootloader(assertNonBreakingMigration):135 - Checking for illegal upgrade...
2023-06-16 08:05:08 INFO i.a.b.Bootloader(isLegalUpgrade):213 - Current Airbyte version: AirbyteVersion{version='0.40.5', major='0', minor='40', patch='5'}
2023-06-16 08:05:08 INFO i.a.b.Bootloader(isLegalUpgrade):214 - Future Airbyte version: AirbyteVersion{version='0.44.12', major='0', minor='44', patch='12'}
2023-06-16 08:05:08 INFO i.a.b.Bootloader(load):106 - Checking protocol version constraints...
2023-06-16 08:05:08 INFO i.a.b.ProtocolVersionChecker(validate):87 - Using AirbyteProtocolVersion range [0.0.0:0.3.0]
2023-06-16 08:05:08 INFO i.a.b.Bootloader(trackProtocolVersion):242 - AirbyteProtocol version support range: [0.0.0:0.3.0]
2023-06-16 08:05:08 INFO i.a.b.Bootloader(load):109 - Running database migrations...
2023-06-16 08:05:08 INFO i.a.b.Bootloader(runFlywayMigration):223 - Creating baseline for config database...
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://aurora-airbyte-prod.cluster-cpkrm5gabev9.eu-west-1.rds.amazonaws.com:5432/airbyte_db (PostgreSQL 11.16)
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_configs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping.
2023-06-16 08:05:08 INFO i.a.b.Bootloader(runFlywayMigration):225 - Creating baseline for job database...
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://aurora-airbyte-prod.cluster-cpkrm5gabev9.eu-west-1.rds.amazonaws.com:5432/airbyte_db (PostgreSQL 11.16)
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_jobs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping.
2023-06-16 08:05:08 INFO i.a.b.Bootloader(runFlywayMigration):229 - Migrating configs database...
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 48 migrations (execution time 00:00.054s)
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.11.002
2023-06-16 08:05:08 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Migrating schema "public" to version "0.40.12.001 - AddWebhookOperationColumns"
2023-06-16 08:05:08 INFO i.a.d.i.c.m.V0_40_12_001__AddWebhookOperationColumns(migrate):25 - Running migration: V0_40_12_001__AddWebhookOperationColumns
2023-06-16 08:05:08 ERROR o.f.c.i.l.s.Slf4jLog(error):57 - Migration of schema "public" to version "0.40.12.001 - AddWebhookOperationColumns" failed! Changes successfully rolled back.
2023-06-16 08:05:08 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration failed !
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:378) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:271) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:66) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:270) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:243) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:141) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:70) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:141) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:141) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:101) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.Flyway$1.execute(Flyway.java:217) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.Flyway$1.execute(Flyway.java:168) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.Flyway.execute(Flyway.java:584) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.Flyway.migrate(Flyway.java:168) ~[flyway-core-7.14.0.jar:?]
at io.airbyte.db.instance.FlywayDatabaseMigrator.migrate(FlywayDatabaseMigrator.java:45) ~[io.airbyte.airbyte-db-db-lib-0.44.12.jar:?]
at io.airbyte.bootloader.Bootloader.runFlywayMigration(Bootloader.java:230) ~[io.airbyte-airbyte-bootloader-0.44.12.jar:?]
at io.airbyte.bootloader.Bootloader.load(Bootloader.java:110) ~[io.airbyte-airbyte-bootloader-0.44.12.jar:?]
at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.44.12.jar:?]
Caused by: org.flywaydb.core.api.FlywayException: Migration failed !
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:75) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
... 18 more
Caused by: org.jooq.exception.DataAccessException: SQL [alter type "operator_type" add value 'webhook']; ERROR: ALTER TYPE ... ADD cannot run inside a transaction block
at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:678) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:355) ~[jooq-3.17.8.jar:?]
at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.addWebhookOperationType(V0_40_12_001__AddWebhookOperationColumns.java:45) ~[io.airbyte.airbyte-db-db-lib-0.44.12.jar:?]
at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.migrate(V0_40_12_001__AddWebhookOperationColumns.java:32) ~[io.airbyte.airbyte-db-db-lib-0.44.12.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:61) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
... 18 more
Caused by: org.postgresql.util.PSQLException: ERROR: ALTER TYPE ... ADD cannot run inside a transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290) ~[postgresql-42.5.4.jar:42.5.4]
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) ~[HikariCP-5.0.1.jar:?]
at org.jooq.tools.jdbc.DefaultStatement.execute(DefaultStatement.java:102) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.SettingsEnabledPreparedStatement.execute(SettingsEnabledPreparedStatement.java:227) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:420) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:341) ~[jooq-3.17.8.jar:?]
at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.addWebhookOperationType(V0_40_12_001__AddWebhookOperationColumns.java:45) ~[io.airbyte.airbyte-db-db-lib-0.44.12.jar:?]
at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.migrate(V0_40_12_001__AddWebhookOperationColumns.java:32) ~[io.airbyte.airbyte-db-db-lib-0.44.12.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:61) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
... 18 more
Could anyone advise what’s going wrong here?