Hi,
I’m experiencing problems with multiple connections failing each night. The error is rather strange and I couldn’t find anything when searching for it.
We have multiple identical sources (Paypal, Stripe and a custom source, each 3 times as we have various accounts). We use custom the same custom namespace for all three identical sources, but different prefixes.
The destination is always the same one.
2022-11-26 03:00:01 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/26/0/logs.log
2022-11-26 03:00:01 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.40.14
2022-11-26 03:00:02 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:02 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2022-11-26 03:00:02 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:02 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-paypal-transaction:0.1.10 exists...
2022-11-26 03:00:03 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-paypal-transaction:0.1.10 was found locally.
2022-11-26 03:00:03 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-paypal-transaction-check-26-0-clvns with resources io.airbyte.config.ResourceRequirements@64ee9512[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2022-11-26 03:00:03 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/26/0 --log-driver none --name source-paypal-transaction-check-26-0-clvns --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/source-paypal-transaction:0.1.10 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.14 -e WORKER_JOB_ID=26 airbyte/source-paypal-transaction:0.1.10 check --config source_config.json
2022-11-26 03:00:09 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Check succeeded
2022-11-26 03:00:09 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:09 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/26/0/logs.log
2022-11-26 03:00:09 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.40.14
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:09 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-bigquery:1.2.7 exists...
2022-11-26 03:00:10 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = destination-bigquery-check-26-0-emqlp with resources io.airbyte.config.ResourceRequirements@64ee9512[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2022-11-26 03:00:10 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-bigquery:1.2.7 was found locally.
2022-11-26 03:00:10 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/26/0 --log-driver none --name destination-bigquery-check-26-0-emqlp --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/destination-bigquery:1.2.7 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.14 -e WORKER_JOB_ID=26 airbyte/destination-bigquery:1.2.7 check --config source_config.json
2022-11-26 03:00:22 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - integration args: {check=null, config=source_config.json}
2022-11-26 03:00:22 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination
2022-11-26 03:00:22 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Command: CHECK
2022-11-26 03:00:22 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-11-26 03:00:23 e[33mWARNe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):106 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-11-26 03:00:23 e[33mWARNe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):106 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-11-26 03:00:24 e[32mINFOe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Selected loading method is set to: GCS
2022-11-26 03:00:31 e[1;31mERRORe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):105 - Check failed.
Stack Trace: com.google.cloud.bigquery.BigQueryException: Already Exists: Dataset npg-data-warehouse-prod:source_airbyte_check_stage_tmp
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:170)
at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:263)
at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:260)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:259)
at io.airbyte.integrations.destination.bigquery.BigQueryUtils.checkHasCreateAndDeleteDatasetRole(BigQueryUtils.java:130)
at io.airbyte.integrations.destination.bigquery.BigQueryDestination.check(BigQueryDestination.java:91)
at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:125)
at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:100)
at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:327)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 409 Conflict
POST https://www.googleapis.com/bigquery/v2/projects/npg-data-warehouse-prod/datasets?prettyPrint=false
{
"code" : 409,
"errors" : [ {
"domain" : "global",
"message" : "Already Exists: Dataset npg-data-warehouse-prod:source_airbyte_check_stage_tmp",
"reason" : "duplicate"
} ],
"message" : "Already Exists: Dataset npg-data-warehouse-prod:source_airbyte_check_stage_tmp",
"status" : "ALREADY_EXISTS"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:168)
... 11 more
2022-11-26 03:00:31 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-11-26 03:00:31 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:31 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2022-11-26 03:00:31 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):114 -
2022-11-26 03:00:31 e[32mINFOe[m i.a.n.SlackNotificationClient(notify):135 - Successful notification (200): ok
2022-11-26 03:00:31 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.method: must be a constant value Standard
2022-11-26 03:00:31 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.credential.hmac_key_secret: object found, string expected, $.credential.hmac_key_access_id: object found, string expected
2022-11-26 03:00:31 e[32mINFOe[m i.a.c.f.EnvVariableFeatureFlags(autoDisablesFailingConnections):19 - Auto Disable Failing Connections: false
- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Debian 11
- Memory / Disk: you can use something like 4Gb / 1 Tb
- Deployment: Docker
- Airbyte Version: 0.40.14
- Source name/version: Multiple
- Destination name/version: BigQuery with GCS staging
- Step: Sync