I get below error while loading data into bigquery using GCS bucket

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte: Yes
  • OS Version / Instance: Windows
  • Memory / Disk: 16Gb / 1Tb SSD
  • Deployment: Docker
  • Airbyte Version: 0.26.2-alpha
  • Source name/version: File 0.24
  • Destination name/version: big query
  • Step: Setting new connection, source / On sync
  • Description: I’m trying to sync for the first time and the process doesn’t finish. I had enabled CDC and other cool features.

errors: $.credential: is not defined in the schema and the schema does not allow additional properties, $.part_size_mb: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, $.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, $.method: must be a constant value Standard

1 Like
2

Please upload the complete log file to us help you with your problem.

3

logs-9.txt (265.9 KB)
Please check log file.

Source: Klaviyo
Target: Big query
Target connection successful
While loading using GCS gives below error

4

Normal SQL inserts are working but are very slow

5

able to push file to GCS but while loading to big query it gives same error. Frustated :frowning:

2022-04-07 08:17:17 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.

6

We are also experiencing the same issue, BQ destination works (slowly) without load staging via GCS

7

FYI i’m getting the same error with MySQL source to Snowflake (CDC incremental + dedupe, and when it’s reset, the first sync fails with this error).
EDIT: this works after resetting (possibly with SCD existing and not requiring a full-refresh)

8

The error from logs are:

2022-04-07 03:38:05 e[43mdestinatione[0m > Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: The provided security credentials are not valid. (Service: Amazon S3; Status Code: 403; Error Code: InvalidSecurity; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null2022-04-07 03:38:05 e[43mdestinatione[0m > at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)2022-04-07 03:38:05 e[43mdestinatione[0m > at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)2022-04-07 03:38:05 e[43mdestinatione[0m > at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)

Pranit if you disable the GCS the connection works? Maybe there is something wrong with your creds.

9

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.