- Is this your first time deploying Airbyte?: Yes
- OS Version / Instance: Ubuntu, 20.04 LTS / Google e2-standard-2
- Memory / Disk: 50Gb
- Deployment: Docker
- Airbyte Version: 0.40.10
- Source name/version: Own Source (Plenigo)
- Destination name/version: BigQuery
- Step: Normalization
Hi,
I am trying to establish a connection between our software-as-a-service platform and our Google BigQuery dataset. The connector should retrieve data via an API interface and write it to a Google BigQuery table. Without additional normalization, the data is successfully written to the _airbyte_raw_incremental_transaction_stream table. However, as soon as the normalization is activated, I get the error that the ‘properties’ field of the JSON schema cannot be processed. The records of the API interface are not modeled and the received records should match the specified schema aswell.
Here is the log file from the failed sync:
error_logs.txt (147.0 KB)
Since I have no further details, I assume that the error can be traced back to this line:
2022-10-07 08:52:33 e[42mnormalizatione[0m > KeyError: “‘json_schema’.‘properties’ are not defined for stream incremental_transaction_stream”
The JSON schema, which is stored in secrets/configure_catalog.json (source), is structured as follows:
{
"streams": [
{
"destination_sync_mode": "append",
"sync_mode": "incremental",
"stream": {
"name": "incremental_transaction_stream",
"json_schema": {
"type": "object",
"required": [
"transactionId",
"paymentProvider",
"customerId"
],
"properties": {
"transactionId": {
"type": "number"
},
"plenigoTransactionId": {
"type": "string"
},
"changedDate": {
"type": "string"
},
"transactionDate": {
"type": "string"
},
"fulfillmentDate": {
"type": "string"
},
"paymentProvider": {
"type": "string"
},
"paymentMethod": {
"type": "string"
},
"paymentAction": {
"type": "string"
},
"paymentStatus": {
"type": "string"
},
"customerId": {
"type": "string"
},
"amount": {
"type": "number"
},
"currency": {
"type": "string"
},
"pspTransactionId": {
"type": "string"
},
"description": {
"type": "string"
},
"errorMessage": {
"type": "string"
},
"errorCode": {
"type": "string"
}
}
},
"supported_sync_modes": [
"full_refresh",
"incremental"
],
"source_defined_cursor": true,
"default_cursor_field": [
"startTime"
],
"source_defined_primary_key": [
[
"transactionId"
]
]
}
}
]
}
Please could someone assist me on this issue. Thanks for your help!