Hi, we have made a custom connector for the source and we’re pushing the data to Postgres.
So for every API call, we pass this form id and it pulls that in JSON and pushes it to Postgres.
this is the source file here
I have a JSON blob that gets created at the end in the postgres. But I want to understand why basic normalization is not working in this case. I’m getting an error something like this. I really can’t get much from this error. Can someone explain this?
2022-11-02 12:49:37 normalization > Running: transform-config --config destination_config.json --integration-type postgres --out /data/195/2/normalize
2022-11-02 12:49:40 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.POSTGRES: 'postgres'>, out='/data/195/2/normalize')
2022-11-02 12:49:40 normalization > transform_postgres
2022-11-02 12:49:40 normalization > Running: transform-catalog --integration-type postgres --profile-config-dir /data/195/2/normalize --catalog destination_catalog.json --out /data/195/2/normalize/models/generated/ --json-column _airbyte_data
2022-11-02 12:49:43 normalization > Processing destination_catalog.json...
2022-11-02 12:49:43 normalization > Generating airbyte_ctes/public/baseline_ig_ab1.sql from baseline_ig
2022-11-02 12:49:43 normalization > WARN: Unknown type for column properties at baseline_ig
2022-11-02 12:49:43 normalization > Generating airbyte_ctes/public/baseline_ig_ab2.sql from baseline_ig
2022-11-02 12:49:43 normalization > Generating airbyte_incremental/public/baseline_ig_stg.sql from baseline_ig
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2022-11-02 12:49:43 normalization > Showing destination_catalog.json to diagnose/debug errors (1):
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2022-11-02 12:49:43 normalization > {
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "streams"
2022-11-02 12:49:43 normalization > {
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "stream"
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "name"
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "json_schema"
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "type"
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "$schema"
2022-11-02 12:49:43 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - "properties"