Issue with File (destination: postgres) Sync

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu 18.04
  • Memory / Disk: 160Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.35.30-alpha
  • Source name/version: File / 0.2.9
  • Destination name/version: Postgres / 0.4.6
  • Step: The issue is happening during sync
  • Description: I faced this issue
2022-06-01 08:35:18 INFO i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:18 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/2417/2/logs.log
2022-06-01 08:35:18 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):103 - start sync worker. job id: 2417 attempt id: 2
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):115 - configured sync modes: {null.us_p0_json_18=full_refresh - append}
2022-06-01 08:35:18 INFO i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.14 exists...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.14 was found locally.
2022-06-01 08:35:18 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/destination-postgres:0.3.14 write --config destination_config.json --catalog destination_catalog.json
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-file:0.2.9 exists...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-file:0.2.9 was found locally.
2022-06-01 08:35:18 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/source-file:0.2.9 read --config source_config.json --catalog source_catalog.json
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):157 - Waiting for source and destination threads to complete.
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):337 - Destination output thread started.
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):278 - Replication thread started.
2022-06-01 08:35:19 destination > SLF4J: Class path contains multiple SLF4J bindings.
2022-06-01 08:35:19 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.16.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-06-01 08:35:19 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-06-01 08:35:19 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-06-01 08:35:19 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-06-01 08:35:19 destination > 2022-06-01 08:35:19 INFO i.a.i.d.p.PostgresDestination(main):69 - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.b.s.SshTunnel(getInstance):172 - Starting connection with method: NO_TUNNEL
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - Write config: WriteConfig{streamName=us_p0_json_18, namespace=null, outputSchemaName=public, tmpTableName=_airbyte_tmp_btt_us_p0_json_18, outputTableName=_airbyte_raw_us_p0_json_18, syncMode=append}
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):125 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - Preparing tmp tables in destination started for 1 streams
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - Preparing tmp table in destination started for stream us_p0_json_18. schema: public, tmp table name: _airbyte_tmp_btt_us_p0_json_18
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - Preparing tables in destination completed.
2022-06-01 08:35:38 INFO i.a.w.DefaultReplicationWorker(run):162 - One of source or destination thread complete. Waiting on the other.
2022-06-01 08:35:39 destination > 2022-06-01 08:35:39 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - Airbyte message consumer: succeeded.
2022-06-01 08:35:39 destination > 2022-06-01 08:35:39 INFO i.a.i.d.b.BufferedStreamConsumer(close):201 - executing on success close procedure.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - Finalizing tables in destination started for 1 streams
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - Finalizing stream us_p0_json_18. schema public, tmp table _airbyte_tmp_btt_us_p0_json_18, final table _airbyte_raw_us_p0_json_18
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - Executing finalization of tables.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - Finalizing tables in destination completed.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - Cleaning tmp tables in destination started for 1 streams
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - Cleaning tmp table in destination started for stream us_p0_json_18. schema public, tmp table name: _airbyte_tmp_btt_us_p0_json_18
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - Cleaning tmp tables in destination completed.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.b.IntegrationRunner(run):133 - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.p.PostgresDestination(main):71 - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):164 - Source and destination threads complete.
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):227 - sync summary: io.airbyte.config.ReplicationAttemptSummary@3d8a891e[status=completed,recordsSynced=1,bytesSynced=15439811,startTime=1654072518591,endTime=1654072540913,totalStats=io.airbyte.config.SyncStats@f50a963[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=0,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@5fbf468d[streamName=us_p0_json_18,stats=io.airbyte.config.SyncStats@39c6a7d7[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=<null>,recordsCommitted=1]]]]
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):249 - Source did not output any state messages
2022-06-01 08:35:40 WARN i.a.w.DefaultReplicationWorker(run):260 - State capture: No state retained.
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-06-01 08:35:40 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):144 - sync summary: io.airbyte.config.StandardSyncOutput@26f7a82a[standardSyncSummary=io.airbyte.config.StandardSyncSummary@2a7693a8[status=completed,recordsSynced=1,bytesSynced=15439811,startTime=1654072518591,endTime=1654072540913,totalStats=io.airbyte.config.SyncStats@f50a963[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=0,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@5fbf468d[streamName=us_p0_json_18,stats=io.airbyte.config.SyncStats@39c6a7d7[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=<null>,recordsCommitted=1]]]],state=<null>,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@5eccadf6[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@7b88379a[stream=io.airbyte.protocol.models.AirbyteStream@60a57639[name=us_p0_json_18,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"type":"object","$schema":"http://json-schema.org/schema#","required":["data","meta"],"properties":{"data":{"type":"array","items":{"type":"array","items":{"type":["integer","null","string"]}}},"meta":{"type":"object","required":["view"],"properties":{"view":{"type":"object","required":["approvals","assetType","attribution","attributionLink","averageRating","category","clientContext","columns","createdAt","description","displayType","downloadCount","flags","grants","hideFromCatalog","hideFromDataJson","id","indexUpdatedAt","license","licenseId","locale","metadata","name","newBackend","numberOfComments","oid","owner","provenance","publicationAppendEnabled","publicationDate","publicationGroup","publicationStage","query","rights","rowClass","rowsUpdatedAt","rowsUpdatedBy","tableAuthor","tableId","tags","totalTimesRated","viewCount","viewLastModified","viewType"],"properties":{"id":{"type":"string"},"oid":{"type":"integer"},"name":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"flags":{"type":"array","items":{"type":"string"}},"owner":{"type":"object","required":["displayName","flags","id","profileImageUrlLarge","profileImageUrlMedium","profileImageUrlSmall","screenName","type"],"properties":{"id":{"type":"string"},"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"screenName":{"type":"string"},"displayName":{"type":"string"},"profileImageUrlLarge":{"type":"string"},"profileImageUrlSmall":{"type":"string"},"profileImageUrlMedium":{"type":"string"}}},"query":{"type":"object"},"grants":{"type":"array","items":{"type":"object","required":["flags","inherited","type"],"properties":{"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"inherited":{"type":"boolean"}}}},"locale":{"type":"string"},"rights":{"type":"array","items":{"type":"string"}},"columns":{"type":"array","items":{"type":"object","required":["dataTypeName","fieldName","format","id","name","position","renderTypeName"],"properties":{"id":{"type":"integer"},"name":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"width":{"type":"integer"},"format":{"type":"object","properties":{"align":{"type":"string"},"noCommas":{"type":"string"},"groupSeparator":{"type":"string"},"precisionStyle":{"type":"string"},"decimalSeparator":{"type":"string"}}},"position":{"type":"integer"},"fieldName":{"type":"string"},"description":{"type":"string"},"dataTypeName":{"type":"string"},"tableColumnId":{"type":"integer"},"cachedContents":{"type":"object","required":["cardinality","count","largest","non_null","null","smallest","top"],"properties":{"top":{"type":"array","items":{"type":"object","required":["count","item"],"properties":{"item":{"type":"string"},"count":{"type":"string"}}}},"null":{"type":"string"},"count":{"type":"string"},"largest":{"type":"string"},"non_null":{"type":"string"},"smallest":{"type":"string"},"cardinality":{"type":"string"}}},"renderTypeName":{"type":"string"}}}},"license":{"type":"object","required":["name","termsLink"],"properties":{"name":{"type":"string"},"termsLink":{"type":"string"}}},"tableId":{"type":"integer"},"category":{"type":"string"},"metadata":{"type":"object","required":["attachments","availableDisplayTypes","custom_fields","filterCondition","rdfClass","rdfSubject","renderTypeConfig","rowLabel"],"properties":{"rdfClass":{"type":"string"},"rowLabel":{"type":"string"},"rdfSubject":{"type":"string"},"attachments":{"type":"array","items":{"type":"object","required":["assetId","filename","name"],"properties":{"name":{"type":"string"},"assetId":{"type":"string"},"filename":{"type":"string"}}}},"custom_fields":{"type":"object","required":["Common Core","Data Quality"],"properties":{"Common Core":{"type":"object","required":["Bureau Code","Contact Email","Contact Name","Geographic Coverage","Homepage","Issued","Program Code","Public Access Level","Publisher","Update Frequency"],"properties":{"Issued":{"type":"string"},"Homepage":{"type":"string"},"Publisher":{"type":"string"},"Bureau Code":{"type":"string"},"Contact Name":{"type":"string"},"Program Code":{"type":"string"},"Contact Email":{"type":"string"},"Update Frequency":{"type":"string"},"Geographic Coverage":{"type":"string"},"Public Access Level":{"type":"string"}}},"Data Quality":{"type":"object","required":["Geographic Coverage","Suggested Citation"],"properties":{"Suggested Citation":{"type":"string"},"Geographic Coverage":{"type":"string"}}}}},"filterCondition":{"type":"object","required":["children","metadata","type","value"],"properties":{"type":{"type":"string"},"value":{"type":"string"},"children":{"type":"array","items":{"type":"object","required":["metadata","type","value"],"properties":{"type":{"type":"string"},"value":{"type":"string"},"metadata":{"type":"object","required":["customValues","operator","tableColumnId"],"properties":{"operator":{"type":"string"},"customValues":{"type":"array","items":{"type":"array","items":{"type":["integer","string"]}}},"tableColumnId":{"type":"object","required":["14919580"],"properties":{"14919580":{"type":"integer"}}}}}}}},"metadata":{"type":"object","required":["advanced","unifiedVersion"],"properties":{"advanced":{"type":"boolean"},"unifiedVersion":{"type":"integer"}}}}},"renderTypeConfig":{"type":"object","required":["visible"],"properties":{"visible":{"type":"object","required":["table"],"properties":{"table":{"type":"boolean"}}}}},"availableDisplayTypes":{"type":"array","items":{"type":"string"}}}},"rowClass":{"type":"string"},"viewType":{"type":"string"},"approvals":{"type":"array","items":{"type":"object","required":["reviewedAt","reviewedAutomatically","state","submissionDetails","submissionId","submissionObject","submissionOutcome","submissionOutcomeApplication","submittedAt","submitter","workflowId"],"properties":{"state":{"type":"string"},"submitter":{"type":"object","required":["displayName","id"],"properties":{"id":{"type":"string"},"displayName":{"type":"string"}}},"reviewedAt":{"type":"integer"},"workflowId":{"type":"integer"},"submittedAt":{"type":"integer"},"submissionId":{"type":"integer"},"submissionObject":{"type":"string"},"submissionDetails":{"type":"object","required":["permissionType"],"properties":{"permissionType":{"type":"string"}}},"submissionOutcome":{"type":"string"},"reviewedAutomatically":{"type":"boolean"},"submissionOutcomeApplication":{"type":"object","required":["failureCount","status"],"properties":{"status":{"type":"string"},"failureCount":{"type":"integer"}}}}}},"assetType":{"type":"string"},"createdAt":{"type":"integer"},"licenseId":{"type":"string"},"viewCount":{"type":"integer"},"newBackend":{"type":"boolean"},"provenance":{"type":"string"},"attribution":{"type":"string"},"description":{"type":"string"},"displayType":{"type":"string"},"tableAuthor":{"type":"object","required":["displayName","flags","id","profileImageUrlLarge","profileImageUrlMedium","profileImageUrlSmall","screenName","type"],"properties":{"id":{"type":"string"},"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"screenName":{"type":"string"},"displayName":{"type":"string"},"profileImageUrlLarge":{"type":"string"},"profileImageUrlSmall":{"type":"string"},"profileImageUrlMedium":{"type":"string"}}},"averageRating":{"type":"integer"},"clientContext":{"type":"object","required":["clientContextVariables","inheritedVariables"],"properties":{"inheritedVariables":{"type":"object"},"clientContextVariables":{"type":"array"}}},"downloadCount":{"type":"integer"},"rowsUpdatedAt":{"type":"integer"},"rowsUpdatedBy":{"type":"string"},"indexUpdatedAt":{"type":"integer"},"attributionLink":{"type":"string"},"hideFromCatalog":{"type":"boolean"},"publicationDate":{"type":"integer"},"totalTimesRated":{"type":"integer"},"hideFromDataJson":{"type":"boolean"},"numberOfComments":{"type":"integer"},"publicationGroup":{"type":"integer"},"publicationStage":{"type":"string"},"viewLastModified":{"type":"integer"},"publicationAppendEnabled":{"type":"boolean"}}}}}}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]]
2022-06-01 08:35:40 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):234 - Stopping temporal heartbeating...
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/2417/2/logs.log
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:40 INFO i.a.w.DefaultNormalizationWorker(run):46 - Running normalization.
2022-06-01 08:35:40 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.66
2022-06-01 08:35:40 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.66 exists...
2022-06-01 08:35:41 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/normalization:0.1.66 was found locally.
2022-06-01 08:35:41 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2/normalize --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/normalization:0.1.66 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
2022-06-01 08:35:41 normalization > Running: transform-config --config destination_config.json --integration-type postgres --out /data/2417/2/normalize
2022-06-01 08:35:41 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/2417/2/normalize')
2022-06-01 08:35:41 normalization > transform_postgres
2022-06-01 08:35:41 normalization > Running: transform-catalog --integration-type postgres --profile-config-dir /data/2417/2/normalize --catalog destination_catalog.json --out /data/2417/2/normalize/models/generated/ --json-column _airbyte_data
2022-06-01 08:35:41 normalization > Processing destination_catalog.json...
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_format (#44) to us_p0_json_18_proper_s_view_columns_format (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_cachedContents (#52) to us_p0_json_18_proper_olumns_cachedContents (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_cachedContents_top (#56) to us_p0_json_18_proper_ns_cachedContents_top (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_attachments (#50) to us_p0_json_18_proper__metadata_attachments (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields (#52) to us_p0_json_18_proper_etadata_custom_fields (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields_Common_Core (#64) to us_p0_json_18_proper_om_fields_Common_Core (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields_Data_Quality (#65) to us_p0_json_18_proper_m_fields_Data_Quality (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition (#54) to us_p0_json_18_proper_adata_filterCondition (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children (#63) to us_p0_json_18_proper_terCondition_children (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata (#72) to us_p0_json_18_proper_ion_children_metadata (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata_customValues (#85) to us_p0_json_18_proper_metadata_customValues (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata_tableColumnId (#86) to us_p0_json_18_proper_etadata_tableColumnId (#43)
2022-06-01 08:35:41 normalization > Traceback (most recent call last):
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_metadata (#63) to us_p0_json_18_proper_terCondition_metadata (#43)
2022-06-01 08:35:41 normalization >   File "/usr/local/bin/transform-catalog", line 8, in <module>
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_renderTypeConfig (#55) to us_p0_json_18_proper_data_renderTypeConfig (#43)
2022-06-01 08:35:41 normalization >     sys.exit(main())
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_renderTypeConfig_visible (#63) to us_p0_json_18_proper_derTypeConfig_visible (#43)
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 82, in main
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_availableDisplayTypes (#60) to us_p0_json_18_proper_availableDisplayTypes (#43)
2022-06-01 08:35:41 normalization >     TransformCatalog().run(args)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submitter (#49) to us_p0_json_18_proper_w_approvals_submitter (#43)
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 35, in run
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submissionDetails (#57) to us_p0_json_18_proper_als_submissionDetails (#43)
2022-06-01 08:35:41 normalization >     self.process_catalog()
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submissionOutcomeApplication (#68) to us_p0_json_18_proper_ionOutcomeApplication (#43)
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 62, in process_catalog
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_tableAuthor_flags (#47) to us_p0_json_18_proper_iew_tableAuthor_flags (#43)
2022-06-01 08:35:41 normalization >     processor.process(catalog_file=catalog_file, json_column_name=json_col, default_schema=schema)
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/catalog_processor.py", line 63, in process
2022-06-01 08:35:41 normalization >     for conflict in tables_registry.resolve_names():
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 157, in resolve_names
2022-06-01 08:35:41 normalization >     conflicts = self.resolve_table_names()
2022-06-01 08:35:41 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 196, in resolve_table_names
2022-06-01 08:35:41 normalization >     assert (table_count * 2) == registry_size, f"Mismatched number of tables {table_count * 2} vs {registry_size} being resolved"
2022-06-01 08:35:41 normalization > AssertionError: Mismatched number of tables 76 vs 74 being resolved

...
discover_schema result
...

	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	... 3 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:159) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]
2022-06-01 08:35:41 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-06-01 08:35:41 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):234 - Stopping temporal heartbeating...
2022-06-01 08:35:41 WARN i.t.i.s.POJOActivityTaskHandler(activityFailureToResult):363 - Activity failure. ActivityId=778c0fb1-7098-3e6d-ae31-f07bf524c167, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:232) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]

at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:71) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:286) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:252) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:209) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:193) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.6.0.jar:?]
	at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:412) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.6.0.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:227) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	... 13 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:129) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:227) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	... 13 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:60) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	... 1 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
	... 1 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:159) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]

Why this issue continues to occur when discover_schema works well?

AssertionError: Mismatched number of tables 64 vs 62 being resolved

Hey can you share the complete logs file

logs.log (99.0 KB)

hey this error is happening during the normalisation. Can you check if disabling normalisation is making the sync successful also with the right tables creation and data?

It has already been checked that disabling normalization works well. However, this normalization issue still seems to need to be analyzed.

Hey digging bit more deep: Is it possible that you have the same table names? Could you share a screenshot of the discover step

There is no possibility that a table with the same name exists. I was trying to create a continuous connection (JSON files to postgres) for about 100 sources.

discover_schema results:

{
    "catalog": {
        "streams": [
            {
                "stream": {
                    "name": "us_p0_json_18",
                    "jsonSchema": {
                        "$schema": "http://json-schema.org/draft-07/schema#",
                        "type": "object",
                        "properties": {
                            "$schema": "http://json-schema.org/schema#",
                            "type": "object",
                            "properties": {
                                "meta": {
                                    "type": "object",
                                    "properties": {
                                        "view": {
                                            "type": "object",
                                            "properties": {
                                                "id": {
                                                    "type": "string"
                                                },
                                                "name": {
                                                    "type": "string"
                                                },
                                                "assetType": {
                                                    "type": "string"
                                                },
                                                "attribution": {
                                                    "type": "string"
                                                },
                                                "attributionLink": {
                                                    "type": "string"
                                                },
                                                "averageRating": {
                                                    "type": "integer"
                                                },
                                                "category": {
                                                    "type": "string"
                                                },
                                                "createdAt": {
                                                    "type": "integer"
                                                },
                                                "description": {
                                                    "type": "string"
                                                },
                                                "displayType": {
                                                    "type": "string"
                                                },
                                                "downloadCount": {
                                                    "type": "integer"
                                                },
                                                "hideFromCatalog": {
                                                    "type": "boolean"
                                                },
                                                "hideFromDataJson": {
                                                    "type": "boolean"
                                                },
                                                "indexUpdatedAt": {
                                                    "type": "integer"
                                                },
                                                "licenseId": {
                                                    "type": "string"
                                                },
                                                "locale": {
                                                    "type": "string"
                                                },
                                                "newBackend": {
                                                    "type": "boolean"
                                                },
                                                "numberOfComments": {
                                                    "type": "integer"
                                                },
                                                "oid": {
                                                    "type": "integer"
                                                },
                                                "provenance": {
                                                    "type": "string"
                                                },
                                                "publicationAppendEnabled": {
                                                    "type": "boolean"
                                                },
                                                "publicationDate": {
                                                    "type": "integer"
                                                },
                                                "publicationGroup": {
                                                    "type": "integer"
                                                },
                                                "publicationStage": {
                                                    "type": "string"
                                                },
                                                "rowClass": {
                                                    "type": "string"
                                                },
                                                "rowsUpdatedAt": {
                                                    "type": "integer"
                                                },
                                                "rowsUpdatedBy": {
                                                    "type": "string"
                                                },
                                                "tableId": {
                                                    "type": "integer"
                                                },
                                                "totalTimesRated": {
                                                    "type": "integer"
                                                },
                                                "viewCount": {
                                                    "type": "integer"
                                                },
                                                "viewLastModified": {
                                                    "type": "integer"
                                                },
                                                "viewType": {
                                                    "type": "string"
                                                },
                                                "approvals": {
                                                    "type": "array",
                                                    "items": {
                                                        "type": "object",
                                                        "properties": {
                                                            "reviewedAt": {
                                                                "type": "integer"
                                                            },
                                                            "reviewedAutomatically": {
                                                                "type": "boolean"
                                                            },
                                                            "state": {
                                                                "type": "string"
                                                            },
                                                            "submissionId": {
                                                                "type": "integer"
                                                            },
                                                            "submissionObject": {
                                                                "type": "string"
                                                            },
                                                            "submissionOutcome": {
                                                                "type": "string"
                                                            },
                                                            "submittedAt": {
                                                                "type": "integer"
                                                            },
                                                            "workflowId": {
                                                                "type": "integer"
                                                            },
                                                            "submissionDetails": {
                                                                "type": "object",
                                                                "properties": {
                                                                    "permissionType": {
                                                                        "type": "string"
                                                                    }
                                                                },
                                                                "required": [
                                                                    "permissionType"
                                                                ]
                                                            }, 
. . .

The result exceeded the text limit…

Hey have created an issue around this https://github.com/airbytehq/airbyte/issues/13787. Team will look into it