- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu 18.04
- Memory / Disk: 160Gb / 1 Tb
- Deployment: Docker
- Airbyte Version: 0.35.30-alpha
- Source name/version: File / 0.2.9
- Destination name/version: Postgres / 0.4.6
- Step: The issue is happening during sync
- Description: I faced this issue
2022-06-01 08:35:18 INFO i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:18 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/2417/2/logs.log
2022-06-01 08:35:18 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):103 - start sync worker. job id: 2417 attempt id: 2
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):115 - configured sync modes: {null.us_p0_json_18=full_refresh - append}
2022-06-01 08:35:18 INFO i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.14 exists...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.14 was found locally.
2022-06-01 08:35:18 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/destination-postgres:0.3.14 write --config destination_config.json --catalog destination_catalog.json
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-file:0.2.9 exists...
2022-06-01 08:35:18 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-file:0.2.9 was found locally.
2022-06-01 08:35:18 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/source-file:0.2.9 read --config source_config.json --catalog source_catalog.json
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(run):157 - Waiting for source and destination threads to complete.
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):337 - Destination output thread started.
2022-06-01 08:35:18 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):278 - Replication thread started.
2022-06-01 08:35:19 destination > SLF4J: Class path contains multiple SLF4J bindings.
2022-06-01 08:35:19 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.16.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-06-01 08:35:19 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-06-01 08:35:19 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-06-01 08:35:19 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-06-01 08:35:19 destination > 2022-06-01 08:35:19 INFO i.a.i.d.p.PostgresDestination(main):69 - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.b.s.SshTunnel(getInstance):172 - Starting connection with method: NO_TUNNEL
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - Write config: WriteConfig{streamName=us_p0_json_18, namespace=null, outputSchemaName=public, tmpTableName=_airbyte_tmp_btt_us_p0_json_18, outputTableName=_airbyte_raw_us_p0_json_18, syncMode=append}
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):125 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - Preparing tmp tables in destination started for 1 streams
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - Preparing tmp table in destination started for stream us_p0_json_18. schema: public, tmp table name: _airbyte_tmp_btt_us_p0_json_18
2022-06-01 08:35:20 destination > 2022-06-01 08:35:20 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - Preparing tables in destination completed.
2022-06-01 08:35:38 INFO i.a.w.DefaultReplicationWorker(run):162 - One of source or destination thread complete. Waiting on the other.
2022-06-01 08:35:39 destination > 2022-06-01 08:35:39 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - Airbyte message consumer: succeeded.
2022-06-01 08:35:39 destination > 2022-06-01 08:35:39 INFO i.a.i.d.b.BufferedStreamConsumer(close):201 - executing on success close procedure.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - Finalizing tables in destination started for 1 streams
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - Finalizing stream us_p0_json_18. schema public, tmp table _airbyte_tmp_btt_us_p0_json_18, final table _airbyte_raw_us_p0_json_18
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - Executing finalization of tables.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - Finalizing tables in destination completed.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - Cleaning tmp tables in destination started for 1 streams
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - Cleaning tmp table in destination started for stream us_p0_json_18. schema public, tmp table name: _airbyte_tmp_btt_us_p0_json_18
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - Cleaning tmp tables in destination completed.
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.b.IntegrationRunner(run):133 - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2022-06-01 08:35:40 destination > 2022-06-01 08:35:40 INFO i.a.i.d.p.PostgresDestination(main):71 - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):164 - Source and destination threads complete.
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):227 - sync summary: io.airbyte.config.ReplicationAttemptSummary@3d8a891e[status=completed,recordsSynced=1,bytesSynced=15439811,startTime=1654072518591,endTime=1654072540913,totalStats=io.airbyte.config.SyncStats@f50a963[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=0,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@5fbf468d[streamName=us_p0_json_18,stats=io.airbyte.config.SyncStats@39c6a7d7[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=<null>,recordsCommitted=1]]]]
2022-06-01 08:35:40 INFO i.a.w.DefaultReplicationWorker(run):249 - Source did not output any state messages
2022-06-01 08:35:40 WARN i.a.w.DefaultReplicationWorker(run):260 - State capture: No state retained.
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-06-01 08:35:40 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):144 - sync summary: io.airbyte.config.StandardSyncOutput@26f7a82a[standardSyncSummary=io.airbyte.config.StandardSyncSummary@2a7693a8[status=completed,recordsSynced=1,bytesSynced=15439811,startTime=1654072518591,endTime=1654072540913,totalStats=io.airbyte.config.SyncStats@f50a963[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=0,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@5fbf468d[streamName=us_p0_json_18,stats=io.airbyte.config.SyncStats@39c6a7d7[recordsEmitted=1,bytesEmitted=15439811,stateMessagesEmitted=<null>,recordsCommitted=1]]]],state=<null>,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@5eccadf6[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@7b88379a[stream=io.airbyte.protocol.models.AirbyteStream@60a57639[name=us_p0_json_18,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"type":"object","$schema":"http://json-schema.org/schema#","required":["data","meta"],"properties":{"data":{"type":"array","items":{"type":"array","items":{"type":["integer","null","string"]}}},"meta":{"type":"object","required":["view"],"properties":{"view":{"type":"object","required":["approvals","assetType","attribution","attributionLink","averageRating","category","clientContext","columns","createdAt","description","displayType","downloadCount","flags","grants","hideFromCatalog","hideFromDataJson","id","indexUpdatedAt","license","licenseId","locale","metadata","name","newBackend","numberOfComments","oid","owner","provenance","publicationAppendEnabled","publicationDate","publicationGroup","publicationStage","query","rights","rowClass","rowsUpdatedAt","rowsUpdatedBy","tableAuthor","tableId","tags","totalTimesRated","viewCount","viewLastModified","viewType"],"properties":{"id":{"type":"string"},"oid":{"type":"integer"},"name":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"flags":{"type":"array","items":{"type":"string"}},"owner":{"type":"object","required":["displayName","flags","id","profileImageUrlLarge","profileImageUrlMedium","profileImageUrlSmall","screenName","type"],"properties":{"id":{"type":"string"},"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"screenName":{"type":"string"},"displayName":{"type":"string"},"profileImageUrlLarge":{"type":"string"},"profileImageUrlSmall":{"type":"string"},"profileImageUrlMedium":{"type":"string"}}},"query":{"type":"object"},"grants":{"type":"array","items":{"type":"object","required":["flags","inherited","type"],"properties":{"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"inherited":{"type":"boolean"}}}},"locale":{"type":"string"},"rights":{"type":"array","items":{"type":"string"}},"columns":{"type":"array","items":{"type":"object","required":["dataTypeName","fieldName","format","id","name","position","renderTypeName"],"properties":{"id":{"type":"integer"},"name":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"width":{"type":"integer"},"format":{"type":"object","properties":{"align":{"type":"string"},"noCommas":{"type":"string"},"groupSeparator":{"type":"string"},"precisionStyle":{"type":"string"},"decimalSeparator":{"type":"string"}}},"position":{"type":"integer"},"fieldName":{"type":"string"},"description":{"type":"string"},"dataTypeName":{"type":"string"},"tableColumnId":{"type":"integer"},"cachedContents":{"type":"object","required":["cardinality","count","largest","non_null","null","smallest","top"],"properties":{"top":{"type":"array","items":{"type":"object","required":["count","item"],"properties":{"item":{"type":"string"},"count":{"type":"string"}}}},"null":{"type":"string"},"count":{"type":"string"},"largest":{"type":"string"},"non_null":{"type":"string"},"smallest":{"type":"string"},"cardinality":{"type":"string"}}},"renderTypeName":{"type":"string"}}}},"license":{"type":"object","required":["name","termsLink"],"properties":{"name":{"type":"string"},"termsLink":{"type":"string"}}},"tableId":{"type":"integer"},"category":{"type":"string"},"metadata":{"type":"object","required":["attachments","availableDisplayTypes","custom_fields","filterCondition","rdfClass","rdfSubject","renderTypeConfig","rowLabel"],"properties":{"rdfClass":{"type":"string"},"rowLabel":{"type":"string"},"rdfSubject":{"type":"string"},"attachments":{"type":"array","items":{"type":"object","required":["assetId","filename","name"],"properties":{"name":{"type":"string"},"assetId":{"type":"string"},"filename":{"type":"string"}}}},"custom_fields":{"type":"object","required":["Common Core","Data Quality"],"properties":{"Common Core":{"type":"object","required":["Bureau Code","Contact Email","Contact Name","Geographic Coverage","Homepage","Issued","Program Code","Public Access Level","Publisher","Update Frequency"],"properties":{"Issued":{"type":"string"},"Homepage":{"type":"string"},"Publisher":{"type":"string"},"Bureau Code":{"type":"string"},"Contact Name":{"type":"string"},"Program Code":{"type":"string"},"Contact Email":{"type":"string"},"Update Frequency":{"type":"string"},"Geographic Coverage":{"type":"string"},"Public Access Level":{"type":"string"}}},"Data Quality":{"type":"object","required":["Geographic Coverage","Suggested Citation"],"properties":{"Suggested Citation":{"type":"string"},"Geographic Coverage":{"type":"string"}}}}},"filterCondition":{"type":"object","required":["children","metadata","type","value"],"properties":{"type":{"type":"string"},"value":{"type":"string"},"children":{"type":"array","items":{"type":"object","required":["metadata","type","value"],"properties":{"type":{"type":"string"},"value":{"type":"string"},"metadata":{"type":"object","required":["customValues","operator","tableColumnId"],"properties":{"operator":{"type":"string"},"customValues":{"type":"array","items":{"type":"array","items":{"type":["integer","string"]}}},"tableColumnId":{"type":"object","required":["14919580"],"properties":{"14919580":{"type":"integer"}}}}}}}},"metadata":{"type":"object","required":["advanced","unifiedVersion"],"properties":{"advanced":{"type":"boolean"},"unifiedVersion":{"type":"integer"}}}}},"renderTypeConfig":{"type":"object","required":["visible"],"properties":{"visible":{"type":"object","required":["table"],"properties":{"table":{"type":"boolean"}}}}},"availableDisplayTypes":{"type":"array","items":{"type":"string"}}}},"rowClass":{"type":"string"},"viewType":{"type":"string"},"approvals":{"type":"array","items":{"type":"object","required":["reviewedAt","reviewedAutomatically","state","submissionDetails","submissionId","submissionObject","submissionOutcome","submissionOutcomeApplication","submittedAt","submitter","workflowId"],"properties":{"state":{"type":"string"},"submitter":{"type":"object","required":["displayName","id"],"properties":{"id":{"type":"string"},"displayName":{"type":"string"}}},"reviewedAt":{"type":"integer"},"workflowId":{"type":"integer"},"submittedAt":{"type":"integer"},"submissionId":{"type":"integer"},"submissionObject":{"type":"string"},"submissionDetails":{"type":"object","required":["permissionType"],"properties":{"permissionType":{"type":"string"}}},"submissionOutcome":{"type":"string"},"reviewedAutomatically":{"type":"boolean"},"submissionOutcomeApplication":{"type":"object","required":["failureCount","status"],"properties":{"status":{"type":"string"},"failureCount":{"type":"integer"}}}}}},"assetType":{"type":"string"},"createdAt":{"type":"integer"},"licenseId":{"type":"string"},"viewCount":{"type":"integer"},"newBackend":{"type":"boolean"},"provenance":{"type":"string"},"attribution":{"type":"string"},"description":{"type":"string"},"displayType":{"type":"string"},"tableAuthor":{"type":"object","required":["displayName","flags","id","profileImageUrlLarge","profileImageUrlMedium","profileImageUrlSmall","screenName","type"],"properties":{"id":{"type":"string"},"type":{"type":"string"},"flags":{"type":"array","items":{"type":"string"}},"screenName":{"type":"string"},"displayName":{"type":"string"},"profileImageUrlLarge":{"type":"string"},"profileImageUrlSmall":{"type":"string"},"profileImageUrlMedium":{"type":"string"}}},"averageRating":{"type":"integer"},"clientContext":{"type":"object","required":["clientContextVariables","inheritedVariables"],"properties":{"inheritedVariables":{"type":"object"},"clientContextVariables":{"type":"array"}}},"downloadCount":{"type":"integer"},"rowsUpdatedAt":{"type":"integer"},"rowsUpdatedBy":{"type":"string"},"indexUpdatedAt":{"type":"integer"},"attributionLink":{"type":"string"},"hideFromCatalog":{"type":"boolean"},"publicationDate":{"type":"integer"},"totalTimesRated":{"type":"integer"},"hideFromDataJson":{"type":"boolean"},"numberOfComments":{"type":"integer"},"publicationGroup":{"type":"integer"},"publicationStage":{"type":"string"},"viewLastModified":{"type":"integer"},"publicationAppendEnabled":{"type":"boolean"}}}}}}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]]
2022-06-01 08:35:40 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):234 - Stopping temporal heartbeating...
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/2417/2/logs.log
2022-06-01 08:35:40 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.30-alpha
2022-06-01 08:35:40 INFO i.a.w.DefaultNormalizationWorker(run):46 - Running normalization.
2022-06-01 08:35:40 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.66
2022-06-01 08:35:40 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.66 exists...
2022-06-01 08:35:41 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/normalization:0.1.66 was found locally.
2022-06-01 08:35:41 INFO i.a.w.p.DockerProcessFactory(create):157 - Preparing command: docker run --rm --init -i -w /data/2417/2/normalize --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local airbyte/normalization:0.1.66 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
2022-06-01 08:35:41 normalization > Running: transform-config --config destination_config.json --integration-type postgres --out /data/2417/2/normalize
2022-06-01 08:35:41 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/2417/2/normalize')
2022-06-01 08:35:41 normalization > transform_postgres
2022-06-01 08:35:41 normalization > Running: transform-catalog --integration-type postgres --profile-config-dir /data/2417/2/normalize --catalog destination_catalog.json --out /data/2417/2/normalize/models/generated/ --json-column _airbyte_data
2022-06-01 08:35:41 normalization > Processing destination_catalog.json...
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_format (#44) to us_p0_json_18_proper_s_view_columns_format (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_cachedContents (#52) to us_p0_json_18_proper_olumns_cachedContents (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_columns_cachedContents_top (#56) to us_p0_json_18_proper_ns_cachedContents_top (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_attachments (#50) to us_p0_json_18_proper__metadata_attachments (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields (#52) to us_p0_json_18_proper_etadata_custom_fields (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields_Common_Core (#64) to us_p0_json_18_proper_om_fields_Common_Core (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_custom_fields_Data_Quality (#65) to us_p0_json_18_proper_m_fields_Data_Quality (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition (#54) to us_p0_json_18_proper_adata_filterCondition (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children (#63) to us_p0_json_18_proper_terCondition_children (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata (#72) to us_p0_json_18_proper_ion_children_metadata (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata_customValues (#85) to us_p0_json_18_proper_metadata_customValues (#43)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_children_metadata_tableColumnId (#86) to us_p0_json_18_proper_etadata_tableColumnId (#43)
2022-06-01 08:35:41 normalization > Traceback (most recent call last):
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_filterCondition_metadata (#63) to us_p0_json_18_proper_terCondition_metadata (#43)
2022-06-01 08:35:41 normalization > File "/usr/local/bin/transform-catalog", line 8, in <module>
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_renderTypeConfig (#55) to us_p0_json_18_proper_data_renderTypeConfig (#43)
2022-06-01 08:35:41 normalization > sys.exit(main())
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_renderTypeConfig_visible (#63) to us_p0_json_18_proper_derTypeConfig_visible (#43)
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 82, in main
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_metadata_availableDisplayTypes (#60) to us_p0_json_18_proper_availableDisplayTypes (#43)
2022-06-01 08:35:41 normalization > TransformCatalog().run(args)
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submitter (#49) to us_p0_json_18_proper_w_approvals_submitter (#43)
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 35, in run
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submissionDetails (#57) to us_p0_json_18_proper_als_submissionDetails (#43)
2022-06-01 08:35:41 normalization > self.process_catalog()
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_approvals_submissionOutcomeApplication (#68) to us_p0_json_18_proper_ionOutcomeApplication (#43)
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 62, in process_catalog
2022-06-01 08:35:41 normalization > Truncating us_p0_json_18_properties_view_tableAuthor_flags (#47) to us_p0_json_18_proper_iew_tableAuthor_flags (#43)
2022-06-01 08:35:41 normalization > processor.process(catalog_file=catalog_file, json_column_name=json_col, default_schema=schema)
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/catalog_processor.py", line 63, in process
2022-06-01 08:35:41 normalization > for conflict in tables_registry.resolve_names():
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 157, in resolve_names
2022-06-01 08:35:41 normalization > conflicts = self.resolve_table_names()
2022-06-01 08:35:41 normalization > File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 196, in resolve_table_names
2022-06-01 08:35:41 normalization > assert (table_count * 2) == registry_size, f"Mismatched number of tables {table_count * 2} vs {registry_size} being resolved"
2022-06-01 08:35:41 normalization > AssertionError: Mismatched number of tables 76 vs 74 being resolved
...
discover_schema result
...
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
... 3 more
Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:159) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2022-06-01 08:35:41 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-06-01 08:35:41 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):234 - Stopping temporal heartbeating...
2022-06-01 08:35:41 WARN i.t.i.s.POJOActivityTaskHandler(activityFailureToResult):363 - Activity failure. ActivityId=778c0fb1-7098-3e6d-ae31-f07bf524c167, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:232) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:71) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:286) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:252) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:209) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:193) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.6.0.jar:?]
at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:412) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.6.0.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:227) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
... 13 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:129) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:227) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
... 13 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:60) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
... 1 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
... 1 more
Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:159) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Why this issue continues to occur when discover_schema works well?
AssertionError: Mismatched number of tables 64 vs 62 being resolved