Source Hubspot 0.1.53: ticket_pipelines stream with wrong data type

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Debian
  • Memory / Disk: 12 GB
  • Deployment: Kubernetes deployment
  • Airbyte Version: 0.35.64-alpha
  • Source name/version: Hubspot
  • Destination name/version: Bigquery
  • Step: Sync
  • Description:

Seems I got 2 errors in models that I’m currently investigating on my side.
But there is also a strange error that fails the JSON Schema validation?

Here are the end logs =>
logs-90.txt (4.4 MB)

2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.102712 [debug] [MainThread]: Connection 'model.airbyte_utils.engagements_metadata_bcc' was properly closed.
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.188253 [info ] [MainThread]: 
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.188673 [info ] [MainThread]: e[31mCompleted with 2 errors and 0 warnings:e[0m
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.189019 [info ] [MainThread]: 
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.189306 [error] [MainThread]: e[33mDatabase Error in model ticket_pipelines (models/generated/airbyte_tables/hubspot_/ticket_pipelines.sql)e[0m
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.189600 [error] [MainThread]:   Bad int64 value: 1970-01-01T00:00:00Z
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.189853 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/hubspot_/ticket_pipelines.sql
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.190107 [info ] [MainThread]: 
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.190381 [error] [MainThread]: e[33mDatabase Error in model contacts_properties (models/generated/airbyte_incremental/hubspot_/contacts_properties.sql)e[0m
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.190657 [error] [MainThread]:   Invalid date: '1 an et demi'
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.191009 [info ] [MainThread]: 
2022-04-13 05:22:22 e[42mnormalizatione[0m > 05:22:18.191296 [info ] [MainThread]: Done. PASS=73 WARN=0 ERROR=2 SKIP=2 TOTAL=77
2022-04-13 05:22:22 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.credential: is not defined in the schema and the schema does not allow additional properties, $.part_size_mb: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, $.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, $.method: must be a constant value Standard
2022-04-13 05:22:22 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:60) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	... 3 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:160) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]
2022-04-13 05:22:22 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-04-13 05:22:22 e[32mINFOe[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating...
2022-04-13 05:22:22 e[33mWARNe[m i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=b3bf8196-7d42-34a5-9555-0704916657f7, activityType=Normalize, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:233) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:71) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:228) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	... 13 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:129) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$1(NormalizationActivityImpl.java:97) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:228) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	... 13 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:60) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	... 1 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
	... 1 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:160) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:45) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:18) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]

And btw, thanks for this discourse! :slight_smile:

The JSON validation is only an info don’t need to be worry about it.
What version of the Hubspot connector version are you using?

Ok, we’re using version 0.1.52

See issue: Hubspot Connector 0.1.53 uses incorrect ticket_pipeline date type · Issue #12318 · airbytehq/airbyte · GitHub

@RemyTinco can you update to version 0.1.53?