Normalization don't run because of Schema validation error (Salesloft -> BigQuery)

  • Machine type: Compute Engine e2-standard-2
  • CloudSQL : PostgreSQL 13 | vCPUs = 1 | Memory= 1.7 GB | SSD storage = 10 GB
  • airbyte-version → 0.38.4-alpha
  • airbyte/destination-bigquery → 1.1.6
  • airbyte/source-salesloft → 0.1.3

:warning: BTW Source salesloft documentation (:arrow_up:) is linked to https://docs.airbyte.io/integrations/sources/lever-onesignal and not https://docs.airbyte.com/integrations/sources/salesloft/


Hello,

Just noticed since i upgraded bigquery destination and salesloft source, the sync work well but normalization fail in the destination !

I don’t know why ! Just notices some schema validation errors

Can someone help me ? Thanks !

2022-06-01 17:47:13 e[44msourcee[0m > Finished syncing SourceSalesloft
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$6):334 - Source has no more messages, closing connection.
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$6):342 - Total records read: 453883 (525 MB)
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 410 schema validation errors found for stream emails. Error message: Record schema validation failed for emails. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"step":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"action":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"attachments":{"type":["null","integer"]},"unique_devices":{"type":["null","integer"]},"unique_locations":{"type":["null","integer"]}},"additionalProperties":false},"status":{"type":["null","string"]},"bounced":{"type":["null","boolean"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"headers":{"type":["null","object"],"properties":{"cc":{"type":["null","string"]},"to":{"type":["null","string"]},"bcc":{"type":["null","string"]},"References":{"type":["null","string"]},"In-Reply-To":{"type":["null","string"]},"in_reply_to":{"type":["null","string"]}},"additionalProperties":false},"mailing":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]}},"additionalProperties":false},"sent_at":{"type":["null","string"],"format":"date-time"},"recipient":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"created_at":{"type":"string","format":"date-time"},"send_after":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"crm_activity":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"view_tracking":{"type":["null","boolean"]},"click_tracking":{"type":["null","boolean"]},"email_template":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"personalization":{"type":["null","string"]},"recipient_email_address":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 64 schema validation errors found for stream team_templates. Error message: Record schema validation failed for team_templates. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"body":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"title":{"type":["null","string"]},"_links":{"type":["null","object"],"properties":{"attachments":{"type":["null","string"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"bounces":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"sent_emails":{"type":["null","integer"]}},"additionalProperties":false},"subject":{"type":["null","string"]},"created_at":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"body_preview":{"type":["null","string"]},"last_used_at":{"type":["null","string"],"format":"date-time"},"last_modified_at":{"type":["null","string"],"format":"date-time"},"last_modified_user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"open_tracking_enabled":{"type":["null","string"]},"click_tracking_enabled":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 16408 schema validation errors found for stream accounts. Error message: Record schema validation failed for accounts. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"city":{"type":["null","string"]},"name":{"type":["null","string"]},"size":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"owner":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"phone":{"type":["null","string"]},"state":{"type":["null","string"]},"counts":{"type":["null","object"],"properties":{"people":{"type":["null","integer"]}},"additionalProperties":false},"crm_id":{"type":["null","string"]},"domain":{"type":["null","string"]},"locale":{"type":["null","string"]},"street":{"type":["null","string"]},"country":{"type":["null","string"]},"creator":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"crm_url":{"type":["null","string"]},"founded":{"type":["null","string"]},"website":{"type":["null","string"]},"industry":{"type":["null","string"]},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"description":{"type":["null","string"]},"postal_code":{"type":["null","string"]},"account_tier":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"company_type":{"type":["null","string"]},"linkedin_url":{"type":["null","string"]},"owner_crm_id":{"type":["null","string"]},"company_stage":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"revenue_range":{"type":["null","string"]},"do_not_contact":{"type":["null","string"]},"twitter_handle":{"type":["null","string"]},"crm_object_type":{"type":["null","string"]},"last_contacted_at":{"type":["null","string"],"format":"date-time"},"last_contacted_by":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"conversational_name":{"type":["null","string"]},"last_contacted_type":{"type":["null","string"]},"last_contacted_person":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 3954 schema validation errors found for stream email_templates. Error message: Record schema validation failed for email_templates. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"body":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"title":{"type":["null","string"]},"_links":{"type":["null","object"],"properties":{"attachments":{"type":["null","string"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"bounces":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"sent_emails":{"type":["null","integer"]}},"additionalProperties":false},"groups":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"shared":{"type":["null","string"]},"subject":{"type":["null","string"]},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"body_preview":{"type":["null","string"]},"last_used_at":{"type":["null","string"],"format":"date-time"},"team_template":{"type":["null","object"],"properties":{"id":{"type":["null","string"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"template_owner":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"cadence_template":{"type":["null","string"]},"open_tracking_enabled":{"type":["null","string"]},"click_tracking_enabled":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 15435 schema validation errors found for stream actions. Error message: Record schema validation failed for actions. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"due":{"type":["null","string"]},"step":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"type":{"type":["null","string"]},"user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"due_on":{"type":["null","string"],"format":"date-time"},"person":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"status":{"type":["null","string"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"action_details":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"multitouch_group_id":{"type":["null","integer"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 3791 schema validation errors found for stream steps. Error message: Record schema validation failed for steps. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"day":{"type":["null","string"]},"name":{"type":["null","string"]},"type":{"type":["null","string"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"details":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"disabled":{"type":["null","string"]},"created_at":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"step_number":{"type":["null","string"]},"display_name":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):168 - One of source or destination thread complete. Waiting on the other.

logs-1491.txt (459.0 KB)

Can you double check normalization is enabled?

:sweat_smile: damn :poop:

i didn’t notice when added new streams that the settings went back to defaults

sorry for that stupid one

No problem this bug is solved latest. Please upgrade Airbyte.

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.