Normalization don't run because of Schema validation error (Salesloft -> BigQuery)

Connector Questions & Issues
, ,
1
  • Machine type: Compute Engine e2-standard-2
  • CloudSQL : PostgreSQL 13 | vCPUs = 1 | Memory= 1.7 GB | SSD storage = 10 GB
  • airbyte-version → 0.38.4-alpha
  • airbyte/destination-bigquery → 1.1.6
  • airbyte/source-salesloft → 0.1.3

:warning: BTW Source salesloft documentation (:arrow_up:) is linked to https://docs.airbyte.io/integrations/sources/lever-onesignal and not https://docs.airbyte.com/integrations/sources/salesloft/

Hello,

Just noticed since i upgraded bigquery destination and salesloft source, the sync work well but normalization fail in the destination !

I don’t know why ! Just notices some schema validation errors

Can someone help me ? Thanks !

2022-06-01 17:47:13 e[44msourcee[0m > Finished syncing SourceSalesloft
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$6):334 - Source has no more messages, closing connection.
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$6):342 - Total records read: 453883 (525 MB)
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 410 schema validation errors found for stream emails. Error message: Record schema validation failed for emails. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"step":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"action":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"attachments":{"type":["null","integer"]},"unique_devices":{"type":["null","integer"]},"unique_locations":{"type":["null","integer"]}},"additionalProperties":false},"status":{"type":["null","string"]},"bounced":{"type":["null","boolean"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"headers":{"type":["null","object"],"properties":{"cc":{"type":["null","string"]},"to":{"type":["null","string"]},"bcc":{"type":["null","string"]},"References":{"type":["null","string"]},"In-Reply-To":{"type":["null","string"]},"in_reply_to":{"type":["null","string"]}},"additionalProperties":false},"mailing":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]}},"additionalProperties":false},"sent_at":{"type":["null","string"],"format":"date-time"},"recipient":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"created_at":{"type":"string","format":"date-time"},"send_after":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"crm_activity":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"view_tracking":{"type":["null","boolean"]},"click_tracking":{"type":["null","boolean"]},"email_template":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"personalization":{"type":["null","string"]},"recipient_email_address":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 64 schema validation errors found for stream team_templates. Error message: Record schema validation failed for team_templates. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"body":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"title":{"type":["null","string"]},"_links":{"type":["null","object"],"properties":{"attachments":{"type":["null","string"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"bounces":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"sent_emails":{"type":["null","integer"]}},"additionalProperties":false},"subject":{"type":["null","string"]},"created_at":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"body_preview":{"type":["null","string"]},"last_used_at":{"type":["null","string"],"format":"date-time"},"last_modified_at":{"type":["null","string"],"format":"date-time"},"last_modified_user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"open_tracking_enabled":{"type":["null","string"]},"click_tracking_enabled":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 16408 schema validation errors found for stream accounts. Error message: Record schema validation failed for accounts. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"city":{"type":["null","string"]},"name":{"type":["null","string"]},"size":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"owner":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"phone":{"type":["null","string"]},"state":{"type":["null","string"]},"counts":{"type":["null","object"],"properties":{"people":{"type":["null","integer"]}},"additionalProperties":false},"crm_id":{"type":["null","string"]},"domain":{"type":["null","string"]},"locale":{"type":["null","string"]},"street":{"type":["null","string"]},"country":{"type":["null","string"]},"creator":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"crm_url":{"type":["null","string"]},"founded":{"type":["null","string"]},"website":{"type":["null","string"]},"industry":{"type":["null","string"]},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"description":{"type":["null","string"]},"postal_code":{"type":["null","string"]},"account_tier":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"company_type":{"type":["null","string"]},"linkedin_url":{"type":["null","string"]},"owner_crm_id":{"type":["null","string"]},"company_stage":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"revenue_range":{"type":["null","string"]},"do_not_contact":{"type":["null","string"]},"twitter_handle":{"type":["null","string"]},"crm_object_type":{"type":["null","string"]},"last_contacted_at":{"type":["null","string"],"format":"date-time"},"last_contacted_by":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"conversational_name":{"type":["null","string"]},"last_contacted_type":{"type":["null","string"]},"last_contacted_person":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 3954 schema validation errors found for stream email_templates. Error message: Record schema validation failed for email_templates. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"body":{"type":["null","string"]},"tags":{"type":["null","array"],"items":{"type":["null","string"]}},"title":{"type":["null","string"]},"_links":{"type":["null","object"],"properties":{"attachments":{"type":["null","string"]}},"additionalProperties":false},"counts":{"type":["null","object"],"properties":{"views":{"type":["null","integer"]},"clicks":{"type":["null","integer"]},"bounces":{"type":["null","integer"]},"replies":{"type":["null","integer"]},"sent_emails":{"type":["null","integer"]}},"additionalProperties":false},"groups":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"shared":{"type":["null","string"]},"subject":{"type":["null","string"]},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"archived_at":{"type":["null","string"],"format":"date-time"},"body_preview":{"type":["null","string"]},"last_used_at":{"type":["null","string"],"format":"date-time"},"team_template":{"type":["null","object"],"properties":{"id":{"type":["null","string"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"template_owner":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"cadence_template":{"type":["null","string"]},"open_tracking_enabled":{"type":["null","string"]},"click_tracking_enabled":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 15435 schema validation errors found for stream actions. Error message: Record schema validation failed for actions. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"due":{"type":["null","string"]},"step":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"type":{"type":["null","string"]},"user":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"due_on":{"type":["null","string"],"format":"date-time"},"person":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"status":{"type":["null","string"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"created_at":{"type":"string","format":"date-time"},"updated_at":{"type":"string","format":"date-time"},"action_details":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"multitouch_group_id":{"type":["null","integer"]}}}
2022-06-01 17:47:13 e[33mWARNe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):345 - 3791 schema validation errors found for stream steps. Error message: Record schema validation failed for steps. Expected schema: {"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"day":{"type":["null","string"]},"name":{"type":["null","string"]},"type":{"type":["null","string"]},"cadence":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"details":{"type":["null","object"],"properties":{"id":{"type":["null","integer"]},"_href":{"type":["null","string"]}},"additionalProperties":false},"disabled":{"type":["null","string"]},"created_at":{"type":["null","string"],"format":"date-time"},"updated_at":{"type":["null","string"],"format":"date-time"},"step_number":{"type":["null","string"]},"display_name":{"type":["null","string"]}}}
2022-06-01 17:47:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):168 - One of source or destination thread complete. Waiting on the other.

logs-1491.txt (459.0 KB)

2

Can you double check normalization is enabled?

3

:sweat_smile: damn :poop:

i didn’t notice when added new streams that the settings went back to defaults

sorry for that stupid one

4

No problem this bug is solved latest. Please upgrade Airbyte.

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.