Pipedrive Source Connector v0.1.10 - Custom fields won't be normalized on default

Connector Questions & Issues
, ,
1

I want to use the latest changes of the Pipedrive Source Connector (v0.1.10) to be able to retrieve custom fields.
The raw data of deals, orgs, and persons also seem to be loaded to BigQuery.

image
image1314×96 9.06 KB

But normalization doesn’t automatically apply the custom fields to the final normalized table. (see schema in the bottom)

So the deals or orgs tables don’t show any new columns with the custom field ids. What step do I need to take? Custom dbt script required?

→

  • Is this your first time deploying Airbyte?: No / Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk:
  • Deployment: Kubernetes
  • Airbyte Version: 0.35.17-alpha
  • Source name/version: Pipedrive 0.1.10
  • Destination name/version: BigQuery 1.0.1

Current Deal Table Schema without custom fields

Field name Type Mode Collation Policy Tags Description
id INTEGER NULLABLE
label STRING NULLABLE
title STRING NULLABLE
value FLOAT NULLABLE
active BOOLEAN NULLABLE
org_id INTEGER NULLABLE
status STRING NULLABLE
deleted BOOLEAN NULLABLE
user_id INTEGER NULLABLE
add_time TIMESTAMP NULLABLE
cc_email STRING NULLABLE
currency STRING NULLABLE
org_name STRING NULLABLE
stage_id INTEGER NULLABLE
won_time TIMESTAMP NULLABLE
lost_time TIMESTAMP NULLABLE
person_id INTEGER NULLABLE
close_time TIMESTAMP NULLABLE
org_hidden BOOLEAN NULLABLE
owner_name STRING NULLABLE
visible_to STRING NULLABLE
files_count INTEGER NULLABLE
lost_reason STRING NULLABLE
notes_count INTEGER NULLABLE
person_name STRING NULLABLE
pipeline_id INTEGER NULLABLE
probability FLOAT NULLABLE
rotten_time STRING NULLABLE
update_time TIMESTAMP NULLABLE
person_hidden BOOLEAN NULLABLE
first_won_time TIMESTAMP NULLABLE
products_count INTEGER NULLABLE
stage_order_nr INTEGER NULLABLE
weighted_value FLOAT NULLABLE
creator_user_id INTEGER NULLABLE
followers_count INTEGER NULLABLE
formatted_value STRING NULLABLE
activities_count INTEGER NULLABLE
last_activity_id INTEGER NULLABLE
next_activity_id INTEGER NULLABLE
stage_change_time TIMESTAMP NULLABLE
last_activity_date DATE NULLABLE
next_activity_date DATE NULLABLE
next_activity_note STRING NULLABLE
next_activity_time STRING NULLABLE
next_activity_type STRING NULLABLE
participants_count INTEGER NULLABLE
expected_close_date DATE NULLABLE
email_messages_count INTEGER NULLABLE
done_activities_count INTEGER NULLABLE
2

Hey @martinibach,
Did you refresh your schema in the connection config after upgrading the Pipedrive connector?

3

@alafanechere yes I did but the custom fields don’t show up there.

4

@martinibach do you confirm that your custom fields value exists in the raw table but not in the destination table?
Do you mind trying to re-create the connection?
Could you please also share the source_catalog.json and destination_catalog.json files?

docker run -it --rm --volume airbyte_workspace:/data busybox cat /data/<job-id>/0/source_catalog.jsondocker run -it --rm --volume airbyte_workspace:/data busybox cat /data/<job-id>/0/destination_catalog.json
5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.