Summary
User is looking to extract events/email data from SendGrid using Airbyte connector. They are seeking confirmation if the existing connector supports this data or if a custom connector needs to be built.
Question
Hey hey team,
I am looking to get events/email data out of SendGrid. I can see a Connector exists, but none of the streams seem to contain any of that granular data.
Just looking for confirmation that that is correct and want to ask if anyone has managed to build a custom connector to do this instead?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["sendgrid", "events-email-data", "connector", "custom-connector"]
<@U06DQ4RHPUJ> We ended up going the webhook route because our client didn’t want to pay for the add-on.
Interestingly the schema listed here doesn’t show anything about offset or a pagination object in the response like you see on other endpoints:
https://www.twilio.com/docs/sendgrid/api-reference/email-activity/filter-all-messages
Not sure if that’s a docs omission or if it actually doesn’t paginate at all and if limit is just when you only want a subset of the data. You could also try NOT passing limit and see if pagination fields show up in the response
You could also check with their support since this IS a paid add-on . . . I wonder if the docs are just poor since it isn’t used nearly as much. But to me if you can’t paginate those results, there isn’t much of a reason for the add-on (unless they’re expecting everyone to always filter by both the msg_id and to_email or something)
From what I can see so far, I just don’t think pagination is supported, and I think the offset parameter is just not doing anything. You have to specify a limit (not adding it causes the call to fail), so I think my best bet is to try and filter down to a timeframe that is small enough that I will be able to capture all the activity based on the last_event_time without hitting the 1000 objects returned of the limit
Out of curiosity, how does your setup with the webhooks look? I’ve never really worked with them so don’t know what tools/infra they require