Summary
User is new to Airbyte and looking for guidance on configuring Change Data Capture (CDC) mode in the UI for real-time sync from MySQL to Redshift. Also inquiring about ongoing replication capabilities similar to AWS DMS.
Question
Hey everyone, I’m new to Airbyte and need some advice. I can’t seem to find the CDC (Change Data Capture) mode in the UI. I’m replicating data from MySQL to Redshift and am considering using Airbyte instead of AWS DMS.
Could someone also guide me on whether it’s possible to configure ongoing replication (real-time sync) instead of scheduled sync, similar to how DMS handles it?
For reference:
• MySQL connector uses Debezium to read binlogs.
• Redshift connector is configured to use S3 for data insertion.
Thanks in advance!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["cdc-mode", "real-time-sync", "mysql-connector", "redshift-connector", "ongoing-replication", "airbyte-ui"]
I looking something similar to this
first full load and than apply cdc (delete update insert )
Also is it possible to refresh/rewrite some data based on timestamp?
for example i need to refill missed data)
Did you set the cdc option in mysql source configuration ?
if yes then in connection, you can select a stream in incremental mode andthe cursor should be _ab_cdc_cursor
Yes i sets) just interesting for me is it possible setup ongoing update instead of scheduling)
I mean when new event received it shoud start proccessing immediatly and not waiting nest schedule time)
airbyte doesn’t have streaming connector. only with schedule right now
we have similar use case and we decided to increase the schedule
or if you are looking for streaming pipeline maybe you need to check with debezium or apache beam or something else