Summary
Setting up an EL process to move data from a table to S3 as separate files and delete the records from the database using Airbyte.
Question
I have a table which currently has 5 GB of data, growing by 100GB of data per month. I would like to setup a EL process where I move each row information to S3 as a separate file under a folder/bucket. I understand that this can be setup with Airbyte, but I also need to remove the same record from my database after uploading to S3. How should I handle this?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.