Summary
When not setting Delete Messages After Read to TRUE in Amazon SQS source connector, only the last Max Batch Size messages (max 10) are retrieved instead of all messages in the queue. Is this the expected behavior?
Question
Hi everyone, I am trying to extract data from an Amazon SQS queue but, if I do not set Delete Messages After Read to TRUE, I am only able to retrieve the last Max Batch Size messages (max 10) from the queue rather than all of the messages in the queue. Is that the expected behaviour? Or am I missing something?
I could not figure that out neither through http://Kapa.AI|Kapa.AI nor from the documentation of the SQS source connector. Thanks
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["amazon-sqs", "delete-messages-after-read", "sqs-source-connector", "data-extraction"]
Hi Alberto, from my experience with SQS, it’s the consumer responsibility to delete messages from the queue after successful processing. So even though I haven’t used it with AirByte, I would suspect that you need to delete the messages after consumption.
Hi <@U06E1FUNL9F>, thanks a lot for your answer, understood.
I guess that means that, if I do not delete the messages from the queue when extracting them, then the connector only consumes the last 10 messages in the queue and stops. While if I delete them, it goes through the whole queue and extracts all messages.
Yes, though it even may be that the connector tries to get the other messages, but SQS gives it the first unprocessed/undeleted ones, so the connector sees, that it already handled them, so stops.
I would imagine that the “Delete Messages After Read” setting to be there in order to enable testing without fear of losing data from the queue. But after the initial testing, the only reasonable setting is Delete Messages After Read = true.