Can I enforce message order by stream name in a destination?

Connector Development
1

Hi all! I am building a custom destination, and I wanted to know if there is any way to enforce message order by stream name? My destination can support multiple streams, but some have dependencies, such as “a message from PARENT must be inserted before a message from CHILD that references PARENT”. Should this be handled by having a separate connection for source -> destination[PARENT] and source -> destination[CHILD] instead?

2

Hey @jyates-om1,
This ordering, on the destination side, is not possible. I’d be interested in understanding your use case a bit more because from my perspective this inter stream dependency should be expressed in the source connector.
In our source connectors, some streams depend on others and this kind of ordering scenario is quite common (You can check the HTTPSubStream class of our CDK).
I think that having an ordering on the destination side would mean that your destination is not source-agnostic and can’t work with all our connectors.

If you still want to force stream ordering on the destination you can try the following workaround:

  • Create one connection per stream and trigger sync jobs with an orchestrator such as Airflow, on which you will express the dependencies
  • A more hacky approach would be to benefit from the lack of parallelization we have in sync jobs: if you have control over the stream names (tables name with database connector) you could rename these streams and they will be replicated in the alphabetical order.
3

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.