Can I enforce message order by stream name in a destination?

Hi all! I am building a custom destination, and I wanted to know if there is any way to enforce message order by stream name? My destination can support multiple streams, but some have dependencies, such as “a message from PARENT must be inserted before a message from CHILD that references PARENT”. Should this be handled by having a separate connection for source -> destination[PARENT] and source -> destination[CHILD] instead?

Hey @jyates-om1,
This ordering, on the destination side, is not possible. I’d be interested in understanding your use case a bit more because from my perspective this inter stream dependency should be expressed in the source connector.
In our source connectors, some streams depend on others and this kind of ordering scenario is quite common (You can check the HTTPSubStream class of our CDK).
I think that having an ordering on the destination side would mean that your destination is not source-agnostic and can’t work with all our connectors.

If you still want to force stream ordering on the destination you can try the following workaround:

  • Create one connection per stream and trigger sync jobs with an orchestrator such as Airflow, on which you will express the dependencies
  • A more hacky approach would be to benefit from the lack of parallelization we have in sync jobs: if you have control over the stream names (tables name with database connector) you could rename these streams and they will be replicated in the alphabetical order.