Notify subscribers after new messages have stopped coming in - message-queue

Question

In an app where users are expected to make several changes in a short period of time, I'd like to use a message queue to collect these events, and only notify listeners when new changes have stopped coming in for some period X.
The expected workflow would be:
User makes an edit -> message added to queue
User makes another edit -> message added to queue
Some time passes
Consumer is notified of all pending changes
I've looked into documentation for several different message queues, but none of them seem to have this kind of message batching out of the box.
I did find some features that might help to roll my own, e.g. Kafka has a producer config called linger that tells it to wait X ms for more messages to add to a batch, but this is clearly intended as a performance improvement. In addition, this option is at the producer side, whereas for my use case it would make more sense on the consumer side.
Is this a use case message queues can support? The lack of results makes me think that I may be trying to use message queues wrong.

Answer 1

Queues are not a good fit for such use cases. I would recommend using Cadence Workflow to implement your logic with a minimal effort.
Here is a straw-man design that satisfies your requirements:
Send signalWithStart request that contains an edit information to a user workflow using userID as the workflow ID. It either delivers the signal to the workflow or first starts the workflow and delivers signal to it.
All request to that workflow are buffered by it. Cadence provides hard guarantee that only one workflow with given ID can exist in open state. So all signals (events) are guaranteed to be buffered in the workflow that belongs to the user.
After configured timeout an activity that notifies users about the pending changes is invoked.
The pending changes are applied by the next activity.
The workflow complete.
Here is the workflow code that implements it in Java (Go client is also supported):
public interface BufferedEditsWorkflow {
#WorkflowMethod
void execute(String userId, Duration notifyAfter, Edit firstEdit);
#SignalMethod
void addEdit(Edit edit);
}
public interface BufferedEditsActivities {
void notifyUser(String userId, List<Edit> edits);
void process(String userId, List<Edit> edits);
}
public class BufferedEditsWorkflowImpl implements BufferedEditsWorkflow {
private final List<Edit> edits = new ArrayList<>();
private final BufferedEditsActivities activities = Workflow.newActivityStub(BufferedEditsActivities.class);
#Override
public void execute(String userId, Duration notifyAfter, Edit firstEdit)
{
edits.add(firstEdit);
// Cadence doesn't have limit on sleep duration.
// It can sleep at this line for a year with no problem.
Workflow.sleep(notifyAfter);
activities.notifyUser(userId, edits);
activities.process(userId, edits);
}
#Override
public void addEdit(Edit edit) {
edits.add(edit);
}
}
Code that starts the workflow for the first edit:
private void addFirstEdit(WorkflowClient cadenceClient, Edit edit) {
WorkflowOptions options = new WorkflowOptions.Builder().setWorkflowId(edit.getUserId()).build();
BufferedEditsWorkflow workflow = cadenceClient.newWorkflowStub(BufferedEditsWorkflow.class, options);
workflow.execute(edit.getUserId(), Duration.ofHours(1), edit);
}
Code that adds more edits.
private void addEdit(WorkflowClient cadenceClient, Edit edit) {
WorkflowOptions options = new WorkflowOptions.Builder().setWorkflowId(edit.getUserId()).build();
BufferedEditsWorkflow workflow = cadenceClient.newWorkflowStub(BufferedEditsWorkflow.class, options);
workflow.addEdit(edit);
}
Cadence offers a lot of other advantages over using queues for task processing.
Built it exponential retries with unlimited expiration interval
Failure handling. For example it allows to execute a task that notifies another service if both updates couldn't succeed during a configured interval.
Support for long running heartbeating operations
Ability to implement complex task dependencies. For example to implement chaining of calls or compensation logic in case of unrecoverble failures (SAGA)
Gives complete visibility into current state of the update. For example when using queues all you know if there are some messages in a queue and you need additional DB to track the overall progress. With Cadence every event is recorded.
Ability to cancel an update in flight.
See the presentation that goes over Cadence programming model.