It seems like most of the messaging systems I've looked at have basic, if any, support for priority message queues. For example, the AMQP only specifies a minimum of 2 priorities. RabbitMQ, an AMQP implementation, doesn't support any priorities. ActiveMQ will be getting support for 10 message priorities in version 5.4 in a couple days. 10 priority levels is the specified by the JMS spec.
A priority queue in the non-messaging sense of the word orders its contents based on an arbitrary field with an unconstrained range of priorities. Why does an implementation like this not exist as part of a messaging system? As I asked in the title, is priority an inherently non-messaging concept?
I realize that one answer might be that the concept of priority introduces the possibility of messages infinitely languishing in the queue while higher priority messages are processed. Are there other reasons?
BTW ActiveMQ now supports priority messaging in 5.4.x via JMSPriority headers.
Rather than getting the message broker to re-order messages within some buffer as they arrive, there are often better techniques to implement priority consumption, such as having a dedicated consumer pool for high priority messages. Then irrespective of how much noise there is from low priority messages, high priority messages will always get though.
Given the asynchronous nature of messaging its easy to fill up buffers, network pipes and prefetch queues with low priority messages if using things like JMSPriority headers etc.
In general, message queue systems are used to ensure delivery of messages between disparate systems.
Usually, there is some sort of once-and-only-once guarantee, and often a further promise that the messages will come in order.
By and large, that then informs the design of the system(s) that you are building and hooking together.
The concepts of priority between decoupled systems often don't make that much sense.
That said, one common workaround is to have two queues, one high priority and one background priority. The inherent problem is then made clear however, because of course the receiving system probably cannot halt processing the low-level request when a higher priority request comes in, so they are generally done sequentially at that level of granularity.
It seems to me that the idea is probably more akin to "process priority" than the priority values in a priority queue. Certainly that is consistent with the sentence-or-two about it in the JMS spec, and evidently with the AMQP spec as well.
One has to be careful in that too many priorities aren't used to the point where using the program becomes more burdensome than going through each message.
Messaging systems are designed and optimized for chronological ordering. File systems are optimized for appending files and not inserting data at the beginning or in the middle. Queue-like data structures are usually optimized for append at the end and removal from the head. For file systems, this means appending to file (adding) and appending to a transaction log (removal), and deleting message files once they are consumed (removal).
Introducing priorities to a processing queue effectively turns the queue into a data structure that has both chronological and priority sorting. Basically, when it comes to working with file storage, it's quite sub-optimal as you have to create some sort of indexing strategy.
Related
For Create operations it is clear that putting the message in the queue is a good idea in case the processing or creation of that entity takes longer than expected and other the other benefits queues bring.
However, for read operations that are timebound (must return to the UI in less than 3 seconds) it is not entirely clear if a queue is a good idea.
http://masstransit-project.com/MassTransit/usage/request-response.html provides a nice abstraction but it goes through the queue.
Can someone provide some suggestions as to why or why not I would use mass transit or that effect any technology like nservicebus etc for database read operation that are UI timebound?
Should I only use mass transit only for long running processes?
Request/Reply is a perfectly valid pattern for timebound operations. Transport costs in case of, for example, RabbitMQ, are very low. I measured performance of request/response using ServiceStack (which is very fast) and MassTransit. There is an initial delay with MassTransit to cache the endpoints, but apart from that the speed is pretty much the same.
Benefits here are:
Retries
Fine tuning of timeouts
Easy scaling with competing consumers
just to name the most obvious ones.
And with error handling you get your requests ending up in the error queue so there is no data loss and you can always look there to find out what and why went wrong.
Update: There is a SOA pattern that describes this (or rather similar) approach. It is called Decoupled Invocation.
I am trying to understand what is the use case of a queue in distributed system.
And also how it scales and how it makes sure it's not a single point of failure in the system?
Any direct answer or a reference to a document is appreciated.
Use case:
I understand that queue is a messaging system. And it decouples the systems that communicate between each other. But, is that the only point of using a queue?
Scalability:
How does the queue scale for high volumes of data? Both read and write.
Reliability:
How does the queue not becoming a single point of failure in the system? Does the queue do a replication, similar to data-storage?
My question is not specified to any particular queue server like Kafka or JMS. Just in general.
Queue is a mental concept, the implementation decides about 1 + 2 + 3
A1: No, it is not the only role -- a messaging seems to be main one, but a distributed-system signalling is another one, by no means any less important. Hoare's seminal CSP-paper is a flagship in this field. Recent decades gave many more options and "smart-behaviours" to work with in designing a distributed-system signalling / messaging services' infrastructures.
A2: Scaling envelopes depend a lot on implementation. It seems obvious that a broker-less queues can work way faster, that a centralised, broker-based, infrastructure. Transport-classes and transport-links account for additional latency + performance degradation as the data-flow volumes grow. BLOB-handling is another level of a performance cliff, as the inefficiencies are accumulating down the distributed processing chain. Zero-copy ( almost ) zero-latency smart-Queue implementations are still victims of the operating systems and similar resources limitations.
A3: Oh sure it is, if left on its own, the SPOF. However, Theoretical Cybernetics makes us safe, as we can create reliable systems, while still using error-prone components. ( M + N )-failure-resilient schemes are thus achievable, however the budget + creativity + design-discipline is the ceiling any such Project has to survive with.
my take:
I would be careful with "decouple" term - if service A calls api on service B, there is coupling since there is a contract between services; this is true even if the communication is happening over a queue, file or fax. The key with queues is that the communication between services is asynchronous. Which means their runtimes are decoupled - from practical point of view, either of systems may go down without affecting the other.
Queues can scale for large volumes of data by partitioning. From clients point of view, there is one queue, but in reality there are many queues/shards and number of shards helps to support more data. Of course sharding a queue is not "free" - you will lose global ordering of events, which may need to be addressed in you application.
A good queue based solution is reliable based on replication/consensus/etc - depends on set of desired properties. Queues are not very different from databases in this regard.
To give you more direction to dig into:
there an interesting feature of queues: deliver-exactly-once, deliver-at-most-once, etc
may I recommend Enterprise Architecture Patterns - https://www.enterpriseintegrationpatterns.com/patterns/messaging/Messaging.html this is a good "system design" level of information
queues may participate in distributed transactions, e.g. you could build something like delete a record from database and write it into queue, and that will be either done/committed or rolledback - another interesting topic to explore
Does ZeroMQ guarantee the order of messages( FIFO ).
Is there an option for persistence.
Is it the best fit for IPC communications.
Does it allow prioritizing the messages.
Does it allow prioritizing the receivers.
Does it allow both synchronous as well asynchronous way of communication?
https://lists.zeromq.org/pipermail/zeromq-dev/2015-January/027748.html
The author said:"Messages carried over TCP or IPC will be delivered in order if they pass through the same network paths. This is guaranteed and it's a TCP
guarantee, nothing to do with ZeroMQ. ZeroMQ does not reorder
messages, ever. However if you pass messages through two or more
paths, and then merge those flows again, you will in effect shuffle
the messages."
Zeromq is best understood as a udp like messaging system. Thus is does not intrinsically guarantee any of that. It DOES guarantee that parts of a single message are received atomically and in order, since ZMQ allows for sending a message consisting of several parts. All communication is always asynchronous by design.
see http://zguide.zeromq.org/ for more advanced patterns.
that being said, all the features requested would by definition make the transmission slower and more complicated. If they are needed you should implement or use one of the available patterns of the guide.
I'm evaluating possible solutions for handling a large quantity of queued messages, which must be delivered to workers at a certain date and time. The result of executing them is mostly updates to stored data, and they may or may not be originally triggered by user action.
For example, think of what you'd implement in a hypothetical large-scale StarCraft game server for storing and executing users' actions, like upgrading a building, hatching a soldier, all of which requires to be applied to the game state after several seconds or minutes after the player initiates them.
The problem is I can't seem to find the right term to name this problem area. There are several that looks similar, but different:
cron/task/job scheduler
The content of the queue is not dynamic, it's predefined.
Each task is scheduled.
message queue
The content of the queue is dynamic.
Each task is intended to be delivered immediately.
???
The content of the queue is dynamic.
Each task is scheduled.
If there are message queues that allow conditional delivery of messages, that might be it.
Summary:
What are these kind of technology called?
What are some of the solutions out there?
This just sounds like a trivial priority queue on the surface. The priority in this case is the time of completion, and you check the front of the queue to see when the next event is due. Pretty much every language comes with a priority queue or something that can easily be used as one, so I'm not sure what the actual problem is here.
Is it that you're worried about scalability, when it comes to millions of messages? Obviously 'millions' is a meaningless term - if that's millions per day, it's a trivial problem. If it's millions per second, then you can just scale horizontally, splitting the queue across multiple processes. (And the benefit of such a queue system is that this parallelization is really simple.)
I would bet that when implementing a large scale real-time strategy game server you would hit networking problems long before you start hitting problems with the message queue.
Have you tried looking at push queues by Iron.io? The content of the queue can be anything you like, and you specify a webhook to where the messages will be pushed to. You can also set a delay for each of the messages.
The webhook is static though for each queue and delay isn't always exactly on time (could be up to a minute off). If timing is more important or the ability of providing a different webhook per message is important, try looking at boomerang.io.
They say they are pretty accurate on the timing, you can provide a delay or unix timestamp for the webhook to return and that is per message. Sounds like either of those might work for you.
For StarCraft, I would use the Red Dwarf server.
For a Java EE app, I would use Quartz Scheduler.
It seems to me that a queue-based solution would be best in this case for a number of reasons:
Management. Most queuing solutions provide support for inspecting the content of queues which makes it easier to debug, easier to take action when certain threshold are exceeded, ...
Performance. You can divide workload by having multiple enqueue/dequeue processes (gives you the ability to scale out).
Prioritizing. Most queues support prioritizing of messages (probably not all messages are equally important).
...
Remaining problem is the immediate delivery of messages in the queue. You have two ways to solve this: either delay enqueuing of messages or delay execution of dequeued messages. I would go with the first approach, delayed enqueuing.
A message then has two properties: (content, delay). You provide the message to a component in your system that queues the message at the appropriate time.
I'm not sure what programming language you're using, but the MS .NET 4 framework has support for such a scenario (delayed execution of tasks).
For message-oriented middleware that does not consistently support priority messages (such as AMQP) what is the best way to implement priority consumption when queues have only FIFO semantics? The general use case would be a system in which consumers receive messages of a higher priority before messages of a lower priority when a large backlog of messages exists in Queue(s).
Given only FIFO support for a given single queue, you will of course have to introduce either multiple queues, an intermediary, or have a more complex consumer.
Multiple queues could be handled in a couple of ways. The producer and consumer could agree to have two queues between them, one for high-priority, and one for background tasks.
If your producer is constrained to a single queue, but you have control over the consumer, consider introducing a fan-out router in the path. So producer->Router is a single queue, and the router then has two queues to the consumer.
Another way to tackle it, which is likely less than ideal, would be to have your consumer spin a thread to front the queue, then dispatch the work internally. Something like the router version above, but inside a single app. This has the downside of having multiple messages in flight inside your app, which may complicate recovery in the event of a failure.
Don't forget to consider starvation of the effectively low-priority events, whatever they are, if some of them should be processed even if there are higher-priority events still hanging about.