1, How long does a message live in the listener queue? Until the dispatcher reads the message out of the queue in a "1 publisher 1 consumer" scenario?
Listener listener = new Listener(Queue.Default, transport, subject, new object());
listener.MessageReceived += OnMessageReceived;
Dispatcher dispatcher = new Dispatcher(listener.Queue);
2, Tibco RV is typically used in a large fanout system, with relatively loose requirements on delivery reliability, say, market data published to 20 applications in an enterprise. I've heard Tibco RV implements a "no copy" solution for fanouts - how is that even possible? I'd assume we need to at least go through all registered listeners for that queue and notify each of them, in which process the message is copied 20 times. Please enlighten me.
3, Combine question 1 and 2, it doesn't make sense for a message to live in the listener queue until ALL registered listeners have consumed the message - What happens if 1 of the 20 applications goes offline? It's going to bring down the rv daemon process due to ever increasing messages. Does Tibco RV have a lifetime limit(ttl) for each message? How do I check it and set it to new values?
There isn't much related info on Google, so please help.
Thanks.
Great questions.
Keep in mind that unless you are using RV certified messaging there will be no persistence to disk. Messages sent will remain in the memory of the sending Rendezvous daemon until they are delivered to all consumers.
That said, another thing to understand is that RV is an optimistic protocol as opposed to say, TCP which is a pessimistic protocol. Every packet is sent using TCP must be acknowledged. This round-trip protocol slows things down. Rendezvous on the other hand uses a protocol that sits on top of UDP which is session-less and does not require acks. Therefore, when a Rendezvous daemon sends a message, it is assumed to have been delivered successfully unless a retransmission request is received. So to completely answer your first question the default behavior of a Rendezvous daemon is to keep messages that it has sent in memory for 60 seconds after it has sent the message. This allows it to honor retransmission requests.
Fan out is achieved using broadcast or multicast protocols on top of UDP. The use of broadcast is discouraged and multicast is encouraged. Using multicast groups uses considerably less network resources. At the network interface level only those hosts that have joined the multicast group will receive packets associated with the Rendezvous traffic. Similarly at the network switch level multicast is a lot less resource-intensive.
Bottom line is that the sending Rendezvous daemon only sends the message out once, and the network delivers a copy of the associated packets to either each host on the subnet if broadcast is used, or the hosts that have registred interest if multicast is used.
In pub-sub, typically consumers receive messages that are sent while they are alive and consuming. Thus with pure Rendezvous, if a consumer was to go down, the subscription would be cancelled for that consumer. If we think about your example of market data, this is exactly the behavior we want. IBM trades at thousands of times a second, so if I miss a price quote it's no big deal. I'll get the next one. Furthermore, I don't want stale prices.
That said, sometimes consumers do want messages sent while they were down. This can be achieved using certified messaging and setting up persistent correspondents. For more on this see the Rendezvous Concepts Guide. Lastly, the 60-second behavior that I mentioned in point #1 can be changed using the -reliability parameter when starting the Rendezvous daemon. There are some cases where this may make sense (although the default is best for most common cases). For more details on this take a look at the Rendezvous Admin Guide.
Related
We use SQS for queueing use-cases in our company. All developers connect to the same queue for local development. If we're producing some messages for testing in local development, it can happen that the message is consumed on other person's locally running consumer, if that person has the app running at the same time.
How do you make sure that messages produced by one person don't end up getting lost by consumption on other person's locally running consumer. Is using different different queues for each person the only solution? Wondering what is standard followed to avoid this in the industry?
This is very open-ended IMO. Would recommend adding some context as to how you're using SQS.
But from what I could understand:
Yes, I would recommend creating queues per "developer"
OR
Although not elegant, you can maybe add an SQS message attribute (this is metadata other than message body) with a developer's username.
And each developer should then only process a message if it's meant for them. Arguably, you could also add a flag in the message itself, but, I am not sure about the constraints on your message format. Message attributes are meant to be used for these situations, where you want to know if you really need to process a message before even parsing the message body.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes
But you'll have to increase the maxReceives to a high number (so that message does not move to dead letter queue, if you have configured one). This is not exhaustive, it will just decrease the chances of your messages being deleted by someone else. Because if say, 10 people read the message and did not delete it because username was not their username, and maxReceives is 8, it will still move to DLQ and cause unnecessary confusion.
im having some trouble with our mail server since yesterday.
First, the server was down for couple days, thanks to KVM, VMs were paused because storage was apparently full. So i managed to fix the issue. But since the mail server is back online, CPU usage was always at 100%, i checked logs, and there was "millions", of mails waiting in the postfix queue.
I tried to flush the queue, thanks to the PFDel script, it took some times, but all mails were gone, and we were finally able to receive new emails. I also forced a logrotate, because fail2ban was also using lots of CPU.
Unfortunately, after couple hours, postfix active queue is still growing, and i really dont understand why.
Another script i found is giving me that result right now:
Incoming: 1649
Active: 10760
Deferred: 0
Bounced: 2
Hold: 0
Corrupt: 0
is there a possibility to desactivate ""Undelivered Mail returned to Sender" ?
Any help would be very helpful.
Many thanks
You could firstly temporarily stop sending bounce mails completely or set more strict rules in order to analyze the reasons of the flood. See for example:http://domainhostseotool.com/how-to-configure-postfix-to-stop-sending-undelivered-mail-returned-to-sender-emails-thoroughly.html
Sometimes the spammers find some weakness (or even vulnerability) in your configuration or in SMTP server and using that to send the spam (also if it could reach the addressee via bounce only). Mostly in this case, you will find your IP/domain in some common blacklist services (or it will be blacklisted by large mail providers very fast), so this will participate additionally to the flood (the bounces will be rejected by recipient server, what then let grow you queue still more).
So also check your IP/domain using https://mxtoolbox.com/blacklists.aspx or similar service (sometimes they provide also the reason why it was blocked).
As for fail2ban, you can also analyze logs (find some pattern) to detect the evildoers (initial sender), and write custom RE for fail2ban to ban them for example after 10 attempts in 20 minutes (or add it to ignore list for bounce messages in postfix)... so you'd firstly send X bounces, but hereafter it'd ban the recidive IPs, what could also help to reduce the flood significantly.
An last but not least, check your config (follow best practices for it) and set up at least MX/SPF records, DKIM signing/verification and DMARC policies.
I am working for a while with ZeroMQ. I read already some whitepapers and a lot from the guide, but one question remained open to me:
Lets say we use PUB-SUB.
Where or on which system are the messaging queues? On the publisher system side, on the subscriber system side or something in between?
Sorry for the maybe stupid question.
This picture is from the Broker vs. Brokerless whitepaper.
Every zeromq socket has both send and recv queues (The limits are set via high water mark).
In the case of a brokerless PUB/SUB the messages could be queued on the sender or the receiver.
Example:
If there is network congestion the sender may queue on its send queue
If the receiver is consuming messages slower than they arrive they will be queued on the receiver recv queue
If the queues reach the high water mark the messages will be lost.
Q : Where or on which system are the messaging queues?
The concept of the Zen-of-Zero, as built-into ZeroMQ uses smart and right enough approches, on a case by case principle.
There cases, where there are no-Queues, as you know them, at all :
for example, the inproc:// transport-class has no Queue, as one may know 'em, as there are just memory-regions, across which the messages are put and read-from. The whole magic thus appears inside the special component - the Context()-instance. There the message-memory-mapping takes place. The inproc:// case is a special case, where no I/O-thread(s) are needed at all, as the whole process is purely memory-mapping based and the Context()-instance manipulates its internal states, so as to emulate both an externally provided abstraction of the Queue-alike behaviour and also the internal "Queue"-management.
Others, similarly, operate localhost-located internal Queue(s) endpoints :
for obvious reason, as the ZeroMQ is a broker-less system, there is no "central"-place and all the functionality is spread across the participating ( coordinated and cooperating ) nodes.
Either one of the Queue-ends reside inside the Context()-instances, one in the one-side's localhost-itself, the other in the "remote"-host located Context()-instance, as was declared and coordinated during building of the ZMTP/RFC-specified socket-archetype .bind()/.connect() construction. Since a positive acknowledgement was made for such a ZMTP-specific-socket, the Queue-abstraction ( implemented in a distributed-system-manner ) holds till such socket does not get .close()-ed or the Context()-instance did not get forcefully or by a chance .term()-ed.
For these reasons, the proper capacity sizings are needed, as the messages may reside inside the Context()-operated memory before it gets transported across the { network | ipc-channel }-mediated connection ( on the .send()-side ) or before being .recv()-ed ( from inside the remote Context()-instance ) by the remote application-code for any further use of the message-payload data ( a Zero-copy message-data management is possible, yet not all use-cases indeed use this mode for avoiding replicated memory-allocations and data-transfer costs ).
I'm trying to use RabbitMq in a more unconventional way (though at this point i can pick any other message queue implementation if needed). Instead of leaving Rabbit push messages to my consumers, the consumer connects to a queue and fetches a batch of N messages (during which it consumes some and possible rejects some), after which it jumps to another queue and so on. This is done for redundancy. If some consumers crash all messages are guaranteed to be consumed by some other consumer.
The problem is that I have multiple consumers and I don't want them to compete over the same queue. Is there a way to guarantee a lock on a queue? If not, can I at least make sure that if 2 consumers are connected to the same queue they don't read the same message? Transactions might help me to some degree but I've heard talk that they'll get removed from RabbitMQ.
Other architectural suggestions are welcomed too.
Thanks!
EDIT:
As pointed in the comment there's an a particularity in how I need to process the messages. They only make sense taken in groups and there's a high probability that related messages are clumped together in a queue. If for example I pull a batch of 100 messages, there's a high probability that I'll be able to do something with messages 1-3, 4-5,6-10 etc. If I fail to find a group for some messages I'll resubmit them to the queue. WorkQueue wouldn't work because it would spread messages from the same group to multiple workers that wouldn't know what to do with them.
Have you had a look at this free online book on Enterprise Integration Patterns?
It sounds like you really need a workflow where you have a batcher component before the messages get to your workers. With RabbitMQ there are two ways to do that. Either use an exchange type (and message format) that can do the batching for you, or have one queue, and a worker that sorts out batches and places each batch on its own queue. The batcher should probably also send a "batch ready" message to a control queue so that a worker can discover the existence of the new batch queue. Once the batch is processed the worker could delete the batch queue.
If you have control over the message format, you might be able to get RabbitMQ to do the batching implicitly in a couple of ways. With a topic exchange, you could make sure that the routing key on each message is of the format work.batchid.something and then a worker that learns of the existence of batch xxyzz would use a binding key like #.xxyzz.# to only consume those messages. No republishing needed.
The other way is to include a batch id in a header and use the newer headers exchange type. Of course you can also implement your own custom exchange types if you are willing to write a small amount of Erlang code.
I do recommend checking the book though, because it gives a better overview of messaging architecture than the typical worker queue concept that most people start with.
Have your consumers pull from just one queue. They will be guaranteed not to share messages (Rabbit will round-robin the messages among the currently-connected consumers) and it's heavily optimized for that exact usage pattern.
It's ready-to-use, out of the box. In the RabbitMQ docs it's called the Work Queue model. One queue, multiple consumers, with none of them sharing anything. It sounds like what you need.
You can set a channel/consumer level prefetch count to consume messages in batches. In order to re-submit messages, you should use the basic.reject AMQP method and those messages can be chosen to be requeued or forwarded to a dead letter queue. Multiple consumers trying to pull messages from the same queue is not an issue asthe AMQP basic.get method will be synchronized to handle concurrent consumers.
https://groups.google.com/forum/#!topic/rabbitmq-users/hJ8f5du-GCA
I have some requirements for a system in need of a message queue:
The subscribers shall get individual queues.
The individual queues shall NOT be deleted when the subscriber disconnects
The subscriber shall be able to reconnect to its own queue if it looses connection
Only the subscriber shall be able to use the queue assigned to it
Nice to have: the queues survive a server restart
Can RabbitMQ be used to implement this, and in that case how?
I have only recently started using Rabbit but I believe your requirements can be addressed fairly easily.
1) I have implemented specific queues for individual subscribers by having the subscriber declare the queue (and related routing key) using its machine name as part of the queue name. The exchange takes care of routing messages appropriately by way of the binding/routing keys. In my case, all subscribers get a copy of the same message posted by the publisher and an arbitrary number of subscribers can declare their own queues and start receiving messages.
2) That's pretty much standard. If you declare a queue then it will remain in the exchange, and if it is set as durable then it will survive broker restarts. In any case, your subscriber should call queue.Declare() at startup to ensure that the queue exists but in terms of the subscriber disconnecting, the queue will remain.
3) If the queue is there and a subscriber is listening to that queue by name then there's no reason why it shouldn't be able to reconnect.
4) I haven't really delved in to the security aspects of Rabbit yet. There may be a means of securing individual queues though I'll let someone else comment on this as I'm no authority.
5) See (2). Messages will also survive a restart if set as durable as they are then written to disk. This incurs a performance penalty as there's disk I/O but that's kind of what you'd expect.
So basically, yes. Rabbit can do as you ask. In terms of 'how', there are varying degrees of 'how'. Will happily try to provide you with code-level answers should you have trouble implementing any of the above. In the meantime, and if you haven't already done so, I suggest reading through the docs:
http://www.rabbitmq.com/documentation.html
HTH. Steve