I'm a big fan of using message queue systems(like Apache ActiveMQ) for the tasks which are rather slow and do not require instant feedback in User Interface.
The question is: Should I use it for other tasks(which are pretty fast) and do not require instant feedback in User Interface?
Or does it in involve another level of complexity without not so much benefits?
Well, if you think about it, Win32 is more or less built around message queues. That said, just because you have a hammer, doesn't make every problem a nail. Frankly though, it depends. Queues for example do not work well towards multiple receivers.
If you're already using a MQ system in your app, I would consider moving most non-synchronous tasks there. Any fire-and-forget or event driven task is a good candidate. But then, don't overdo it either, and I certainly wouldn't consider adding the dependency if you have no other use for MQ in the application.
Is it necessary for requests to be processed in the background, independent of other components in your application? If the answer is yes, I think it warrants a queue of some kind.
If you're processing them inline in your application, that just means your application has to be running to process them. If you want them to run on their own, then you need something else to process them. I don't think adding the queue is enough extra work to justify avoid it if your message processing needs to always be running.
The right tool for the job. Is a message queue the right tool for the job you have in mind?
Related
I'm not after any code in particular but I want to know what is the most efficient way to build a function that will constantly check for updates for things such as messages e.g. Have a chat conversation window and I want live updates such as Facebook.
Currently I have implemented it by putting a while loop in my core code that checks if the view is currently visible run a Task every 5 seconds to get new messages. This works but I don't believe its the most efficient way to do it and I need to consider battery life. *Note I do change visibility when the view goes away e.g. on iOS i do
public override ViewDidDissapper {
Model.SetVisible(false)
}
Has anyone implemented some sort of polling on a cross platform app?
There are many different possible solutions here - which one you prefer depends a lot on your requirements in terms of latency, reliability, efficiency, etc - and it depends on how much you can change server side.
If your server is fixed as a normal http server, then frequent polling may be your best route forwards, although you could choose to modify the 5 seconds occasionally when you think updates aren't likely.
One step up from this is that you could try long polling http requests within your server.
Another step beyond that are using Socket (TCP, UDP or websocket) communications to provide "real time" messaging.
And in parallel to these things, you could also consider using PUSH notifications both within your app and in the background.
Overall, this is a big topic - I'd recommend reading up about PushSharp from #Redth and about SignalR from Microsoft - #gshackles has some blog posts about using this in Xamarin. Also, services like AzureMobileServices, UrbanAirship, Buddy, Parse, etc may help
A common beanstalkd workflow would be to have many workers listening for jobs on a queue/tube, locking that job while they process it, and deleting that job so that no other workers can re-process it. If the job fails (eg. resources are unavailable to complete processing) the job can slip back onto the queue for another worker to pick up the job.
Is this approach possible with ZeroMQ? Eg, using the pub/sub model can multiple subscribers receive the same job and process it at the same time? Would push/pull or req/rep provide a similar setup?
I'm certain ZeroMQ can provide this for you. However keep in mind that ZeroMQ is not really a queue. It's an advanced networking library. Naturally, with the provided primatives, you can do what you describe.
You specific case seems like it could be implemented as a pub/sub system, if you don't mind having the same work done many times over. I recommend reading the ZeroMQ guide and especially chapter 5.
Although I'm certain you can do what you describe with ZeroMQ, I would first search for a queue which does this already.
We've been using SysV Message Queue for our distributed data processing system for over 15 years. For some reason, we want to replace it with newer Message Queue mechanism. Is there any suggestions?
Requirements:
Fast response, minimizing message queue system overhead
Multiple client language library support, mainly c, c# and java
Can do some HA configuration to prevent SPOF
Have logging ability to check who sends message and who receives message
I've found Apache ActiveMQ and RabbitMQ, but it seems RabbitMQ lacks of stable C client library support?
While I have not used it personally, the toolkit from 0MQ is quite impressive.
It seems to meet all of your criteria, although #4 you would have to implement yourself, but that seems straightforward.
My question back would be why you are moving away from SysV Message Queue? The "for some reason" is a disconcerting statement.
That said, there are many excellent messaging products out there, having a useful set of selection criteria is key.
I would suggest extending your requirements list a bit, then doing website bench-marking against that list. Take the top two or three only, and do some real-world project spikes (or a bake-off if you prefer the term) to give you some actual feedback on which to base your final decision.
Good Luck
I am currently evaluating message queue systems and RabbitMq seems like a good candidate, so I'm digging a little more into it.
To give a little context I'm looking to have something like one exchange load balancing the message publishing to multiple queues. I don't want to replicate the messages, so a fanout exchange is not an option.
Also the reason I'm thinking of having multiple queues vs one queue handling the round-robin w/ the consumers, is that I don't want our single point of failure to be at the queue level.
Sounds like I could add some logic on the publisher side to simulate that behavior by editing the routing key and having the appropriate bindings in place. But that's kind of a passive approach that wouldn't take the pace of the message consumption on each queue into account, potentially leading to fill up one queue if the consumer applications for that queue are dead.
I was looking for a more pro-active way from the exchange entity side, that would decide where to send the next message based on each queue size or something of that nature.
I read about Alice and the available RESTful APIs but that seems kind of a heavy duty solution to implement fast routing decisions.
Anyone knows if round-robin between the exchange the queues is feasible w/ RabbitMQ then? Thanks.
Exchanges are generally stateless in the AMQP model, though there have been some recent experiments in stateful exchanges now that there's both a system for managing RabbitMQ plugins and for providing new experimental exchange types.
There's nothing that does quite what you want, I don't think, though I'm not completely sure I understand the requirement. Aside from the single-point-of-failure point, would having a single queue with workers reading from it solve your problem? If so, then your problem reduces to configuring RabbitMQ in an HA configuration that permits you to use that solution. There are a couple of approaches to doing that: either use HALinux and a shared store to get active/passive HA with quick failover, or set up more than one parallel broker and deduplicate on the client, perhaps using redis or similar to do so.
I suggest asking your question again on the rabbitmq-discuss mailing list, where more people will be able to offer suggestions, and where the discussion can be archived for posterity.
Agree with Tony on the approach.
Here is a 'mashup' of RabbitMQ, Redis that you could use instead of rolling your own -
http://xing.github.com/beetle/
One built in way you can do a form of sharing a form exchange to queues, but not exactly round robin, is Consistent Hashing. rabbitmq_consistent_hash_exchange
How too
https://medium.com/#eranda/rabbitmq-x-consistent-hashing-with-wso2-esb-27479b8d1d21
Paper to explain, it puts queues at a weighted distribution on a circle and then by sending random routing key it will send to the closest queue.
http://www8.org/w8-papers/2a-webserver/caching/paper2.html
I've just found myself in situation where I needed to handle exception I'll probably never get, so out of curiosity, let's do a small poll.
Do you validate the presence of resources in your programs? I mean, those resources which are installed with your program, like icons, images and similar. Generally, if those are missing, either your install didn't do its job, or the user randomly deleted files in your app.
If you do validate the presence, what do you do when the files are not there?
Of course, for web apps, you'll have nice 404 page or broken link, but what about the rest? Fail early, yes, but leave handling failures to your compiler, or what?
In Python, many folks rely on simple exception handling for this kind of thing. We might wrap an application in a Big-Old-Try-Block that reports "serious problems" for unhandleable exceptions like this and tries to clean up and exit gracefully.
It's hardly worth checking deeply in advance.
If it's even possible for the user to get to some super delicate and precious part of the application and the app dies undoing hours (or years) of work, then you should rethink that use case to create a more robust scenario where a crash is not so destructive.
Graceful Degradation:
... but no, in a lot of cases it doesn't really matter if resources are missing, that's violating the structural integrity of the program. In the case in my picture above, it's a video game where resources are loaded over the net, and sometimes (I think?) the server doesn't supply the textures needed so they are displayed as pink and black checkers. In that case it makes a bit of sense.