It's very straightforward on HTTP call between microservices to propagate exception to caller/front-end.
But how to propagate exception on event-driven/message queue (ie. RabbitMQ) microservice to the caller/front-end?
I would recommend Cadence Workflow which is much more powerful solution for microservice orchestration and provides exception handling propagation across long running operations out of the box.
It offers a lot of other advantages over using queues for your use case.
Built it exponential retries with unlimited expiration interval
Support for long running heartbeating operations
Ability to implement complex task dependencies. For example to implement chaining of calls or compensation logic in case of unrecoverble failures (SAGA)
Gives complete visibility into current state of the update. For example when using queues all you know if there are some messages in a queue and you need additional DB to track the overall progress. With Cadence every event is recorded.
Ability to cancel an update in flight.
See the presentation that goes over Cadence programming model.
Related
In storm we execute collector.fail(tuple) to replay the tuple. What is the similar approach in Flink?
Flink takes care of this automatically, so long as checkpointing is enabled. The underlying mechanism is described here, but in a nutshell, Flink periodically snapshots all of the cluster's state, and restores it in the event of a failure. This includes rewinding the sources and replaying events as needed to achieve exactly-once processing semantics.
Update:
Sometimes it is necessary to loop events back through some or all of the job graph. Flink's DataSet API supports this with iterations, but unfortunately, there are issues with iterations and streaming. The best available solution is to write events to a sink and re-ingest them from a source.
For Create operations it is clear that putting the message in the queue is a good idea in case the processing or creation of that entity takes longer than expected and other the other benefits queues bring.
However, for read operations that are timebound (must return to the UI in less than 3 seconds) it is not entirely clear if a queue is a good idea.
http://masstransit-project.com/MassTransit/usage/request-response.html provides a nice abstraction but it goes through the queue.
Can someone provide some suggestions as to why or why not I would use mass transit or that effect any technology like nservicebus etc for database read operation that are UI timebound?
Should I only use mass transit only for long running processes?
Request/Reply is a perfectly valid pattern for timebound operations. Transport costs in case of, for example, RabbitMQ, are very low. I measured performance of request/response using ServiceStack (which is very fast) and MassTransit. There is an initial delay with MassTransit to cache the endpoints, but apart from that the speed is pretty much the same.
Benefits here are:
Retries
Fine tuning of timeouts
Easy scaling with competing consumers
just to name the most obvious ones.
And with error handling you get your requests ending up in the error queue so there is no data loss and you can always look there to find out what and why went wrong.
Update: There is a SOA pattern that describes this (or rather similar) approach. It is called Decoupled Invocation.
I have studied Message Queues System in my class but I still don't get it how these Message Queues System work in real time scenarios? Is there any tutorial which can help me to get the complete picture?
Can someone explain me how these systems work?
An example: My thread or process can send a message to your message queue, and having sent it, my code goes on to do something else. Your code, when it gets around to it, reads the next message from the message queue, and then decides what to do about that message. Message queues avoid needing to have a critical section or mutex shared between the two threads, or processes. The underlying message queue layer itself takes care of making sure that messages get into the queue without race conditions affecting the integrity of the queue.
Message queues can be used for both one-way and two-way, asynchronous messaging. For one-way use, my thread can use it to keep your thread appraised of key events in my thread, without acknowledgement back from your thread. For two-way use, after my thread sends a message to your thread, your thread may need to send data back to my thread via my message queue.
The message queue layer uses lower level synchronization schemes to insure that no two writers to the queue can write at the same time. It insures that all writes to the queue are atomic. It also insures that a reader of the queue cannot read a partially written message from the queue.
Most message queue APIs also offer support for reading messages from the queue based on a filter that you designate. Say for instance that you consider messages from a time critical thread to be more important that other messages. You can each time you check your queue for messages, first check for messages from the critical thread, and service those messages first. Your thread would then go onto to process the rest of the messages as normal, provided no more messages from the critical thread are found.
A C tutorial of the UNIX message queues
That's a complex topic but to put it simply:
Message Queues are one of the best ways, if not the best, to
implement distributed systems.
Now you might ask, what is a distributed system? It is an integrated system that spans multiple machines, clients or nodes which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance.
Then you might ask, what is a message queue? It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for inter-node communication through the network.
And finally you might ask, what is all that good for? This is good for implementing applications with a lot of moving parts called nodes which needs real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down).
Check this article which discusses a message queue infrastructure in detail.
I studied interrupts vs cyclical polling and learnt the advantages of interrupts that don't have to wait for a poll. Polling seemed to me just like event-driven programming or at least similar to a listener and what the polling does is actually much like listening to input or output. Do you agree or did I misunderstand any crucial difference between polling (cyclical listening) and event-driven programming (also listening with so-called listeners)?
Nope, quite the contrary interrupt driven programming is pretty much what event driven programming is at the hardware level. Both interrupt driven code and event driven code waits for event before running a code, while polling will attempt to query for event whether or not one actually exists.
However, it should be noted that interrupt- and event-driven programs are generally implemented in the lower level using a form of polling; there is no truly interrupt or event driven system that does not involve some sort of polling, although usually in hardware. In the case of interrupts, the CPU actually polls the interrupt line every clock cycle, and likewise with event driven programming because restarting a paused thread involves an interrupt being raised by the source of event (usually drivers).
You can say that interrupt- and event- driven programming is a disciplined way to poll that have lots of advantage compared to actually doing polling yourself.
Polling and interrupt handling are two ways to find out about events. Neither is in contradiction with event-driven programming, which is building your program around handling of incoming events.
Both the answers are correct and addresses the original question. I am attempting to add a few points/considerations on event-driven / polling as is generally utilized from a bit higher level, from application containers.
If we assume we have a table where states of some components are written and based on the state some action needs to be taken; this could very well be designed as Q based eventing system or thread based polling system.
In the polling system in face of system/container termination/shutdown, the polling thread will find the changes in the next cycle and act on it.
In case of an event based system, if the container terminates before the event could be processes, it would result in the loss of the event. However, having said that there are transactional queues which are available and the capability comes out of the box; implementation of transaction, in general is a non-trivial process.
Both the methods are very much viable and not superior/inferior to each other. However, there are considerations for each model.
I'm evaluating possible solutions for handling a large quantity of queued messages, which must be delivered to workers at a certain date and time. The result of executing them is mostly updates to stored data, and they may or may not be originally triggered by user action.
For example, think of what you'd implement in a hypothetical large-scale StarCraft game server for storing and executing users' actions, like upgrading a building, hatching a soldier, all of which requires to be applied to the game state after several seconds or minutes after the player initiates them.
The problem is I can't seem to find the right term to name this problem area. There are several that looks similar, but different:
cron/task/job scheduler
The content of the queue is not dynamic, it's predefined.
Each task is scheduled.
message queue
The content of the queue is dynamic.
Each task is intended to be delivered immediately.
???
The content of the queue is dynamic.
Each task is scheduled.
If there are message queues that allow conditional delivery of messages, that might be it.
Summary:
What are these kind of technology called?
What are some of the solutions out there?
This just sounds like a trivial priority queue on the surface. The priority in this case is the time of completion, and you check the front of the queue to see when the next event is due. Pretty much every language comes with a priority queue or something that can easily be used as one, so I'm not sure what the actual problem is here.
Is it that you're worried about scalability, when it comes to millions of messages? Obviously 'millions' is a meaningless term - if that's millions per day, it's a trivial problem. If it's millions per second, then you can just scale horizontally, splitting the queue across multiple processes. (And the benefit of such a queue system is that this parallelization is really simple.)
I would bet that when implementing a large scale real-time strategy game server you would hit networking problems long before you start hitting problems with the message queue.
Have you tried looking at push queues by Iron.io? The content of the queue can be anything you like, and you specify a webhook to where the messages will be pushed to. You can also set a delay for each of the messages.
The webhook is static though for each queue and delay isn't always exactly on time (could be up to a minute off). If timing is more important or the ability of providing a different webhook per message is important, try looking at boomerang.io.
They say they are pretty accurate on the timing, you can provide a delay or unix timestamp for the webhook to return and that is per message. Sounds like either of those might work for you.
For StarCraft, I would use the Red Dwarf server.
For a Java EE app, I would use Quartz Scheduler.
It seems to me that a queue-based solution would be best in this case for a number of reasons:
Management. Most queuing solutions provide support for inspecting the content of queues which makes it easier to debug, easier to take action when certain threshold are exceeded, ...
Performance. You can divide workload by having multiple enqueue/dequeue processes (gives you the ability to scale out).
Prioritizing. Most queues support prioritizing of messages (probably not all messages are equally important).
...
Remaining problem is the immediate delivery of messages in the queue. You have two ways to solve this: either delay enqueuing of messages or delay execution of dequeued messages. I would go with the first approach, delayed enqueuing.
A message then has two properties: (content, delay). You provide the message to a component in your system that queues the message at the appropriate time.
I'm not sure what programming language you're using, but the MS .NET 4 framework has support for such a scenario (delayed execution of tasks).