My problem, shortly:
I have to write the system, what publish data by ServiceMix. The target endpoints are Web services. That is the real possibility, one or more of then stopped.
The business process requires the correct order of messages.
Because of every system has one dedicated queue, what contains the messages. The target web services has been called by a dedicated route, where the defining of endpoint happens by a dynamic router.
It works correctly, if no error, but...
If the publication doesn't success, then the messages landing in DLQ.
I understand the logic of it, but I think this problem is not only mine.
I want to leave the message in original queue, same position as the message arrived. I can stop the queue processor route, and I can restart it. This enough the handling of the output errors.
My question, How should I meet business expectations, without manual action? (manual means AcitveMQ console)
All solution are interested. (I try the separate DLQ by systems, endless re delivery number, etc.,)
If someone knows the solution, please share me.
Thank you!
Feri
#Feri You need to use JMS message delivery with client acknowledgement i.e. if your route delivers successfully to the web-service then send a positive acknowledgement to the JMS broker else send negative. So, the message is marked as dequeued , only when it is being successfully delivered else it remains in the queue. Read more about message delivery in JMS refer
http://www.javaworld.com/article/2074123/java-web-development/transaction-and-redelivery-in-jms.html
http://wso2.com/library/articles/2013/01/jms-message-delivery-reliability-acknowledgement-patterns/
Related
When we use the example tutorials given such as in
https://fiware-tutorials.readthedocs.io/en/1.0.0/time-series-data/index.html
we can perform sending data to Cratedb with no problem. However we are having difficulty to configure FiROS to subscribe to Context broker and have the Context broker notify CrateDB. Where should we focus in terms of this configuration ? The things to note:
We can see example robot sensor data generated with Gazebo simulator under FiROS
We can send an example manual dummy data via Postman messages directly to CB and perform subscriptions that can be notified to Quantum leap
somehow we can see FiROS gets subscription to CB however we cannot see that is notified to Cratedb via quantum leap
Any pointers would be much appreciated.
Thanks
If you are looking for a strategy for checking this, you can follow the paradigm described in the Debugging IoT Agents Webinar - it describes a scenario using MQTT topics rather than ROS, but the strategy is much the same.
Maximise your debug for all components - FIROS, Context Broker, QuantumLeap
Test sending a dummy measure direct into the Context Broker and check that it has arrived - this open issues around your containerization strategy (e.g. misuse of localhost)
Once you are able to send dummy messages directly, create the subscription and check that it is firing - this checks that the subscription is broad enough
Check that the subscription is being received properly - QuantumLeap could be misconfigured. Check the subscription for lastSuccess and failure
Briefly make a call to the QuantumLeap endpoints to check that data is received in the right format (e.g. make a lastN call or check the database directly.
At this point you can start connecting FIROS, ensure the appropriate topics are being monitored and check the logs.
This strategy should narrow down where in the chain of events the break is occurring and give you debug to work out what is going wrong. If necessary you could post some debug here on StackOverflow to give the community a chance to respond (at the moment it is impossible to tell where your system is breaking)
If the problem can be narrowed down to a single component, you could also consider raising an issue on the appropriate GitHub repository - e.g. FIROS or FIROS2
We use SQS for queueing use-cases in our company. All developers connect to the same queue for local development. If we're producing some messages for testing in local development, it can happen that the message is consumed on other person's locally running consumer, if that person has the app running at the same time.
How do you make sure that messages produced by one person don't end up getting lost by consumption on other person's locally running consumer. Is using different different queues for each person the only solution? Wondering what is standard followed to avoid this in the industry?
This is very open-ended IMO. Would recommend adding some context as to how you're using SQS.
But from what I could understand:
Yes, I would recommend creating queues per "developer"
OR
Although not elegant, you can maybe add an SQS message attribute (this is metadata other than message body) with a developer's username.
And each developer should then only process a message if it's meant for them. Arguably, you could also add a flag in the message itself, but, I am not sure about the constraints on your message format. Message attributes are meant to be used for these situations, where you want to know if you really need to process a message before even parsing the message body.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes
But you'll have to increase the maxReceives to a high number (so that message does not move to dead letter queue, if you have configured one). This is not exhaustive, it will just decrease the chances of your messages being deleted by someone else. Because if say, 10 people read the message and did not delete it because username was not their username, and maxReceives is 8, it will still move to DLQ and cause unnecessary confusion.
I have been fighting the same very simple problem with NServiceBus all day today. The problem is that there is lots of documentation on how to change the configuration, but almost nothing that helps me to know what configuration I need.
There are sample applications, and they work, but there is nothing explaining how they work, what limitations they have, or how to do something just a little bit different than the sample. The sample applications also present a "Hello world" type simplicity, and in any real application you need something different from the sample application, but again there is no help on how to make these changes, or the implications of configuration choices.
From all the things that are very difficult to guess from the documentation, it is the relationship between the endpoint name, the UnicastBusConfig mappings, and pub/sub persistence that is causing the most frustration right now.
Is the endpoint name the name of the MSMQ queue? Does that mean that every application has only one input queue for all message types? Does adding a mapping in UnicastBusConfig cause a subscription message to be sent to the publisher, or does it add a subscription record in subscription DB? Why can't you add the same message type more than once to UnicastBusConfig? Why can't I just subscribe to messages of a certain type without having to know which server they come from?
For someone that understands NServiceBus this probably seems so simple that it wasn't worth documenting, but for someone coming to this for the first time, it's the very simple stuff that's the most difficult to infer from the morass of low level detail.
Is the endpoint name the name of the MSMQ queue?
Yes.
Does that mean that every application has only one input queue for all message types?
Yes. Each endpoint has a single queue associated with it, so all messages for that endpoint go through the same queue.
Does adding a mapping in UnicastBusConfig cause a subscription message to be sent to the publisher, or does it add a subscription record in subscription DB?
Neither really. The UnicastBusConfig section is for setting up the relationship between types (or assemblies) and endpoints. So it doesn't actually cause a subscription to be set up (per se), but it tells the framework where the messages will be coming from (and therefore how to subscribe to them).
The actual subscription gets created when the system starts up and NSB finds a handler for a particular type of message that matches a section in the UnicastBusConfig (assuming auto-subscribing is turned on).
This also works for sending Commands--the config section lets the framework know to which endpoint to Send() a Command.
Why can't you add the same message type more than once to UnicastBusConfig?
Because a Command can have only one (logical) endpoint that handles it, and an Event can have only one (logical) endpoint that publishes it.
Why can't I just subscribe to messages of a certain type without having to know which server they come from?
This question is a bit more difficult to answer definitively, as it gets into the philosophy of having a central broker (hub and spoke) vs. bus-style architecture.
But in a nutshell, something, somewhere needs to know how to find the publisher in order to subscribe to it. Because NServiceBus does not have a central broker or routing table, it is left to the client to be configured with knowledge of the endpoints it consumes.
You might want to check out the NServiceBus documentation at http://docs.particular.net/nservicebus/, it's quite comprehensive and should provide answers to most of your questions.
Suppose in the picture below that the middle message queue fails. Senders can still get messages sent by using other message queues.
But what happens if the message queue dies after receiving the message. How does the sender know if the message was sent to the receiver or not to decide whether or not to resend on a different message queue?
Similar what happens if the receiver dies after the message queue delivers its message to it? How would the sender know whether or not its intended request had been fulfilled by the receiver or not?
As a starting point, you need to read http://en.wikipedia.org/wiki/Two_Generals%27_Problem.
This is an instance of a very famous, and very common, problem in computer-science. Technically it is considered 'solved' as we know the answer; however, the short-story is: what you are asking for is (strictly speaking) impossible. There are protocols you can devise that will allow you to achieve any level of confidence the message has (or has not) been delivered, provided that confidence is <1.0.
In practice variations on two and three-phase distributed transaction protocols are used, along with various retransmit and resynchronisation fallbacks. The specifics depend on the implementation.
Often the choice is to permit the possibility of duplicates and require the Receiver to respond appropriately. This is the choice made by TCP, which if you think about it is trying to find a reasonable answer to the same question.
I come from a web background where I only have to deal with HTTP so please excuse my ignorance.
I have an app which where clients listen for changes in a message queue which uses stomp. Previously the client only needed to listen to the relevant channels for messages telling them about changes on the server and update themselves accordingly. Simple stuff.
There is now a requirement for the client to be able to edit data and push those changes back to the server. The data on the server is already exposed via restful resources so my first thought was just to make REST put requests to change the data on the server, but then I started to wonder whether I could find a solution using messaging. I could just open up another channel which the clients could publish changes to and the server could subscribe to that channel and update itself accordingly. Implementing this would obviously be simple but I would love to have some of the potential pitfalls pointed out to me ahead of time.
I am familiar with REST so I want to ask some questions in the context of REST:
Would I map a group of queues to REST/CRUD verbs for each resource i.e. itemPostQueue, itemPutQueue, itemDeleteQueue?
What about GET's how can I request data to read using a queue?
What do I use to replace my status code mechanism to catch problems or do I just fire and forget (gulp) or use error/receipt headers in Stomp somehow?
Any answers and advise will be much appreciated.
Regards,
Chris
While I am not clear on why you must use messaging here, a few thoughts:
You could map to REST on the wire like itemPostQueue, but this would likely feel unnatural to a message-oriented person. If you are using some kind of queue with a guaranteed semantic and deliver-once built in, then go ahead and use that mechanism. For a shopping-cart example, then you could put an AddItem message on the wire, and you trust the infrastructure to deliver it once to the server.
There is no direct GET like concept here in message queuing. You can simulate it with a pair of messages, I send you a request and you send me back a response. This is much like RPC, but even further decoupled. So I send you a PublishCart request and later on, the server sends a CartContents message on a channel that the client is listening to.
Status codes are more complex, and generally fall into two camps. First are the actual queue-library messages - deal with them just as you would any normal system message. Second you may have your own messages you want to put on the wire that signal failure at some place in the chain.
One thing that messaging does do is significantly decouple your app. Unlike HTTP, where you know that something happened, with a queue, you send a letter to somebody. It may get there. The postman might drop it in the snow. The dog might eat it. If you don't get a response in some period of time, you try other means to contact your relatives, or to pull back the analogy, to contact the server. Monitoring of the health of the queue infrastructure and depth of queues and the like take on added importance, as they are the plumbing that you are now depending upon.
Good Luck