This is with regards to the use case of letting a Producer know if the message has been consumed by the Consumer.
My idea is that every time a message is consumed, Consumer commits the offset and the Producer can track and read current offsets to see if corresponding messages were consumed.
Also, please don't hesitate to let me know if this is the right way to handle the use case as I have no experience in Kafka. I know Kafka is not designed to work this way (handling the above use case) but I have to persist with Kafka.
If the question is the title, the answers is no, at least in the latest version of kafka 0.9.0.1 there is no way to know what is the offset for a partition from the producer.
Related
Our system requirements say that we need to build a slightly unusual producer-consumer processing system. Imagine we have multiple data streams and we take a snapshot each X seconds and put it into the queue for processing. The streams count is not constant. The more clients we have, the more streams we need to process. At the same time, we don't need to process ALL taken snapshots. If we have too many clients and we are not able to process all items in real-time, we would prefer to skip old snapshots and process only the latest ones.
So as I see, the requirements can be met by keeping only one item in a queue for each stream. If there is a new snapshot, while the previous is still there, we need to REPLACE it using stream id as a key.
Is it possible to implement such behavior by Service Bus queue or something similar? Or maybe it makes sense to look into some other solutions like Redis?
So as I see, the requirements can be met by keeping only one item in a
queue for each stream. If there is a new snapshot, while the previous
is still there, we need to REPLACE it using stream id as a key. Is it
possible to implement such behavior by Service Bus queue or something
similar?
To the best of my knowledge, Azure Service Bus does not support this scenario. Through it's duplicate detection functionality, in fact it supports exact opposite of that. You would need to use some other mechanism (like Redis Cache you mentioned) to accomplish this.
I have been fighting the same very simple problem with NServiceBus all day today. The problem is that there is lots of documentation on how to change the configuration, but almost nothing that helps me to know what configuration I need.
There are sample applications, and they work, but there is nothing explaining how they work, what limitations they have, or how to do something just a little bit different than the sample. The sample applications also present a "Hello world" type simplicity, and in any real application you need something different from the sample application, but again there is no help on how to make these changes, or the implications of configuration choices.
From all the things that are very difficult to guess from the documentation, it is the relationship between the endpoint name, the UnicastBusConfig mappings, and pub/sub persistence that is causing the most frustration right now.
Is the endpoint name the name of the MSMQ queue? Does that mean that every application has only one input queue for all message types? Does adding a mapping in UnicastBusConfig cause a subscription message to be sent to the publisher, or does it add a subscription record in subscription DB? Why can't you add the same message type more than once to UnicastBusConfig? Why can't I just subscribe to messages of a certain type without having to know which server they come from?
For someone that understands NServiceBus this probably seems so simple that it wasn't worth documenting, but for someone coming to this for the first time, it's the very simple stuff that's the most difficult to infer from the morass of low level detail.
Is the endpoint name the name of the MSMQ queue?
Yes.
Does that mean that every application has only one input queue for all message types?
Yes. Each endpoint has a single queue associated with it, so all messages for that endpoint go through the same queue.
Does adding a mapping in UnicastBusConfig cause a subscription message to be sent to the publisher, or does it add a subscription record in subscription DB?
Neither really. The UnicastBusConfig section is for setting up the relationship between types (or assemblies) and endpoints. So it doesn't actually cause a subscription to be set up (per se), but it tells the framework where the messages will be coming from (and therefore how to subscribe to them).
The actual subscription gets created when the system starts up and NSB finds a handler for a particular type of message that matches a section in the UnicastBusConfig (assuming auto-subscribing is turned on).
This also works for sending Commands--the config section lets the framework know to which endpoint to Send() a Command.
Why can't you add the same message type more than once to UnicastBusConfig?
Because a Command can have only one (logical) endpoint that handles it, and an Event can have only one (logical) endpoint that publishes it.
Why can't I just subscribe to messages of a certain type without having to know which server they come from?
This question is a bit more difficult to answer definitively, as it gets into the philosophy of having a central broker (hub and spoke) vs. bus-style architecture.
But in a nutshell, something, somewhere needs to know how to find the publisher in order to subscribe to it. Because NServiceBus does not have a central broker or routing table, it is left to the client to be configured with knowledge of the endpoints it consumes.
You might want to check out the NServiceBus documentation at http://docs.particular.net/nservicebus/, it's quite comprehensive and should provide answers to most of your questions.
I'm using Kafka's high-level consumer. Because I'm using Kafka as a 'queue of transactions' for my application, I need to make absolutely sure I don't miss or re-read any messages. I have 2 questions regarding this:
How do I commit the offset to zookeeper? I will turn off auto-commit and commit offset after every message successfully consumed. I can't seem to find actual code examples of how to do this using high-level consumer. Can anyone help me with this?
On the other hand, I've heard committing to zookeeper might be slow, so another way may be to locally keep track of the offsets? Is this alternative method advisable? If yes, how would you approach it?
You could first disable auto commit: auto.commit.enable=false
Then commit after fetching the message: consumer.commitOffsets(true)
There are two relevant settings from http://kafka.apache.org/documentation.html#consumerconfigs.
auto.commit.enable
and
auto.commit.interval.ms
If you want to set it such that the consumer commits the offset after each message, that will be difficult since the only setting is after a timer interval, not after each message. You will have to do some rate prediction of the incoming messages and accordingly set the time.
In general, it is not recommended to keep this interval too small because it vastly increases the read/write rates in zookeeper and zookeeper gets slowed down because it's strongly consistent across its quorum.
I've solved my problem by using:
consumerConfig.EnableAutoCommit = false;
after
var consumer = consumerBuilder.Consume(cancelToken.Token);
using
consumerBuilder.Commit(consumer);
I'm using:
Confluent.Kafka
for my C# client
I was using a jquery plugin to talk with bosh:
https://github.com/maxpowel/jQuery-XMPP-plugin
I read that i can use strophe to send data in stream mode, in order to save bandwidth.
What i need to have:
One normal channel where i exchange commands between chat computer users
Several channels to send not formatted data, but with the smallest amount of information, because the time of each message can round between 100ms
I've made a graph for better understanding.
How to do this with Strophe?
Second question is that i read about this json plugin for strophe http://code.google.com/p/prosody-modules/wiki/mod_json_streams .
Do you have already tried this? Is it stable and gives really a good win in bandwidth?
Thanks for your answer.
Best Regards,
Regarding your second question, thats just an april fool joke (note the date it was published), and also, the XEP 0295 that the page references make that clear:
NOTICE: This document is Humorous. It MAY provide amusement but SHOULD
NOT be taken seriously.
I come from a web background where I only have to deal with HTTP so please excuse my ignorance.
I have an app which where clients listen for changes in a message queue which uses stomp. Previously the client only needed to listen to the relevant channels for messages telling them about changes on the server and update themselves accordingly. Simple stuff.
There is now a requirement for the client to be able to edit data and push those changes back to the server. The data on the server is already exposed via restful resources so my first thought was just to make REST put requests to change the data on the server, but then I started to wonder whether I could find a solution using messaging. I could just open up another channel which the clients could publish changes to and the server could subscribe to that channel and update itself accordingly. Implementing this would obviously be simple but I would love to have some of the potential pitfalls pointed out to me ahead of time.
I am familiar with REST so I want to ask some questions in the context of REST:
Would I map a group of queues to REST/CRUD verbs for each resource i.e. itemPostQueue, itemPutQueue, itemDeleteQueue?
What about GET's how can I request data to read using a queue?
What do I use to replace my status code mechanism to catch problems or do I just fire and forget (gulp) or use error/receipt headers in Stomp somehow?
Any answers and advise will be much appreciated.
Regards,
Chris
While I am not clear on why you must use messaging here, a few thoughts:
You could map to REST on the wire like itemPostQueue, but this would likely feel unnatural to a message-oriented person. If you are using some kind of queue with a guaranteed semantic and deliver-once built in, then go ahead and use that mechanism. For a shopping-cart example, then you could put an AddItem message on the wire, and you trust the infrastructure to deliver it once to the server.
There is no direct GET like concept here in message queuing. You can simulate it with a pair of messages, I send you a request and you send me back a response. This is much like RPC, but even further decoupled. So I send you a PublishCart request and later on, the server sends a CartContents message on a channel that the client is listening to.
Status codes are more complex, and generally fall into two camps. First are the actual queue-library messages - deal with them just as you would any normal system message. Second you may have your own messages you want to put on the wire that signal failure at some place in the chain.
One thing that messaging does do is significantly decouple your app. Unlike HTTP, where you know that something happened, with a queue, you send a letter to somebody. It may get there. The postman might drop it in the snow. The dog might eat it. If you don't get a response in some period of time, you try other means to contact your relatives, or to pull back the analogy, to contact the server. Monitoring of the health of the queue infrastructure and depth of queues and the like take on added importance, as they are the plumbing that you are now depending upon.
Good Luck