I’m working with the 0.27.0 version of context broker. I'm using the Cygnus generic enabler and I have established a MQTT agent that connects external devices to the context broker.
My major concern right now is how to prevent from data loss. I established the context broker and the Cygnus mongodb databases as replica sets, but that won't ensure that all data will be persisted into the databases. I have seen that Cygnus uses Apache flume. Looking at its configuration, the re-injection retries can be configured:
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = -1
¿It is a good idea to establish the retries value to -1? I have read about events re-injected in the channel forever.
¿What can be done to ensure that all the data will be persisted?
¿Is there any functionality into fiware ecosystem oriented to that purpose?
Regarding Cygnus, the TTL is for sure the way of controlling the persistence retries after an error. A retry means the data is reinjected in the internal channel communicating the source (which receives Orion notifications) and the sink (which persists the data in the final storage) for future persistence attempts.
Possible values for this TTL are:
TTL = 0: there are no retries, i.e. if the first time a notified data cannot be persisted in the final storage (because of a network fail, a storage error, whatever) then the data is dropped.
TTL > 0: there are as much retries as configured TTL. Once exhausted the TTL the data is dropped.
TTL = -1: infinite retries, i.e. the data is reinjected in the channel forever until it is persisted or the channel gets full.
As commented, a -1 TTL may consume the channel capacity if the final storage never gets OK, avoiding new received data is put into the channel. Nevertheless, if the final storage never gets OK, such a drawback does not matter, right? :)
Thus, we could say the rules for choosing a TTL are:
If you don't want retries, simply configure 0.
If you want retries but you don't mind to loose data afeter certain number of retries, then configure a positive value.
If you want retries but you don't want to loose data, then configure -1 and a large channel capacity since the final storage may be down for an unknown time.
In any case, the TTL feature is changing during this sprint. The behaviour will be the same, but instead of being applied to single events, it will applied to batches of events (batches may be about 1 single event, of course). You'll see this change in the next release of Cygnus (0.13.0), and it will be available at the end of February 2016 (at the moment of writing this, the next week :)). My recommendation is to wait for such a release if you want to instensively use the TTL feature.
Related
Each user in my application has gmail account. the application needs to be in sync with incoming emails. for each user every 1 minute the application should ask gmail servers if there is something new. 99% of the time nothing is new.
From what I know gmail dosen't provide web-hooks
In order to reduce the load from my servers and specially from the DB I want to use the service bus queue in the following manner.
queue properties:
query method: PEEK_AND_LOCK
lock time : 1 minute
max delivery count: X
flow:
queue listener receiving message A from the queue and process it.
if nothing is new the listener will not delete the message from the queue
the message will be delivered again after lock time (1 minute)
basically instead of sending new message to the queue again and again to be processed we just leave them in the queue and relying on the re-delivery mechanism.
we are expecting many users in the near future (100,000-500,000) which means many messages in the queue in a given moment which needs to be processed each minute.
lets assume the messages size is very small and less the 5GB all together
I am assuming that the re delivery mechanism is used mainly for error handling and I wonder whether our design is reasonable and the queue is apt for that task? or if there are any other suggestions to achieve our goal.
Thanks
You are trying to use the Service Bus Queue as a scheduler which it not really is. As a result SB Queue will be main bottleneck. With your architecture and expected number of users you will find yourself quickly blocked by limitations of the Service Bus queue. For example you have only max 100 concurrent connections, which means only 100 listeners (assuming long-pooling method).
Another issue might be max delivery count property of SB Queue. Even if you set it to int.MaxValue now, there is no guarantee that Azure Team will not limit it in the future.
Alternative solution might be that you implement your own scheduler worker role (using already existing popular tools, like Quartz.NET for example). Then you can experiment - you can host N jobs (which actually do Gmail api requests) in one worker role (each job runs every X minute(s)), and each job may handle M number of users concurently. Worker role could be easily scaled if number of users increases. Numbers N and M can depend on the worker role configuration and can be determined on practice. If applicable, just to save some resources, X can be made variable, for example, based on the time of the day (maybe you don't need to check emails so often at night). Hope it helps.
From 0.8 Documentation under producer config
the property request.required.acks
value controls when the producer receives an acknowledgement from the broker.
Typical values are
(1) 0, which means that the producer never waits for an acknowledgement from the broker
(2) 1, which means that the producer gets an acknowledgement after the leader replica has received the data
(3) -1, which means that the producer gets an acknowledgement after all in-sync replicas have received the data
How do I receive this acknowledgement in producer when the request.required.acks value is 1. The producer.send(MessageKey) being a void I couldn't find any options to retrieve it.
The API for the producer send leaves much to be desired, particularly in the async mode. Those acks are hidden from the user of the producer object. If they fail, you will eventually see an exception.
The problem in the async case is that you will not know were the batch that failed began, so some guessing will be involved if you want to retry the sends later.
It seems that there are plans to improve this in future releases (> 0.8.0).
We have a requirement for an API, which allows asynchronous updates via a MSMQ message queue, that I'm putting together which will allow the developer consuming the API to specify different retry policies per message. So a high priority client system, e.g. for sales will submit all messages with 5 delivery attempts (retries) and 15 minutes between each attempt, whereas a low priority client system, e.g. back-end mail shot system will allow users to update their marketing preferences, submitting messages with 3 retries and an hour between each attempt.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries, retry delay and things like whether messages are sent to a dead letter queue or just deleted? (and if so, how?)
I would be open to using other messaging frameworks if they fulfilled this requirement.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries
Depending on which operating system/msmq version you're using, specifying retry semantics is highly sophisticated in WCF. The following is for Windows 2008 and MSMQ4 using a transactional queue.
The main setting on the binding is called MaxRetryCycles. One retry cycle is an attempt to successfully read a message from a queue and process it inside the handling method. This "attempt" can actually be made up of multiple attempts, as defined by the msmq binding property ReceiveRetryCount. ReceiveRetryCount is the number of times an application will try to read the message and process it before rolling back the de-queue transaction. This marks the end of one retry cycle.
You can also introduce a delay in between cycles with the RetryCycleDelay property.
A more complicated consideration is what to do with the messages which fail even after multiple retry cycles.
allow the developer consuming the API to specify different retry policies per message
I am not sure how you could do this with MSMQ - as far as I'm aware it's only possible to set retry semantics on a per-endpoint basis. If you're using transactions then you can't even allow API users to set the priority of the messages being sent (transactional queues guarantee delivery in order).
The only thing you could do is host a another instance of your API as high-priority and one for low priority. These could be hosted on different environments, and this has the added benefit that low priority messages won't be competing for system resources with high priority messages.
Hope this helps.
I need a high performance message bus for my application so I am evaluating performance of ZeroMQ, RabbitMQ and Apache Qpid. To measure the performance, I am running a test program that publishes say 10,000 messages using one of the message queue implementations and running another process in the same machine to consume these 10,000 messages. Then I record time difference between the first message published and the last message received.
Following are the settings I used for the comparison.
RabbitMQ: I used a "fanout" type exchange and a queue with default configuration. I used the RabbitMQ C client library.
ZeroMQ: My publisher publises to tcp://localhost:port1 with ZMQ_PUSH socket, My broker listens on tcp://localhost:port1 and resends the message to tcp://localhost:port2 and my consumer listens on tcp://localhost:port2 using ZMQ_PULL socket. I am using a broker instead of peer to to peer communication in ZeroMQ to to make the performance comparison fair to other message queue implementation that uses brokers.
Qpid C++ message broker: I used a "fanout" type exchange and a queue with default configuration. I used the Qpid C++ client library.
Following is the performance result:
RabbitMQ: it takes about 1 second to receive 10,000 messages.
ZeroMQ: It takes about 15 milli seconds to receive 10,000 messages.
Qpid: It takes about 4 seconds to receive 10,000 messages.
Questions:
Have anyone run similar performance comparison between the message queues? Then I like to compare my results with yours.
Is there any way I could tune RabbitMQ or Qpid to make it performance better?
Note:
The tests were done on a virtual machine with two allocated processor. The result may vary for different hardware, however I am mainly interested in relative performance of the MQ products.
RabbitMQ is probably doing persistence on those messages. I think you need to set the message priority or another option in messages to not do persistence. Performance will improve 10x then. You should expect at least 100K messages/second through an AMQP broker. In OpenAMQ we got performance up to 300K messages/second.
AMQP was designed for speed (e.g. it does not unpack messages in order to route them) but ZeroMQ is simply better designed in key ways. E.g. it removes a hop by connecting nodes without a broker; it does better asynchronous I/O than any of the AMQP client stacks; it does more aggressive message batching. Perhaps 60% of the time spent building ZeroMQ went into performance tuning. It was very hard work. It's not faster by accident.
One thing I'd like to do, but am too busy, is to recreate an AMQP-like broker on top of ZeroMQ. There is a first layer here: http://rfc.zeromq.org/spec:15. The whole stack would work somewhat like RestMS, with transport and semantics separated into two layers. It would provide much the same functionality as AMQP/0.9.1 (and be semantically interoperable) but significantly faster.
Hmm, of course ZeroMQ will be faster, it is designed to be and does not have a lot of the broker based functionality that the other two provide. The ZeroMQ site has a wonderful comparison of broker vs brokerless messaging and drawbacks & advantages of both.
RabbitMQ Blog:
RabbitMQ and 0MQ are focusing on different aspects of messaging. 0MQ puts much more focus on how the messages are transferred over the wire. RabbitMQ, on the other hand, focuses on how messages are stored, filtered and monitored.
(I also like the above RabbitMQ post above as it also talks about using ZeroMQ with RabbitMQ)
So, what I'm trying to say is that you should decide on the tech that best fits your requirements. If the only requirement is speed, ZeroMQ. But if you need other aspects such as persistence of messages, filtering, monitoring, failover, etc well, then that's when u need to start considering RabbitMQ & Qpid.
I am using a broker instead of peer to to peer communication in ZeroMQ to to make the performance comparison fair to other message queue implementation that uses brokers.
Not sure why you want to do that -- if the only thing you care about is performance, there is no need to make the playing field level. If you don't care about persistence, filtering, etc. then why pay the price?
I'm also very leery of running benchmarks on VM's -- there are a lot of extra layers that can affect the results in ways that are not obvious. (Unless you're planning to run the real system on VM's, of course, in which case that is a very valid method).
I've tested c++/qpid
I sent 50000 messages per second between two diferent machines for a long time with no queuing.
I didn't use a fanout, just a simple exchange (non persistent messages)
Are you using persistent messages?
Are you parsing the messages?
I suppose not, since 0MQ doesn't have message structs.
If the broker is mainly idle, you probably haven't configured the prefetch on sender and receptor. This is very important to send many messages.
We have compared RabbitMQ with our SocketPro (http://www.udaparts.com/) persistent message queue at the site http://www.udaparts.com/document/articles/fastsocketpro.htm with all source codes. Here are results we obtained for RabbitMQ:
Same machine enqueue and dequeue:
"Hello world" --
Enqueue: 30000 messages per second;
Dequeue: 7000 messages per second.
Text with 1024 bytes --
Enqueue: 11000 messages per second;
Dequeue: 7000 messages per second.
Text with 10 * 1024 bytes --
Enqueue: 4000 messages per second;
Dequeue: 4000 messages per second.
Cross-machine enqueue and dequeue with network bandwidth 100 mbps:
"Hello world" --
Enqueue: 28000 messages per second;
Dequeue: 1900 messages per second.
Text with 1024 bytes --
Enqueue: 8000 messages per second;
Dequeue: 1000 messages per second.
Text with 10 * 1024 bytes --
Enqueue: 800 messages per second;
Dequeue: 700 messages per second.
Try to configure prefetch on sender and receptor with a value like 100. Prefetching just sender is not enough
We've developed an open source message bus built on top of ZeroMQ - we initially did this to replace Qpid. It's brokerless so it's not a totally fair comparison but it provides the same functionality as brokered solutions.
Our headline performance figure is 140K msgs per second between two machines but you can see more detail here: https://github.com/Abc-Arbitrage/Zebus/wiki/Performance
I want to create a fairly simple mathematical model that describes usage patterns and performance trade-offs in a system.
The system behaves as follows:
clients periodically issue multi-cast packets to a network of hosts
any host that receives the packet, responds with a unicast answer directly
the initiating host caches the responses for some given time period, then discards them
if the cache is full the next time a request is required, data is pulled from the cache not the network
packets are of a fixed size and always contain the same information
hosts are symmetic - any host can issue a request and respond to requests
I want to produce some simple mathematical models (and graphs) that describe the trade-offs available given some changes to the above system:
What happens where you vary the amount of time a host caches responses? How much data does this save? How many calls to the network do you avoid? (clearly depends on activity)
Suppose responses are also multi-cast, and any host that overhears another client's request can cache all the responses it hears - thereby saving itself potentially making a network request - how would this affect the overall state of the system?
Now, this one gets a bit more complicated - each request-response cycle alters the state of one other host in the network, so the more activity the quicker caches become invalid. How do I model the trade off between the number of hosts, the rate of activity, the "dirtyness" of the caches (assuming hosts listen in to other's responses) and how this changes with cache validity period? Not sure where to begin.
I don't really know what sort of mathematical model I need, or how I construct it. Clearly it's easier to just vary two parameters, but particularly with the last one, I've got maybe four variables changing that I want to explore.
Help and advice appreciated.
Investigate tokenised Petri nets. These seem to be an appropriate tool as they:
provide a graphical representation of the models
provide substantial mathematical analysis
have a large body of prior work and underlying analysis
are (relatively) simple mathematical models
seem to be directly tied to your problem in that they deal with constraint dependent networks that pass tokens only under specified conditions
I found a number of references (quality not assessed) by a search on "token Petri net"