What is the difference between a message channel and a message queue itself?
They're different things. The queue actually holds messages which will be processed (pushed to the listener) in FIFO manner.
A channel is a medium through which messages are transmitted.
What does that mean exactly? In a book "Enterprise Integration Patterns" it says:
Connect the applications using a Message Channel, where one application writes information to the channel and the other one reads that information from the channel.
Does this mean that this message channel actually abstracts the queue away from the producer and consumer of the message? But it really doesn't right? When a producer has to place a message into a queue, it actually specifies the queue manager and queue names it want's to connect to.
There's also the concept of different protocols in channels and different data formats in channels where you have a separate channel for each protocol you're using maybe and maybe a separate channel for each data format (XML, JSON etc).
This would facilitate the different queues to pick up from different channels. But why not directly call different queues for different data formats? What exactly is the role of the channel? Is it just a connection?
I'm a completely new at MQM. I've just been assigned to this project which involves producing and consuming messages and I'm trying to wrap my mind around this.
To expand a bit on Shashi's answer, please keep in mind that the EIP book referenced talks about high level messaging patterns. In that context the authors needed a generic term for the medium by which messages are transferred between two points and chose the word "channel".
For purposes of the book the a channel connects any two endpoints that move messages, for any message transport vendor. In this case a channel has attributes that are classes of service and support the various patterns. It may be 1-1, 1-many, many-1, many-many, etc.
So for example if it is ZeroMQ, the endpoints are two peer-to-peer nodes and there's no messaging engine between them. For IBM MQ one endpoint is always the queue manager (a type of messaging engine) and the other is an application or another queue manager.
Based on this example, it should be obvious that channel as used in the book and channel as defined by any messaging transport are at different levels of abstraction. As used by MQ, a channel is a specific set of configuration parameters that define a communication path and includes such attributes as CONNAME, MAXMSGL, tuning parameters, SSL parameters, etc. Once an MQ channel is successfully started, you can see a running instance of it by displaying the channel status. In the case of CLUSRCVR, SVRCONN, and (less commonly) RCVR or RQSTR channels, you may see multiple instances of the same channel active simultaneously.
If you are still with me, you may have noticed that the MQ usage of the term channel always describes one or more point-to-point network connections whereas the EIP book's usage of the term channel is roughly translated as "the thing that moves messages between application endpoints." Consider that two applications connected directly to the queue manager using shared memory would be using a channel as defined in EIP but not as the term is defined by IBM MQ.
Based on that example, it should be clear that the EIP version of the term channel includes the queue manager and any MQ connections between the queue manager and application endpoints.
To sum up:
The EIP book's channel is all messaging infrastructure that isn't one of the application endpoints, and in an MQ context it includes the queue manager and any MQ channels.
The IBM MQ channel is a specific configuration defining network connectivity between the queue manager and another queue manager or a client application.
I hope this clarifies the terminology rather than confusing things further. I will update based on any comments if needed.
A message queue stores messages sent by producers so that they can be delivered to consumers.
A channel is the media or communication link for transmitting messages from
producer to queue,
queue to consumer,
or one queue in a queue manager to another queue in another queue manager.
There are two types of channels:
1) A Message channel is a unidirectional communications link between two queue managers.
Message channels are used to transfer messages between the two queue managers.
2) A MQI channel connects an application (producer or consumer) to a queue manager on a server machine.
MQI channels are required to transfer MQ API calls and responses between MQ client applications and queue managers.
So,
in simple terms,
channel is a communication media between a client application and a queue manager (or between queue managers) for sending and/or receiving messages.
MQ uses a proprietary protocol to transmit messages from client applications to queue managers and between queue managers.
The format of the data contained in the message does not matter,
it can be anything including bytes, XML, or JSON.
Any type of data can be sent over the same channel.
Hope this helped.
WebSphere MQ queues
A queue is a container for messages. Business applications that are connected to the queue manager that hosts the queue can retrieve messages from the queue or can put messages on the queue. A queue has a limited capacity in terms of both the maximum number of messages that it can hold and the maximum length of those messages.
Reference
Channels
WebSphere® MQ uses two different types of channels:
A message channel, which is a unidirectional communications link between two queue managers. WebSphere MQ uses message channels to transfer messages between the queue managers. To send messages in both directions, you must define a channel for each direction.
An MQI (Message Queue Interface) channel, which is bidirectional and connects an application (MQI client) to a queue manager on a server machine. WebSphere MQ uses MQI channels to transfer MQI calls and responses between MQI clients and queue managers
Reference
Related
From messaging point of view with systems like Kafka, RabbitMQ, HornetQ, ActiveMQ, etc. is there a difference between pub/sub and prod/con or are they used interchangeably?
There is a difference between the publish/subscribe and producer/consumer models.
Publish/Subscribe : Subscribers subscribe to the publisher. Each message the Publisher publishes is sent to all the subscribers. That is, all subscribers receive the same message. (Think of a newspaper or magazine subscription. All subscribers receive the same magazine or newspaper)
Producer/Consumer : Each message the producer produces will be consumed by a single consumer. This is a mechanism to distribute the work load to multiple consumers. (Think of the several cash registers at the supermarket. Each customer goes to a single cash register. The customers are like the messages that are produced and the cash registers are the consumers)
Although both Publish/Subscribe and Producer/Consumer terms are related to messaging, they are different and can't be used interchangeably.
Publish/Subscribe is a messaging pattern where a publication is distributed to multiple receivers. A Producer is the sender of messages and consumer is the receiver of messages. Producers and consumers are an integral part of both Publish/Subscribe and Point-to-Point messaging patterns.
Hope this helps.
In producer consumer , message produced by producer is only consumed by one consumers . Even with n number of consumers each message will be consumed by only one consumer . Producing n messages to n consumers , each message will be consumed by one consumer (round robin algorithm) .
In contrast in Publisher subscriber , message produced by publisher is consumed by all subscribers that have subscribed to specific queue . Publishers produce messages while subscriber consumes message . Messages are posted in subscription queue .
As per my understanding
- Prod/Consu - Producer produces and Consumer consumes the produced task .
- Publisher/Subscriber - Publisher produces and Subscriber consumes the produced task only if Subscriber is subscribed to Publisher.
Both can be interchanged but the later one makes sense in terms of Architectural real use cases .Produce and publish only if it is subscribed based upon subscription agreement .
Thanks
I have a XBee Radio on a device that we are trying to get to communicate with another XBee Radio 5' away (attached to PC). However, there is a lot of noise on the channel and the XBee is receiving a lot of gibberish only on that particular channel. My question: Is it possible to program the channel of the Xbee not attached to the computer using the one from the computer?? Will the noise make this impossible to do over wireless and will I need a hardwired connection to the second Xbee?
I think you may have misdiagnosed your problem. A noisy channel would result in delays of sending data, but won't result in random data. The coordinator typically checks all available channels and selects the one with the least noise when establishing a network.
It's more likely that another device joined the network and is sending data. Noise will limit the XBee modules' ability to send, but won't corrupt the data sent wirelessly.
Is it possible you have the XBee module in API mode when you're expecting Transparent Serial mode (also called AT mode)? In Transparent Serial, data on the module's serial port is passed directly to a destination device (specified in ATDH and ATDL).
If you're still interested in changing channels, you can control channel selection using ATSC (Scan Channels). It's a bitmask of channels the coordinator considers when establishing a network, and channels a router or end device will use when looking for a network to join. If you needed to avoid a specific channel, you could send a remote ATSC command removing the current channel from the bitmask, then possibly an ATNR (Network Reset) command. Then do the same on the coordinator so it creates a new network on a new channel.
If you've done everything correctly, the remote device will join the newly created network on the new channel. You might need to send an ATWR (Write) command to the remote device at that point, so it stores the new ATAC setting.
Each user in my application has gmail account. the application needs to be in sync with incoming emails. for each user every 1 minute the application should ask gmail servers if there is something new. 99% of the time nothing is new.
From what I know gmail dosen't provide web-hooks
In order to reduce the load from my servers and specially from the DB I want to use the service bus queue in the following manner.
queue properties:
query method: PEEK_AND_LOCK
lock time : 1 minute
max delivery count: X
flow:
queue listener receiving message A from the queue and process it.
if nothing is new the listener will not delete the message from the queue
the message will be delivered again after lock time (1 minute)
basically instead of sending new message to the queue again and again to be processed we just leave them in the queue and relying on the re-delivery mechanism.
we are expecting many users in the near future (100,000-500,000) which means many messages in the queue in a given moment which needs to be processed each minute.
lets assume the messages size is very small and less the 5GB all together
I am assuming that the re delivery mechanism is used mainly for error handling and I wonder whether our design is reasonable and the queue is apt for that task? or if there are any other suggestions to achieve our goal.
Thanks
You are trying to use the Service Bus Queue as a scheduler which it not really is. As a result SB Queue will be main bottleneck. With your architecture and expected number of users you will find yourself quickly blocked by limitations of the Service Bus queue. For example you have only max 100 concurrent connections, which means only 100 listeners (assuming long-pooling method).
Another issue might be max delivery count property of SB Queue. Even if you set it to int.MaxValue now, there is no guarantee that Azure Team will not limit it in the future.
Alternative solution might be that you implement your own scheduler worker role (using already existing popular tools, like Quartz.NET for example). Then you can experiment - you can host N jobs (which actually do Gmail api requests) in one worker role (each job runs every X minute(s)), and each job may handle M number of users concurently. Worker role could be easily scaled if number of users increases. Numbers N and M can depend on the worker role configuration and can be determined on practice. If applicable, just to save some resources, X can be made variable, for example, based on the time of the day (maybe you don't need to check emails so often at night). Hope it helps.
We have a requirement for an API, which allows asynchronous updates via a MSMQ message queue, that I'm putting together which will allow the developer consuming the API to specify different retry policies per message. So a high priority client system, e.g. for sales will submit all messages with 5 delivery attempts (retries) and 15 minutes between each attempt, whereas a low priority client system, e.g. back-end mail shot system will allow users to update their marketing preferences, submitting messages with 3 retries and an hour between each attempt.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries, retry delay and things like whether messages are sent to a dead letter queue or just deleted? (and if so, how?)
I would be open to using other messaging frameworks if they fulfilled this requirement.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries
Depending on which operating system/msmq version you're using, specifying retry semantics is highly sophisticated in WCF. The following is for Windows 2008 and MSMQ4 using a transactional queue.
The main setting on the binding is called MaxRetryCycles. One retry cycle is an attempt to successfully read a message from a queue and process it inside the handling method. This "attempt" can actually be made up of multiple attempts, as defined by the msmq binding property ReceiveRetryCount. ReceiveRetryCount is the number of times an application will try to read the message and process it before rolling back the de-queue transaction. This marks the end of one retry cycle.
You can also introduce a delay in between cycles with the RetryCycleDelay property.
A more complicated consideration is what to do with the messages which fail even after multiple retry cycles.
allow the developer consuming the API to specify different retry policies per message
I am not sure how you could do this with MSMQ - as far as I'm aware it's only possible to set retry semantics on a per-endpoint basis. If you're using transactions then you can't even allow API users to set the priority of the messages being sent (transactional queues guarantee delivery in order).
The only thing you could do is host a another instance of your API as high-priority and one for low priority. These could be hosted on different environments, and this has the added benefit that low priority messages won't be competing for system resources with high priority messages.
Hope this helps.
I have studied Message Queues System in my class but I still don't get it how these Message Queues System work in real time scenarios? Is there any tutorial which can help me to get the complete picture?
Can someone explain me how these systems work?
An example: My thread or process can send a message to your message queue, and having sent it, my code goes on to do something else. Your code, when it gets around to it, reads the next message from the message queue, and then decides what to do about that message. Message queues avoid needing to have a critical section or mutex shared between the two threads, or processes. The underlying message queue layer itself takes care of making sure that messages get into the queue without race conditions affecting the integrity of the queue.
Message queues can be used for both one-way and two-way, asynchronous messaging. For one-way use, my thread can use it to keep your thread appraised of key events in my thread, without acknowledgement back from your thread. For two-way use, after my thread sends a message to your thread, your thread may need to send data back to my thread via my message queue.
The message queue layer uses lower level synchronization schemes to insure that no two writers to the queue can write at the same time. It insures that all writes to the queue are atomic. It also insures that a reader of the queue cannot read a partially written message from the queue.
Most message queue APIs also offer support for reading messages from the queue based on a filter that you designate. Say for instance that you consider messages from a time critical thread to be more important that other messages. You can each time you check your queue for messages, first check for messages from the critical thread, and service those messages first. Your thread would then go onto to process the rest of the messages as normal, provided no more messages from the critical thread are found.
A C tutorial of the UNIX message queues
That's a complex topic but to put it simply:
Message Queues are one of the best ways, if not the best, to
implement distributed systems.
Now you might ask, what is a distributed system? It is an integrated system that spans multiple machines, clients or nodes which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance.
Then you might ask, what is a message queue? It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for inter-node communication through the network.
And finally you might ask, what is all that good for? This is good for implementing applications with a lot of moving parts called nodes which needs real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down).
Check this article which discusses a message queue infrastructure in detail.