Update: the default queue in question is part of the MassTransit implementation I was researching, rending the question moot.
As I understand MassTransit, when you publish a message, the message first goes to the publishing application's default queue.
As long as a consumer is registered for that message type, does MassTransit immediately route the message to the appropriate message queue, or does it wait until a worker thread for that consumer is available?
I suppose what I really want to know is this: is the default queue a bottleneck?
I'm using MassTransit 2.7.0 with MSMQ.
First, what version of MassTransit are you talking about? And second, what transport are you using?
If you are using RabbitMQ, messages are immediately published to the exchange. Any queues that are bound to the destination exchange will immediately receive a copy of the message. If there are no queues bound, the message evaporates into the atmosphere like an exhaled breath.
When a receive endpoint is created, MassTransit will bind the endpoint queue to the exchanges that match the types of messages consumed. For instance, if a consumer for MessageA is connected to the receive endpoint, a binding is created between MyNamespace.MessageA and the input_queue specified for the receive endpoint. An intermediate exchange, with the same name as the queue, is also created and all bindings go through that matching exchange.
This makes it easy to wire tap a queue, sending a copy of every message delivered to the queue to another queue for auditing, diagnosis, etc.
If you are using MSMQ, well, that's a different story and one that is ending as MassTransit 3 does not support MSMQ (and publishing on MSMQ is just not a great story, although it works well for send/receive).
Related
We want to use ejabberd in the context of a web application having fairly unique and business rules, we'd therefore need to have every chat message (not protocol message, but message a user sends to another one) go through our web application first, and then have the web application deliver the message to ejabberd on behalf of the user (if our business rules allow the message to be sent).
The web application is also the one providing the contact lists (called rosters if I understand correctly to ejabberd). We need to be and remain the single source of truth to ease maintenance.
To us, ejabberd value added would be to deliver chat messages in near real-time to clients, and enable cool things such as presence indicators. Web clients will maintain a direct connection to ejabberd through websocket, but this connection will have to be read-only as far as chat messages are concerned, and read-write as far as presence messages are concerned.
The situation is similar with regards to audio and video calls. While this time the call per see will directly be managed by ejabberd to take advantage of built-in STURN, TURN etc... and will not need to go through our web app, we have custom business logic to manage who is able to call who, when, how often etc... (so in order words, we have custom business logic to authorize the call or not and we'd like to keep all the business logic centralized in the web app).
My question is what would be the proper hooks we'd need to look into to achieve what we are after? I spent an hour or so in the documentation, but I couldn't find what I am after so hopefully someone can provide me pointers. In an ideal world, we'd like to expose API endpoints from our web app that ejabberd hooks can hit. However, the first question is: which relevant hooks is ejabberd offering and where are these hooks documented?
Any help would be greatly appreciated, thank you!
When a client sends a packet to ejabberd, it triggers the user_send_packet hook, providing the packet and the state of the client's session process. Several modules use that hook, for example mod_service_log.
We wish to decouple the systems between 2 separate organizations (as an example: one could be a set of in house applications and the other a set of 3rd party applications). Although we could do this using REST based APIs, we wish to achieve things like temporal decoupling, scalability, reliable and durable communication, workload decoupling (through fan-out), etc. And it is for these reasons, we wish to use a message bus.
Now one could use Amazon's SNS and SQS as the message bus infrastructure, where our org would have an SNS instance which would publish to the 3rd party SQS instance. Similarly, for messages the 3rd party wished to send to us, they would post to their SNS instance, which would then publish to our SQS instance. See: Cross Account Integration with Amazon SNS
I was thinking of how one would achieve this sort of cross-organization integration using Azure Service Bus (ASB), as we are heavily invested in Azure. But, ASB doesnt have the ability to publish from one instance to another instance belonging to a different organization (or even to another instance in the same organization, not yet at least). Given this limitation, the plan is that we would give the 3rd party vendor separate sets of connections strings that would allow them to listen and process messages that we posted and also a separate set of connection strings that would let them post messages to a topic which we could then subscribe to and process.
My question is: Is this a good idea? Or would this be considered an anti-pattern? My biggest concern is the fact, that while the point of using a message bus was to achieve decoupling, the infrastructure piece of ASB is making us tightly coupled to the point that we need to communicate between the 2 organizations on not just the endpoints, but also, how the queue/topic was setup (session, or no session, duplicate detection etc) and the consumer is tightly coupled to how the sender sends messages (what was used as the session id, message id, etc).
Is this a valid concern?
Have you done this?
What other issues might I run into?
Using Azure Service Bus connections string with different Shared Access Policy for senders and receivers (Send and Listen) is intended to be used by senders and receivers with limitted permissions. Just like you intend to use it.
My biggest concern is the fact, that while the point of using a message bus was to achieve decoupling, the infrastructure piece of ASB is making us tightly coupled to the point that we need to communicate between the 2 organizations on not just the endpoints, but also, how the queue/topic was setup (session, or no session, duplicate detection etc) and the consumer is tightly coupled to how the sender sends messages (what was used as the session id, message id, etc).
The coupling always exists. You're coupled to the language you're using. Datastore technology used to persist your data. Cloud vendor you're using. This is not type of coupling I'd be worried, unless you're planning to change those on a monthly basis.
Not more specific to the communication patterns. Sessions would be a business requriement and not a coupling. If you required ordered message delivery, then what else would you do? On Amazon you'd be also "coupling" to FIFO queue to achieve order. Message ID is by no means coupling either. It's an attribute on a message. If receiver chooses to ignore it, they can. Yes, you're coupling to use BrokeredMessage/Message envelope and serialization, but how else would you send and receive messages? This, is more of a contract for partied to agree upon.
One name for the pattern for connecting service buses between organizations is "Shovel" (that's what they are called in RabbitMq)
Sometimes it is necessary to reliably and continually move messages
from a source (e.g. a queue) in one broker to a destination in another
broker (e.g. an exchange). The Shovel plugin allows you to configure a
number of shovels, which do just that and start automatically when the
broker starts.
In the case of Azure, one way to achieve "shovels" is by using logic apps, as they provide the ability to connect to ASB entities in different namespaces.
See:
What are Logic Apps
Service Bus Connectors
Video: Use Azure Enterprise Integration Services to run cloud apps at scale
I am encountering some very long response times from Exchange Online called via the EWS Managed API 2.0 in C#. I suspect I am being throttled, but I cannot find anything that lets me prove this in the Admin portal for my O365 account. I have seen in some search results that using PowerShell you can see messages indicating "micro delays" have been applied, but I'm stuck in C#/EWS, so my question is: is there anything I can look at coming back in the responses to my EWS calls that can identify if these micro delays have been applied? BTW, response times are very close to the 100 second timeout time, which is killing my code.
Thx,
Paul
100 Seconds ins't a micro delay, micro-delays are milliseconds(capped at 500 ms) and are more aimed at delaying a large volume of requests. (eg if an app is going to may 100 sequential requests a microdelay would spread the load of those request out over a greater time by punishing the app more and more and that would lower the resource load on the server). One request taking 100 seconds to fulfill is probably more to do with the request itself. Eg overuse of search filters or overcomplex search etc which my also impact throttling or if your using batching each request withing the batch could have a micro-delay applied.
EWS doesn't return metrics of throttle usage (the new REST API does give a little more information back in this regards). What you need is access to the EWS logs which has that information. Each Exchange request the EWS Managed API makes has and Client Requestid to help correlate the request to log entry there more detail in https://msdn.microsoft.com/en-us/library/office/dn720380(v=exchg.150).aspx
I was wondering how to send messages to another user using Progress 4gl. We are trying to cut down on the PA speaker where I work and I want some way to notify a certain user/users of some predefined messages. I'm not sure if this is even possible with Progress, or if there is a message queue that can be used. If anyone has any samples, ideas or has done this before, please let me know. Thanks!!
You might find this helpful:
Following on from presentations in Boston and Finland, dot.r is
pleased to announce the open source Stomp project, available
immediately.
Download from either http://www.dotr.com or
https://bitbucket.org/jmls/stomp , the dot.r stomp programs allow you
to connect your progress session to any other application or service
that is connected to the same message broker.
Open source , free message brokers that support Stomp are :
Fuse (http://fusesource.com/products/fuse-mq-enterprise/)
[a Progress company now owned by Red Hat inc]
Fuse MQ Enterprise is a standards-based, open source messaging platform
that deploys with a very small footprint. The lack of license
fees combined with high-performance, reliable messaging that can be
used with any development environment provides a solution that
supports integration everywhere
ActiveMQ (http://activemq.apache.org/)
Apache ActiveMQ (tm) is the most popular and powerful open source messaging
and Integration Patterns server. Apache ActiveMQ is fast,
supports many Cross Language Clients and Protocols, comes with
easy to use Enterprise Integration Patterns and many advanced features
while fully supporting JMS 1.1 and J2EE 1.4.
Apache ActiveMQ is released under the Apache 2.0 License
RabbitMQ
RabbitMQ is a message broker. The principal idea is pretty simple: it
accepts and forwards messages. You can think about it as a post
office: when you send mail to the post box you're pretty sure that Mr.
Postman will eventually deliver the mail to your recipient. Using this
metaphor RabbitMQ is a post box, a post office and a postman.
The major difference between RabbitMQ and the post office is the fact
that it doesn't deal with paper, instead it accepts, stores and
forwards binary blobs of data - messages.
Please feel free to log any issues on the
https://bitbucket.org/jmls/stomp issue system, and fork the project in
order to commit back all those new features that you are going to add
...
dot.r Stomp uses the permissive MIT licence
(http://en.wikipedia.org/wiki/MIT_License)
Have fun, enjoy !
Julian
I tried it -- the code is dead-simple to install and run. And peeking inside the source is a pleasure.
ApacheMQ is pretty much painless to get going. This is a really, really easy way to get going with messaging.
If you are an old character-based fossil (such as myself) you might want to skip the GUI samples. You can send a message with:
/* stompQOut.p
*
*/
dotr.Stomp.Helper.SendMessage:ToQueue("myQueue","a test message")
And receive messages with:
/* stompQIn.p
*
*/
define variable stompClient as dotr.Stomp.StompClient no-undo.
define variable msgTxt as character no-undo format "x(60)".
stompClient = new dotr.Stomp.StompClient().
stompClient:Subscribe( this-procedure ).
stompClient:SubscribeToQueue( "myQueue" ).
pause 0 before-hide.
wait-for close of this-procedure.
procedure NewStompMessage:
define input parameter stompMessage as dotr.Stomp.StompMessage no-undo.
message string( stompMessage:Body ).
end.
I would like to learn what are the scenarios/usecases/ where messaging like RabbitMQ can help consumer web applications.
Are there any specific resources to learn from?
What web applications currently are making use of such messaging schemes and how?
In general, a message bus (such as RabbitMQ, but not limited to) allows for a reliable queue of job processing.
What this means to you in terms of a web application is the ability to scale your app as demand grows and to keep your UI quick and responsive.
Instead of forcing the user to wait while a job is processed they can request a job to be processed (for example, clicking a button on a web page to begin transcoding a video file on your server) which sends a message to your bus, let's the backend service pick it up when it's turn in the queue comes up, and maybe notify the user that work has/will begin. You can then return control to the UI, so the user can continue working with the application.
In this situation, your web interface does zero heavy lifting, instead just giving the user visibility into stages of the process as you see fit (for example, the job could incrementally update database records with the state of process which you can query and display to your user).
I would assume that any web application that experiences any kind of considerable traffic would have this type of infrastructure. While there are downsides (network glitches could potentially disrupt message delivery, more complex infrastructure, etc.) the advantages of scaling your backend become increasingly evident. If you're using cloud services this type of infrastructure makes it trivial to add additional message handlers to process your jobs by subscribing to the job queue and just picking off messages to process.
I just did a Google search and came up with the following:
Reddit.com
Digg.com
Poppen.De
That should get you started, at least.