I've been reading about how the GemFire distributed data store/management/cache system performs notifications. While this reading, i had this question.
Gemfire seems to be using MBeans to create notifications during events. How different/suitable is using MBeans to create notifications instead of implementing a Listener based aproach ? (not just in GemFire but, generally)
Note: I am very new to the topic of MBean. Just with the understanding that it's main purpose is to expose resources to be managed.
CONTEXT
...topic of MBean... it's main purpose is to expose resources to be managed.
That is correct. (GemFire) Resources exposed as MBeans can both be queried and altered, depending on what the MBean exposes for the resource (e.g. Region, DiskStore, Gateway, AEQ, etc), using JMX.
GemFire's JMX interface can then be consumed by applications and tools that use the JMX API. GemFire's Gfsh (command-line shell and management tool) along with Pulse (web monitoring tool) are both examples of JMX clients and the kinds of applications you could write that use JMX.
You can also use the standard JDK tools like jconsole or jvisualvm to connect to a GemFire Manager (managing node in the cluster that federates the view of all the members in the cluster as well as the ability to control any single member from the Manager). See GemFire's section in the User Guide on Management for more details.
Contrasting that with GemFire Callbacks, callbacks (e.g. CacheListener) can be used by peer/client cache applications to register interests in certain types of events, like Region entry creation/updates, etc. Other callbacks like CacheLoaders can used to read-through to an external data source (e.g. RDBMS) on a Cache miss. Likewise, the CacheWriter can be used to 'write-through' to an external data source on a Cache (Region) create/update, or perhaps asynchronously with a AEQ/AsyncEventListener performing a 'write-behind' to the external data source.
There are many other callbacks and ways in which these callbacks can be used, but nearly all are used programmatically in an GemFire client/peer Cache application to "receive" notifications of some type.
For more details, see the GemFire User Guide on Events and Event Handling.
ANSWER
Now, when it comes to "sending" notifications, GemFire does a fair amount of distribution on your application's behalf. JMX is primarily used to send notifications about management changes... a Region was add, the eviction policy changed, a Function was deployed, etc. In contrast, GemFire sends distribution events when data changes to other members in the cluster that are interested in the event. "Interested" members typically includes other nodes in the cluster that host the same Region and have the same key/values, which need to be updated, and in certain cases atomically (in a TX) for consistency sakes.
Now, if you want to send notifications from your application, then you are better off using Spring and Spring Data GemFire to configure and access GemFire. Spring provides exceptional support for application messaging.
Of course, other options are available including JMS, which Spring provides integration support.
All and all, the events/notifications that are sent and the distribution mechanism used highly depends on the event/notification type. As well, the manner in which to be notified (JMX Notification vs. GemFire Callback) is also dependent on the type of message and purpose.
Sorry for the lengthy explanation; it is loaded/broad question and complex subject that can vary greatly depending on the use case.
Hope this helps (a little ;-)
Related
I am trying to integrate our third party application with NetSuite. I want to be able to import sales invoice details generated from our third party system (which uses REST API) into the NetSuite invoice form.
The frequency of import is not too crucial- an immediate import will be ideal, but sending data once a day is fine as well.
I want to know what I have to use to do this API integration - SuiteTalk, RESTlet or Suitelet.
I am completely new to this topic and after a few days of research, I learned that there are 3 options for an API integration with netsuite (Suitelets, restlets and suitetalk which comprises REST and SOAP based web services). I also learned that there are scheduled scripts and user events, but I'm not too clear on the idea.
I need some help identifying which integration option I should choose.
Any and all information about netsuite API integration is appreciated!
I would avoid REST/SOAP. SOAP is outdated, and REST is incomplete and difficult to use.
Suitelet's are for when you want to present your own custom UI to frontend users, like a special new kind of custom form not relevant to any particular record. Probably not what you want.
What you probably want is to design a restlet. A restlet is a way for you to setup your own custom url inside NetSuite that your program can talk to from outside NetSuite. Like a webpage. You can pass in data to the restlet either inside the URL, or inside the body of an HTTP request (e.g. like a JSON object), and you can get data back out from the body of the HTTP response.
A restlet is a part of SuiteTalk. The method of authenticating a restlet is the same for the method of authenticating a request to the REST API. So, learning about SuiteTalk is helpful. The code you use to write the restlet, SuiteScript, is the same kind of code used to write suitelets and other kinds of scripts.
So you will want to learn about SuiteTalk, and then, in particular, SuiteTalk restlets.
this is a really subjective issue.
It used to be that SOAP/SuiteTalk was a little easier in terms of infrastructure and since Netsuite's offerings are ever changing the REST/SuiteTalk might fill this space in the future.
Since Netsuite deprecated the Full Access role setting up integrations almost always involves the integrator having to provide a permissions spec. The easiest way to do that is via a Bundle. For token based authentication (TBA) there also needs to be an integration record from which you need Consumer Id and Secret Tokens.
So as of this writing the set up for SOAP/SuiteTalk and RESTLets is roughly the same. The easiest way to communicate these is with a bundle so if you are a Netsuite dev with a dev account you can set these up in a bundle and have your customer import them.
So equal so far but differences:
SOAP/Suitetalk is slow. IMO not suiteable for an interactive interface
SOAP/Suitetalk the code is all in your external app so changes to the code don't require any changes in the target account.
RESTlets can be pretty speedy. I've used these for client interactions.
Updates require re-loading your bundle or overwriting your bundle files in the target account (with the resulting havoc if an admin refreshes the bundle)
RESTlets give you access to the features of the account on which you are running so that code can run appropriate chunks For instance features such as matrix items, multi-location inventory, one-world, pick/pack/ship, volume pricing, multi-currency will all change the data model of the account your code is running against. RESTlets can detect which features are enabled; SOAP/SuiteTalk cannot.
So really the only advantage at this point that I see for SOAP/Suitetalk is that code updates don't require access to the target account.
Who is making the changes? If it is your NetSuite developers, then your options are SUITELET or RESTLET.
If its your third-party application team, they own the code and the process and do all their work sitting outside of NetSuite - your option is SUITETALK/SOAP. Of course, they need to know something about NetSuite, but your business analyst would be sufficient to support them. As of 2020.1+, there is also support for native REST APIs in addition to SOAP in case you still want to use REST, but not write your own RESTLETS.
As the above comments mention, Suitetalk does perform a little slower than calling RESTLETS. So that maybe one of the deciding factors.
You may consider SUITELETs for integration only if you want to bypass all authentication schemes, by setting the suitelet as public. Highly inadvisable though.
If the third-party application supports REST APIs, you could call them directly from within NetSuite - either from user events or from scheduled scripts.
You can also consider iPAAS platforms like Dell Boomi, Celigo, Jitterbit, etc. These are general-purpose integration platforms, and make connecting one platform to another easy, with minimal coding. If your Company is already invested in these iPAAS platforms for other enterprise applications, then the choice is that much simpler.
We wish to decouple the systems between 2 separate organizations (as an example: one could be a set of in house applications and the other a set of 3rd party applications). Although we could do this using REST based APIs, we wish to achieve things like temporal decoupling, scalability, reliable and durable communication, workload decoupling (through fan-out), etc. And it is for these reasons, we wish to use a message bus.
Now one could use Amazon's SNS and SQS as the message bus infrastructure, where our org would have an SNS instance which would publish to the 3rd party SQS instance. Similarly, for messages the 3rd party wished to send to us, they would post to their SNS instance, which would then publish to our SQS instance. See: Cross Account Integration with Amazon SNS
I was thinking of how one would achieve this sort of cross-organization integration using Azure Service Bus (ASB), as we are heavily invested in Azure. But, ASB doesnt have the ability to publish from one instance to another instance belonging to a different organization (or even to another instance in the same organization, not yet at least). Given this limitation, the plan is that we would give the 3rd party vendor separate sets of connections strings that would allow them to listen and process messages that we posted and also a separate set of connection strings that would let them post messages to a topic which we could then subscribe to and process.
My question is: Is this a good idea? Or would this be considered an anti-pattern? My biggest concern is the fact, that while the point of using a message bus was to achieve decoupling, the infrastructure piece of ASB is making us tightly coupled to the point that we need to communicate between the 2 organizations on not just the endpoints, but also, how the queue/topic was setup (session, or no session, duplicate detection etc) and the consumer is tightly coupled to how the sender sends messages (what was used as the session id, message id, etc).
Is this a valid concern?
Have you done this?
What other issues might I run into?
Using Azure Service Bus connections string with different Shared Access Policy for senders and receivers (Send and Listen) is intended to be used by senders and receivers with limitted permissions. Just like you intend to use it.
My biggest concern is the fact, that while the point of using a message bus was to achieve decoupling, the infrastructure piece of ASB is making us tightly coupled to the point that we need to communicate between the 2 organizations on not just the endpoints, but also, how the queue/topic was setup (session, or no session, duplicate detection etc) and the consumer is tightly coupled to how the sender sends messages (what was used as the session id, message id, etc).
The coupling always exists. You're coupled to the language you're using. Datastore technology used to persist your data. Cloud vendor you're using. This is not type of coupling I'd be worried, unless you're planning to change those on a monthly basis.
Not more specific to the communication patterns. Sessions would be a business requriement and not a coupling. If you required ordered message delivery, then what else would you do? On Amazon you'd be also "coupling" to FIFO queue to achieve order. Message ID is by no means coupling either. It's an attribute on a message. If receiver chooses to ignore it, they can. Yes, you're coupling to use BrokeredMessage/Message envelope and serialization, but how else would you send and receive messages? This, is more of a contract for partied to agree upon.
One name for the pattern for connecting service buses between organizations is "Shovel" (that's what they are called in RabbitMq)
Sometimes it is necessary to reliably and continually move messages
from a source (e.g. a queue) in one broker to a destination in another
broker (e.g. an exchange). The Shovel plugin allows you to configure a
number of shovels, which do just that and start automatically when the
broker starts.
In the case of Azure, one way to achieve "shovels" is by using logic apps, as they provide the ability to connect to ASB entities in different namespaces.
See:
What are Logic Apps
Service Bus Connectors
Video: Use Azure Enterprise Integration Services to run cloud apps at scale
I read "Those who would like to enjoy the binding, presentation model structuring, testing capabilities, toolkit independence, and all the other benefits of OpenDolphin, but prefer REST (or other) remoting for data access, can use OpenDolphin with the in-memory configuration"
But I could not find any further hints in the docs?
I can't rely on sticky sessions in my load balanced webserver.
Therefore I need to plugin something different for the http session state.
Is there a opendolphin config property prepared for this? If not are there any plugin points available?
since OpenDolphin and Dolphin Platform use the remote presentation model pattern to synchronize presentation models between client and server you need a state on the server. Currently this state is defined in the session. As you said it's no problem to use load balancing with sticky sessions to provide several server instances. If you need dynamic updates between the clients a distributed event bus like hazelcast will help.
Therefore I need to plugin something different for the http session
state.
What do you need? With the last version (0.8.6) of Dolphin Platform you can access the http client in the client API and provide custom headers or cookies. Will this help? Can you please tell us what you need or open an issue at the Dolphin Platform github repo?
I would like to learn what are the scenarios/usecases/ where messaging like RabbitMQ can help consumer web applications.
Are there any specific resources to learn from?
What web applications currently are making use of such messaging schemes and how?
In general, a message bus (such as RabbitMQ, but not limited to) allows for a reliable queue of job processing.
What this means to you in terms of a web application is the ability to scale your app as demand grows and to keep your UI quick and responsive.
Instead of forcing the user to wait while a job is processed they can request a job to be processed (for example, clicking a button on a web page to begin transcoding a video file on your server) which sends a message to your bus, let's the backend service pick it up when it's turn in the queue comes up, and maybe notify the user that work has/will begin. You can then return control to the UI, so the user can continue working with the application.
In this situation, your web interface does zero heavy lifting, instead just giving the user visibility into stages of the process as you see fit (for example, the job could incrementally update database records with the state of process which you can query and display to your user).
I would assume that any web application that experiences any kind of considerable traffic would have this type of infrastructure. While there are downsides (network glitches could potentially disrupt message delivery, more complex infrastructure, etc.) the advantages of scaling your backend become increasingly evident. If you're using cloud services this type of infrastructure makes it trivial to add additional message handlers to process your jobs by subscribing to the job queue and just picking off messages to process.
I just did a Google search and came up with the following:
Reddit.com
Digg.com
Poppen.De
That should get you started, at least.
What problem do MOM (Message Oriented Middleware) solve? Scalability? Integration?
In which domain are they typically used and in which domains are they typically not used?
For example, say, is Google using such solution for it's main search engine or to power GMail?
What about big websites like Walmart, eBay, FedEx (pretty much a Java shop) and buy.com (pretty much an MS shop)? Does MOM solve a need there?
Does it make any sense when you're writing a Webapp where you control the server-side and have an homogenous environment (say tens of Amazon EC2 instances all running Linux + Java JVMs) there and where the clients are, well, Web browsers?
Does it make sense for desktop apps that need to communicate with a server?
Or is it 'only' for big enterprise stuff where you typically have a happy mix of countless of different systems that needs to communicate in a way or another?
I'm a bit confused as to what they're useful for and I think that with example of where they're appropriate and where they're not appropriate I could better understand their use.
This is a great question.
The main uses of messaging are: scaling, offloading work, integration, monitoring, event handling, routing, networking, push, mobility, buffering, queueing, task sharing, alerts, management, logging, batch, data delivery, pubsub, multicast, audit, scheduling, ... and more. Basically: anything where you need data but don't want to make a database request. (Caching is another, longer story).
Another way of looking at this is to notice that many applications used to be built by assuming that users (people) would perform actions that would be fulfilled by executing a transaction on a database (including reads, writes). But today, many actions are not user-initiated. Instead they are application-initiated. For example "tell me when the book that I want to buy is in stock". The best way to solve this class of problems is with messaging of some sort. Whether you call it middleware or web push or real time salad dressing does not matter. It's all messaging.
When you enable applications to initiate or react to events, then it is much easier to scale because your architecture can be based on loosely coupled components. It is also much easier to integrate those components if your messaging is based on a stable, scalable, serviceable tool, preferably using open standard APIs and protocols.
I hope this helps. We try to maintain a list of useful links about messaging here
Please get in touch with questions and comments on any of this, we are dead easy to find.
To address your specific questions:
In which domain are they typically used and in which domains are they typically not used?
Like databases, messaging systems crop up everywhere.
For example, say, is Google using such solution for it's main search engine or to power GMail?
Google uses a lot of home grown technology, but a lot of their open source contributions and known use cases suggest that messaging is (or should be) central to some of the main services.
What about big websites like Walmart, eBay, FedEx (pretty much a Java shop) and buy.com (pretty much an MS shop)? Does MOM solve a need there?
Very much so.
An example use case is scaling web page requests. When the user makes a web request, the web server puts it onto a queue for background processing. This means that the web server can keep working while the request is processed. It also means that the web server does not need to know how the request is handled, making system maintenance, upgrade and rollback much simpler because the main parts are 'decoupled'.
So, anyway, the web request gets processed by a back end service, or possibly by many services, eg 'look up book titles', 'draw shopping cart', 'get advertisement', 'check user account'... Finally all the results get put onto another queue, ready for collection and user response by the web server. Typically the system will include a timeout of around 100ms so that any late requests just get thrown away. The user sees anything that got processed in the time interval. This is one reason why some large ecommerce sites have pages that appear to load in stages.
There are many more use cases...
Does it make any sense when you're writing a Webapp where you control the server-side and have an homogenous environment (say tens of Amazon EC2 instances all running Linux + Java JVMs) there and where the clients are, well, Web browsers?
Definitely. If you have an unknown, or unbounded, number of users, server side instances, and application latencies, then it makes sense to use messaging, even if just as a scalable substrate for non-blocking RPC.
Does it make sense for desktop apps that need to communicate with a server?
In lots of cases. One very common case is when the server pushes events to the desktop app, eg game event, tweets, price feeds in finance, system alerts....
Or is it 'only' for big enterprise stuff where you typically have a happy mix of countless of different systems that needs to communicate in a way or another?
Definitely not only for those 'legacy integration' cases but they are important too. At RabbitMQ, the biggest customers we have in terms of pure scale or message volume are cloud providers and big web application providers.
I will answer only one answer, from prior experience - take a look at this middle-ware that is employed by big companies here - middle-ware has one purpose - to glue dis-connected systems (written in disparate languages) together so that they can interact with one another and streamline the business process - Entera as I have had experience with, creates a middle layer in which the unix box using processes written in C, interact with the mainframe system (DB2, COBOL) via a front-end written in PowerBuilder (I am not naming the company!).
From the description I have given, Entera is a middle-ware which hosts a number of things - smooth integration of the flow of data regardless of the endian format, ability for different languages to talk to the middle-ware broker (a broker is a CORBA or DCE like process, that conforms to 'The Open Group) that listens on a particular port) and is specified by an IDL which makes a process appear to be local - if you understand the terminology used in Remoting under Microsoft's .NET Framework, you are not far off the mark! The middle-ware generates stubs which are linked at compile-time and manages the creation of the process, hosting it off a port, multi-threading at run-time, and also, the modern front-ends (such as .NET, Java, PowerBuilder even the unspeakable VB6...ok...VB.NET for the purists out there) can interact by opening a connection to the specified port on a particular IP address, and using the stubs generated, can interact with it directly.
Obviously, from what was described you can see how the legacy systems can have new life breathed into it and thus scalability of the process, the major downside of this is the cost factor which can run into thousdands of dollars. Big companies who uses mainframes as their back-end processing systems for billing/invoicing, who generate a huge revenue can obviously afford such an expensive product - to them it would seem like throwing pennies into a pool of water...because of the use of middle-ware which prolongs the business process, and breathe new life into it, can extend the business by a good number of years into the future without worrying about 'legacy' tag attached to it.
Incidentally, I carried this out as part of my thesis for my BSc. in Information Systems which covered this commercial front-end. There was an open source version of the middle-ware available on sourceforge called FreeDCE, but development efforts have declined or stopped.
Edit:
#cocotwo: That is exactly what middle-ware does as you said it is a plumbing tool...message oriented middle-ware is not really heard of AFAIK because I would imagine, the processes (functions) would need to be called as if they are locally visible within the application domain of the front-end to make it easy to interact with.
Using messages may have its advantages over RPC calls in that the messages are queued in a safe-holding area in the event that a network disconnection occurs - there may be some data caching going on within that aspect to allow the front-end to continue regardless...it would be useful in the instances of 'updating a status of a particular billing/invoice number' - a one-way write-data to the back-end via the middle-ware.
Ok, big companies would have advanced systems infrastructure in that technicians are constantly around the clock to ensure a smooth delivery of data-flow so that would have to be factored in. The company that I worked with had IBM Global Support contract to fulfill in order to ensure a maximum uptime 99% with 6 nine's after the decimal point...with hot-swapping/balanced-clusters/mirroring systems in place...
Whereas with RPC, if the disconnection occurs, the front-end would have to be restarted or would have to handle the disconnection event. It really depends if the message-queueing middle-ware handles each message in real-time and pass back results to the front-end immediately...
This is where each (Message-queueing and RPC related middle-ware) have their strengths and weaknesses...and also the cost mitigation factor such as support, maximum up-time, development efforts and training - that's a biggie here as middle-ware are really proprietary (despite following the 'The Open Group' layout/standards) and complex to setup and to glue the whole thing together via scripts.
Good answers and discussion here. Our consulting team has two preferred "messaging" solutions: RabittMQ and NXTera a high speed RPC middleware, the contemporary version of Entera mentioned above. My partners and I have developed several solutions using RabittMQ, it is the best tool available in that space right now. Additionally, I happen to work for the company that makes NXTera/Entera.
From experience I can clearly say that both of these products meet the need for reliability and low maintenance as discussed above. There are situations where a messaging service, like RabittMQ, is the right choice -- where Publish and subscribe, certified delivery, Queuing or store-and-forward are required.
In other cases, RPC's (remote procedure calls) are the best and fastest solutions for transactional and distributed processing for enterprise or cloud-based applications. When it is right to use an RPC, but SOAP/.NET (yes these are RPC implementations) are too slow, expensive or complex, a lightwieght high speed RPC middleware like NXTera/Entera is the right choice for us.
There is some use case overlap between RPC middleware and message oriented middleware, and where there are you can use either successfully. But both are strong and dependable choices.
The large companies I work with use both RPC and MoM side-by-side. As far as Internet companies, Google (Protocol Buffers) and Facebook (Thrift) show that RPC's have a roll to play in modern web and cloud-based development.