Some of our partners are telling us that our software needs to interact with an Enterprise Service Bus. After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth. I'm just trying to get a feel for what our partners are telling us. Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
Although ESB is based on messaging, it is not "just" messaging and not just a buzzword.
So if you start with plain old async messaging, the early networks tended to be very point-to-point. You had to wire up (i.e. configure through some admin interface) each connection and each pair of destinations and if you dared to move anything around invariably something broke. Because the connection points were wired by hand these networks never achieved high connection density. The incremental cost was too high and did not scale. There was also a lot of access control and policy embedded in the topology. The lack of connection density actually favors this approach to security, even though it inhibits flexibility.
The ESB attempts to address these issues with...
Run-time resolution of destinations/services/resources
Location transparency
Any-to-any connectivity and maximum connection density
Architected for redundancy, horizontal scalability, failover
Policy, access control, rules externalized from topology
Logical messaging network layer implemented atop the physical messaging network layer
Common namespace
So when your customer asks for ESB compatibility, they want things like the above. From an application standpoint, this also implies...
Avoiding message affinities such as requirements to process in strict sequence or to address requests only to specific nodes instead of to a generic network destination
Ability to resolve destinations dynamically at run time (i.e. add another instance of a queue and it automatically starts getting traffic, delete one and traffic routes to the remaining nodes)
Requestor and provider apps decoupled from knowing where each other "lives". Requestor makes one connection, regardless of how many services it might need to call
Authorize by policy rather than by topology
Service provider apps able to recognize and handle dupes (as per JMS spec, see "functional duplicate" due to session handling)
Ability to run multiple active instances of a service provider application
Instrument the service provider applications so you can inquire on the status of the network or perform a test without sending an actual transaction
On the other hand, if your client cannot articulate these things then they may just want to be able to check a box that says "works with the ESB."
I'll try & keep it buzzword free (but a buzz acronym may creep in).
When services/applications/mainframes/etc... want to integrate (so send messages to each other) you can end up with quite a mess. An ESB hides that mess inside of itself (or itselves) so that an organisation can pretend that there isn't a mess and that it has something manageable. It then wraps a whole load of features around this to make this box even more enticing to the senior people in an organisation who'll make the decision to buy such an expensive product. These people typically will want to introduce a large initiative which costs a lot of money to prove that they are 'doing something' and know how to spend large amounts of money. If this is an SOA initiative then vendors various will have told them that an ESB is required to make the vendors vision of what SOA is work (typically once the number of services which they might want passes a trivial number).
So an ESB is:
A vehicle for vendors to make lots of money;
A vehicle for consultants to make lots of money;
A way for senior executives (IT Directors & the like) to show they can spend lots of money;
A box to hide a mess in;
A total PITA for a technical team to work with.
After researching this a bit, my
instinct is to say that this is just
buzz speak for saying that we need to
have a platform-indpendent way to pass
messages back and forth
You are correct, partially because the term ESB is always nice word that fits well with another buzzword, legitimate or not - which is governance (i.e. helps you manage who is accessing your endpoints and reporting metrics - Metrics btw is what all the suits like to see, so that may be a contributor)
Another reason they might want a platform neutral device is so that any services they consume are always exposed as endpoints from a central location, instead of a specific machine resource. The ESB makes the actual physical endpoints of your services irrelevant to them, which they shouldn't care much about anyway, but that enables you to move services around however they will only consume the ESB Endpoint.
Apart from a centralized repository for Discovery, an ESB also makes side by side versioning of services easier. If I had a choice and my company had the budget, we would have purchased IBM's x150 appliance :(
Thirdly, a lot of more advanced buses, like SoftwareAG's product if I recall, is natively able to expose legacy data, like from data sitting on main frames as services without the need for coding via adapters
I don't know if their intent is to leverage all the benefits an ESB provides, or as you said, make it buzzword compliant.
After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth
That's about right. Sometimes an ESB will go a little bit further and include additional features like message delivery guarantees, confirmation/acknowledgement messages, and so on. The presence of an ESB also usually explicitly or implicitly creates a new protocol where none previously existed, which is another important consideration. (That is, some sort of standard or interface has to be set regarding the format of the messages.)
Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
You should always listen to your customers, even if it initially sounds silly. It's usually worth at least spending the effort to decide what's going on. Reading between the lines, what your partners probably mean is that they want a way for your service to integrate more easily with their own services and products.
An enterprise service bus handles the messaging between systems in a standard way. This allows you to communicate with the bus in the same exact way across all your platforms and the bus handles the actual translating to individual communication mechanism needed for the specific endpoint. This means you write all your code to talk to the bus using a common messaging scheme and the bus handles taking your common scheme and translating it so the endpoint understands it.
The simplest explanation is to explain what it provides:
For many years companies acquired different platforms and technologies to achieve specific functions in their business from Finance to HR. These systems needed to talk to each other to share data so middleware became the glue that allowed them to connect. Before the business knew it, they were paying for support and maint on each of these systems and the middleware. As needs in the business changed departments decided to create their own custom solutions to address special needs rather than try to make the aging solutions flexible enough to meet their needs. Before they knew it, they were paying to support and maintain the legacy systems, middleware, and custom solutions. With new laws like Sarbanes Oxley, companies need to have better information available for reporting purposes. A single view requires that they capture data from all of the systems. In addition, CIOs are now being pressured to lower costs and increase customer service. One obvious solution is the eliminate redudant systems, expensive support and maint contracts, and high cost legacy solutions which require specialists to support. Moving to a new platform allows for this, but there needs to be a transition. There are no turnkey solutions that can replicate what the business does. To address the needs for moving information around they go with SOA because it allows for information access through a generic entity. If I ask for AllEmployees from the service bus it gets them whether it is from 15 HR systems or 1. When the 15 HR systems becomes 1 system the call and result does not change, just how it was done behind the scenes. The Service Bus concept standardizes the flow of information and allows IT managers to conduct transitions behind the bus with no long term effect on upstream users.
Related
Why does the `"motivational poster" for the Interface Segregation Principle on this page say "You want me to plug this in, where?"
The Interface Segregation Principle says
Clients should not be forced to depend upon interfaces that they don't
use.
I am not sure how the image and the tagline connect to this principle. It seems to be more of a motto --- albeit vaguely --- for the dependency inversion principle where high level objects should not depend on low level implementations (the plugpoint does not need to know the details of the device being plugged in)
I am aware that these sayings are tongue in cheek, but given how cutely the other
saying illustrate the principles, I don't want to misunderstand this particular poster for the ISP.
If important process A has a reference to not so important process B, you have now made process A dependent upon process B even though process A doesn't really need process B for anything.
"Clients should not be forced to depend upon interfaces they don't use" should read - Clients, classes, modules, systems, people should not know about, have references to, have dependencies on other things that are not needed to meet their functional requirement.
So let's say that you have a high level important thing like a switch/button to launch some nukes. If you plug a coffeemaker into it, then you run the chance of blowing everything up because you made your important missile launcher depend upon whether your coffeemaker is working properly.
Here's a real life example:
A Web client communicates with a web server.
The web server communicates with the database.
If the web client were able to access the database directly, then what would happen if the client goes haywire and sends 1000 requests per second?
The web server in this case acts as the Interface by mediating the communication between the client and the database. Both the client and the database depend upon, know about the web server but not each other. Because the client doesn't know about(is not plugged into) the database, the database can't be screwed over by the client. Why should the database be dependent upon the client? It works just fine on its own.
In the poster, that button looks pretty important. So as a SOLID programmer, you need to say NO when someone asks you to plug a coffeemaker or anything that is not required into a missile launcher.
The poster seems to imply that fat interfaces are more difficult to comprehend and use, which is true, but isn't the point of Interface Segregation.
The ISP is concerned with one aspect of managing coupling and cohesion in an application. Small, cohesive APIs avoid coupling different clients to each other by only serving one client each. In that respect, Interface Segregation overlaps with Single Responsibility, so I find the giant Swiss Army Knife poster for Single Responsibility failure makes equal sense in portraying Interface Segregation failure.
A user who is coupled to some tools they don't need pays the cost for those tools anyway, particularly in the effect those unnecessary tools have on overall design. The useful tools must accommodate the useless ones, and this accommodation affects the user experience.
In software terms, this can be as simple as the useless part of an API pulling in third-party libraries the user doesn't need.
See the tag info for a little more background.
Every time I read about how web service should communicate the first thing that comes up is:
Use REST, because it decouples client and server!
I would like to build a web service where each Query and Command is an Http-Endpoint. With REST I would have fewer endpoints, because of its nature of thinking of resources instead of operations (You typically have more operations than resources).
Why do I have a stronger coupling by using RPC over REST?
What is the benefit of using REST over RPC Json-Messaging style?
Additional information: With messaging I mean synchronous messaging (request/response)
Update: I think it would be also possible/better to only have one single Http endpoint that can handle a Query or Command depending on the given Http verb.
Before I get to the CQRS part, I'd like to spend a little time talking about the advantages and disadvantages of REST. I don't think it's possible to answer the question before we've established a common understanding of when and why to use REST.
REST
As with most other technology options, REST, too, isn't a silver bullet. It comes with advantages and disadvantages.
I like to use Richardson's Maturity Model, with Martin Fowler's additional level 0, as a thinking tool.
Level 0
Martin Fowler also calls level 0 the swamp of POX, but I think that what really distinguishes this level is simply the use of RPC over HTTP. It doesn't have to be XML; it could be JSON instead.
The primary advantage at this level is interoperability. Most system can communicate via HTTP, and most programming platforms can handle XML or JSON.
The disadvantage is that systems are difficult to evolve independently of clients (see level 3).
One of the distinguishing traits of this style is that all communication goes through a single endpoint.
Level 1
At level 1, you start to treat various parts of your API as separate resources. Each resource is identified by a URL.
One advantage is that you can now start to use off-the-shelf software, such as firewalls and proxy servers, to control access to various distinct parts of the system. You can also use HTTP redirects to point clients to different endpoints, although there are some pitfalls in that regard.
I can't think of any disadvantages, apart from those of level 0.
Level 2
At this level, not only do you have resources, but you also use HTTP verbs, such as GET, POST, DELETE, etc.
One advantage is that you can now begin to take more advantage of HTTP infrastructure. For instance, you can instruct clients to cache responses to GET requests, whereas other requests typically aren't cacheable. Again, you can use standard HTTP firewalls and proxies to implement caching. You can get 'web-scale' caching for free.
The reason that level 2 builds on level 1 is that you need each resource to be separate, because you want to be able to cache resources independently from each other. You can't do this if you can't distinguish various resources from each other, or if you can't distinguish reads from writes.
The disadvantage is that it may involve more programming work to implement this. Also, all the previous disadvantages still apply. Clients are tightly coupled to your published API. If you change your URL structure, clients break. If you change data formats, clients break.
Still, many so-called REST APIs are designed and published at this level, so in practice it seems that many organisations find this a good trade-off between advantages and disadvantages.
Level 3
This is the level of REST design that I consider true REST. It's nothing like the previous levels; it's a completely different way to design APIs. In my mind, there's a hard divide between levels 0-2, and level 3.
One distinguishing feature of level 3 is that you must think content negotiation into the API design. Once you have that, though, the reasons to choose this API design style become clearer.
To me, the dominant advantage of level 3 APIs is that you can evolve them independently of clients. If you're careful, you can change the structure, even the navigation graph, of your API without breaking existing clients. If you need to introduce breaking changes, you can use content negotiation to ensure that clients can opt-in to the breaking change, whereas legacy clients will keep working.
Basically, when I'm asked to write an API where I have no control over clients, my default choice is level 3.
Designing a level 3 REST API requires you to design in a way that's unusual and alien to many, so that's a disadvantage. Another disadvantage is that client developers often find this style of API design unfamiliar, so they often try to second-guess, or retro-engineer, your URL structure. If they do, you'll have to expend some effort to prevent them from doing that as well, since this will prevent you from being able to evolve the API.
In other words, level 3 APIs require considerable development effort, particularly on the server-side, but clients also become more complex.
I will, though, reiterate the advantage: you can evolve an level 3 REST API independently of clients. If you don't control clients, backwards compatibility is critical. Level 3 enables you to evolve APIs while still retaining compatibility. I'm not aware of a way you can achieve this with any of the other styles.
CQRS
Now that we've identified some advantages and disadvantages of REST, we can start to discuss whether it's applicable to CQRS.
The most fundamental agreement between Greg Young and Udi Dahan concerning CQRS is that it's not a top-level architecture.
In a nutshell, the reason for this is that the messages (commands and events) and queries that make up a CQRS system are sensitive to interpretation. In order to do something, a client must know which command to issue, and the server must know how to interpret it. The command, thus, is part of the system.
The system may be distributed across clients and servers, but the messages and data structures are coupled to each other. If you change how your server interprets a given message, that change will impact your clients. You can't evolve clients and servers independently in a CQRS architecture, which is the reason why it's not a top-level architecture.
So, given that it's not a top-level architecture, the transport architecture becomes fairly irrelevant. In a sense, the only thing you need in order to send messages is a single 'service bus' endpoint, which could easily be a level 0 endpoint, if all you need is interoperability. After all, the only thing you do is to put a message on a queue.
The final answer, then, is, as always: it depends.
Is speed of delivery the most important criterion? And can you control all clients and servers at the same time? Then, perhaps level 0 is all you need. Perhaps level 2 is fine.
On the other hand, if you have clients out of your control (mobile apps, hardware (IoT), business partners using your public API, etc.), you must consider how to deal with backwards and forwards compatibility, in which case level 3 is (IMO) required. In that case, though, I'd suggest keeping CQRS an implementation detail.
The best answer would probably be "it depends", but one of the big things about real REST is that it is stateless. And it being stateless means all sorts of things.
Basically, if you look at the HATEOAS constraint you have the reason for the decoupling 1
I think that RPC-JSON is not statefull per se, but it definately is not defined as stateless. So there you have the strongest argument of why the decoupling is very high with REST.
1: https://en.wikipedia.org/wiki/HATEOAS , http://restfulapi.net/hateoas/
What reasons are there against (or for) using the features of an Enterprise Service Bus when building an overall service adhering to a micro-service architecture (http://martinfowler.com/articles/microservices.html)? Why should we use dumb pipes and smart endpoints as opposed to using smarter pipes and be able to develop simpler services?
This is a huge question and probably can't be answered effectively in SO's Q&A format.
It depends what you are doing with it.
If you are building a single product which consists of lots of small pieces of function that can be thought of as being independent then microservices maybe the way to go.
If you are a large enterprise organisation where IT is not the main consideration of the board of directors as a competitive advantage and you work in a heavily regulate industry where new standards have to be applied across globally distributed projects with their own IT departments, some from new acquisitions, where you can't centrally control all the endpoints and applications within your organisation, then maybe you need an ESB.
I don't want to be accused of trying to list ALL the advantages of both approaches here as they wouldn't be complete and may be out of date quickly.
Having said that, in an effort to be useful to the OP:
If you look up how Spotify and Netflix do microservices you can find many things they like about the approach, including but not limited to: ease of blue/green deployment of individual services, decoupled team structures, and isolation of failures.
ESBs allow you to centrally administer and enforce policies, like legal requirements, audit everything in one place rather than hoping each team got the memo about logging everything, provide global statistics about load and uptime, as well as many other things. ESBs grew out of large enterprises where the driver was not customer response time on a website and speed of innovation (amongst other things) but Service Level Agreements, cost effectiveness and regulations (amongst other things).
There is a lot of value in both approaches. Microservices are being written about a lot at the moment, just as ESBs were 10-15 years ago. Maybe that's a progression, maybe it's just a change, maybe it's just that consumer product companies need to market themselves and large enterprises like to keep details private. We may find out in another 10 years. For now, it depends heavily on what you are doing. As with most things in programming, I'd start out simple and only move to the more complex solution if you need to.
The term ESB has gotten overloaded, primarily in the Java world, to mean a big and complex piece of infrastructure that ends up hosting a bunch of poorly implemented logic in a central place.
Lighter-weight technologies like Apache Caml or NServiceBus don't encourage this kind of approach and indeed follow the "dumb pipes / smart endpoints" approach that has served as the backbone of the internet from the beginning.
NServiceBus specifically focuses on providing a higher level framework than most messaging libraries to make it easier to build smart endpoints that are more reliable through its deeper support for once-and-only-once message processing.
Full disclosure - I'm the founder of NServiceBus.
Because services are isolated and pipes are reused.
Core idea of microservices is isolation - any part of the system can be replaced without affecting other services. Smart pipes means they have configuration, they have state, they have complex (which often means hard-to-predict) behavior. Thus, smart pipes are less likely to retain their exact behavior over time.
But - pipe change will affect every service attached while service change affects only other services that use it.
The problem with how ESB is used is that it creates a coupling between ESB and services by having some business logic built into the ESB. This would makes it more difficult to deploy a single service independently and increasingly making the ESB more complex and difficult to maintain.
I'm trying to understand if a DHT can be used to solve a problem I'm working on:
I have a trading environment where professional option traders can get an increase in their risk limit by requesting that fellow traders lend them some of their risk limit. The lending trader can either search for traders with certain risk parameters which are part of every trader's profile, i.e. Greeks, or the lending trader can subscribe to requests from certain traders who are looking for risk.
I want this environment to be scalable and decentralized, but I don't know how traders can search for specific profile parameters when the data is contained in a DHT. Could anybody explain how this can be done?
Update:
An example that might make it easier to understand might be SO, but instead of running as a web application, the Risk Exchange runs as a desktop application on each trader's workstation. The request for risk are like questions (which may be tagged by contract, exchange, etc) and each user has a profile which shows their history of requests, their return on borrowed risk, etc.
Obviously the "exchange" can be run on a server, but I was hoping to decentralize it and make it scalable so that the system may support an arbitrary number of traders. How can I search for keywords, tags, and other data pertaining to a trader's profile if this information is stored in a distributed hash table?
Your question holds a contradiction in my ears. DHT is a great way of distributing data in a decentralized manner, but cannot provide the nodes with an information overview. This means that any overview action, such as questioning the network for certain data, will have to be done at a centralized collection point. Solutions to this contradiction has been created, but their fault tolerance does not match a critical system such as financial trading.
So my answer would be to use a centralized server to hold an overview cache of the DHT network.
A little background: I have the opportunity to present the idea of a public API to the management of a large car sharing company in my country. Currently, the only options to book a car are a very slow web interface and a hard to reach call center. So I'm excited of the possiblity of writing my own search interface, integrating this functionality into other products and applications etc.
The problem: Due to the special nature of this company, I'll first have to get my proposal trough a comission, which is entirely made up of non-technical and rather conservative people. How do I explain the concept of an API to such an audience?
Don't explain technical details like an API. State the business problem and your solution to the business problem - and how it would impact their bottom line.
For years, sales people have based pitches on two things: Features and Benefit. Each feature should have an associated benefit (to somebody, and preferably everybody). In this case, you're apparently planning to break what's basically a monolithic application into (at least) two pieces: a front end and a back end. The obvious benefits are that 1) each works independently, so development of each is easier. 2) different people can develop the different pieces, 3) it's easier to increase capacity by simply buying more hardware.
Though you haven't said it explicitly, I'd guess one intent is to publicly document the API. This allows outside developers to take over (at least some) development of the front-end code (often for free, no less) while you retain control over the parts that are crucial to your business process. You can more easily [allow others to] add new front-end code to address new market segments while retaining security/certainty that the underlying business process won't be disturbed in the process.
HardCode's answer is correct in that you should really should concentrate on the business issues and benefits.
However, if you really feel you need to explain something you could use the medical receptionist analogue.
A medical practice has it's own patient database and appointment scheduling system used by it's admin and medical staff. This might be pretty complex internally.
However when you want to book an appointment as a patient you talk to the receptionist with a simple set of commands - 'I want an appointment', 'I want to see doctor X', 'I feel sick' and they interface to their systems based on your medical history, the symptoms presented and resource availability to give you an appointment - '4:30pm tomorrow' - in simple language.
So, roughly speaking using the receptionist is analogous to an exterior program using an API. It allows you to interact with a complex system to get the information you need without having to deal with the internal complexities.
They'll be able to understand the benefit of having a mobile phone app that can interact with the booking system, and an API is a necessary component of that. The second benefit of the API being public is that you won't necessarily have to write that app, someone else will be able to (whether or not they actually do is another question, of course).
You should explain which use cases will be improved by your project proposal. An what benefits they can expect, like customer satisfaction.