Can anyone explain at a beginner-intermediate level the terminology of "bus", "transport" and "endpoint" in the context of an enterprise service bus? I'm a C# developer with a few years experience now, but only just starting working with an ESB.
It seems that the "bus" is effectively a queue to which you can send and receive messages. I'm fine with that. However I'm working on some existing code using NServiceBus and I think if I grokked the "endpoint" and "transport" terminology I'd make a massive leap forward in my understanding.
Let me try to clarify those terms to you:
Bus in context of ESB architecture should not be considered as simple queue for message dispatching. To allow integration of different services, ESB provides much more. Important additional functionalities of ESB:
Routing. Messages can be routed to different services, depending on message content or endpoint specification.
Message Transformations/Mediations between different formats
Transport protocol conversion. ESB should be able to seamlessly integrate applications
that use different transport protocols (JMS, HTTP/S, pure TCP, etc.)
Message enhancement. Messages can be enriched with missing data before further processing.
Security
Management and Monitoring
Those functionalites are provided by services that operate within ESB. Services connect to each other via endpoints - uniform, unique "addresses". Messages dispatched between endpoints are using unified transport (method/protocol that encapsulates message's payload). Application that natively use different transport, need to connect to ESB via suitable adapter - service that will provide necessary transport conversion. This way applications that use ESB are decoupled from each other and don't need to provide conversions themselves.
Of course, those are only very brief descriptions of terms. Remember, Enterprise Service Bus is only catch-term for specific kind of architecture (or concept), but it is not standardized in any way. So specific implementations can be very different from each other.
If you are interested in standardized ESB, you can take a look at JBI (Java Bussiness Integration). There are several open-source implementations of JBI avalable, among them Apache ServiceMix, Mule, OpenESB. Very good introduction to ESB technologies is presented in "Open Source ESBs in Action" book published by Manning.
I would recommend looking at resources related to Enterprise Application Integration (EAI), which revolves around the ESB and various models and patterns used to integrate solutions. Think of it is a GoF for ESB architectures:
http://www.enterpriseintegrationpatterns.com/
and
http://www.enterpriseintegrationpatterns.com/toc.html
All of these patterns would give you an idea of what people use ESB's to achieve and the patterns are useful for providing common pitfalls of do-it-yourself ESB integration. I've learned an immense amount through that book and through people that source from it.
Related
Back in the days, I used to be a IBM Integration Bus (IIB) - then known as IBM WebSphere Message Broker - developer. I would develop message flows to connect various input, output and processing nodes. This development style, of course, extends to other ESB vendors too; so, this question does not lose generality.
The messaging engine for IIB is WebSphere MQ (WMQ) that provides communication in the form of messages on a queue or as topics. Together with internal logic in IIB, the nodes communicate with each other passing on messages.
A typical IIB/WMQ has well-documented HA installation mechanism too. Besides, if a message flow exposes a HTTP(S) end-point, it could do so behind a load balancer too.
Similarly, one can speak about other technologies that comprised the SOA era. Therefore, my question is, if I
develop micro-services that communicated with say, WMQ
deployed each micro-service to a container
used an ESB to orchestrate these micro-services
relied on ESB (and its ancillary technologies) for access control, traffic management, etc.
then, what do I need Istio for - apart from a 'pure containers based architecture'?
https://developer.ibm.com/integration/blog/2014/07/02/ibm-integration-bus-high-availability-overview/
https://developer.ibm.com/integration/docs/ibm-integration-bus/learn-play/an-introduction-to-ibm-integration-bus/
Istio implements the side-car pattern to be coupled to each microservice. The microservices (not necessarily but usually) are deployed in infrastructures that allow elastic scaling, in which the system is delegated the task of adjusting the number of instances of each microservice based on the scaling strategy configured. This means that the number of containers at any given moment is predictable and at the same time unknown in the short term.
Istio solves the problem of abstracting microservices from purely infrastructure tasks and leaving them to focus solely on the functional plane, and at the same time it is able to elastically scale together with the containers to which it is attached.
Delegating this task to an ESB is not impossible, but in my opinion it would introduce a fairly high complexity factor. Maybe you've found a business opportunity ;-)
The TLDR answer is that istio is more flexible and not trying to get the microservices fully dependent on istio, while the IIB stack was mostly "once you go in, you can't go out without a migration project".
IIB previously had a monolithic architecture and your IIB related links provided would help in creating a High Availability architecture. The recent offerings of ESB(any vendor) has been to deploy the ESB as a microservices. Specifically, with respect to IIB, we can run each execution group(Integration server) as a container. With this you have all kinds of advantages of a microservice architecture. Of course as mentioned, you can have these ESB microservice to do orchestration as well.
But for any Enterprise that has microservices based architecture across its various applications and not just ESB as containers, its very difficult to manage, secure, observe etc. Specially when microservices keep growing with thousands of it running in an enterprise. This is where Istio would help.
https://istio.io/docs/concepts/what-is-istio/
Could anybody explain advantage of using json-rpc over json-api and vice-versa? First and second format are JSON-based, but where I should use one, and where is another?
Note: I may come across a little biased. I am the author of the Json-RPC.net server library.
Json-RPC is a remote procedure call specification. There are multiple libraries you can use to communicate using that protocol. It is not REST based, and is transport agnostic. You can run it over HTTP as is very common, you can also use it over a socket, or any other transport you find appropriate. So it is quite flexible in that regard. You can also do server to client along with client to server requests with it by hosting the RPC server on either the client or the server.
Json-API is a specification for building REST APIs. There are multiple libraries you can use to get started with it. In contrast to Json-Rpc it requires you to host it on an HTTP server. You cannot invoke functions on the client with it. You cannot run it over a non-http transport protocol. Being REST based, it excels at providing information about Resources. If you want an API that is based around the idea of Create, Read, Update, Delete on some collections of resources, then this may be a good choice.
Json-API is going to be better if your API is resource-based, and you want your API to be browsable by a human without setting up documentation for it. Though that human would likely need to be in the software engineering field to make any sense of it.
Json-RPC is going to be better if your API is function based, or you want the flexibility it provides. Json-RPC can still be used to manipulate Resources by creating Create, Read, Update, and Delete functions for your resources, but you don't get the browsability with it not being REST based. It can still be explored (not browsed) by a human by generating documentation based off of the functions you expose.
A popular example of something that uses Json-Rpc is BitCoin.
There are a lot of popular REST-based API's and Json-API is a spec with a bunch of tools to help you do REST right.
--
Note: Neither of those (Json-RPC, or Json-API) are good when you consider for developer time, performance, or efficiently using network resources.
If you care about performance, efficiency, or developer time then take a look at Google's gRPC which is fantastic in those regards, and can still reduce developer time more than using a REST API as client and server code can be generated from a protocol definition file.
Me and my team will be working on APIGEE which is an API development platform to expose some services in our application. I am going through their documentation and also trying to understand the need of APIGEE or any other API development platform like Mashery. One very good article on the need of API proxy as been very well explained in the given link, http://apievangelist.com/2011/06/11/the-battle-for-your-api-proxy/
One question that i am confused about is What is the difference between APIGEE and any ESB like ALSB or Mule. We know Apigee too supports message transformation via policies and protocols like http/https/soap.
Can anyone please tell me the differences between the two? Does Esb support more protocols like SMTP/JMS etc.
Any information is most welcome
Though API management definition is still evolving but API management is defined as transforming APIs to reach to your target audience [ here is a good description - http://searchcloudapplications.techtarget.com/definition/API-management].
This technology has it's root in SOA but different from ESB.
ESB is more for system-to-system integration but API tends to be securely exposing your internal systems in a managed fashion to wider audience - we call them "developers".
ESB tends to be equipped with many adapters and strong message oriented middleware for supporting different interaction patterns. It is also generally couple with business process management software to automate internal processes by integrating multiple services or systems.
API management also does some integrations and orchestrations but focus is more simplifying the interfaces to easier consumption of the services - that's why it is always come with developer on-boarding capabilities, security , caching , api design , oauth etc.
Apigee gateway service [ API management platform ] has support for limited JMS and SMTP functionality serving our diverse customer base and our technology stack is capable of building other protocol support.
Many API management products [ including Apigee gateway ] also include API analytics to help you getting insight of API program and API traffic.
Nowadays, APIGEE and Mule are direct competitors in the API management offering realm. In the case of Mule, there was a great transformation on the company, towards API-ification of all systems. Current Mule runtime is integrated with strong API management capabilities including analytics (functionalities from the former Mulesoft's API Gateway product, which was merged into the Mule runtime since v3.8.0), as well the usual ESB capabilities. Further info is available at Mule dev doc site.
API Management & ESB are two different capabilities which may have little overlap in terms of exposing the integrations themselves as API's which Mulesoft does.
Apigee also supports few ESB capabilities but it's not as exhaustive as Mulesoft. Ofcourse ESB is bigger scope & if you don't need that capability & just need API management with Full API lifecyle Mulesoft & APIGee both serves that need.
Personally I am a big fan of Mulesoft's API policies which comes out of box & it's an exhaustive list.
I have few questions regarding HornetQ:
What are differences between HornetQ core API and the JMS API ?
is there any advantage or disadvantage on using one of these ?
Is it true to say if I use the core API and then I decide to
change my Messaging Bus (let's say to ActiveMQ) then I have to
change all my codes ?
HornetQ Core API is a proprietary API from HornetQ, while the JMS API is a standard API defined by the Java Community Process.
There are a few features that are not supported on JMS API, that are available through core-api:
It's not possible to have multiple consumers on a single topic subscription (say if you wanted to have multiple consumers to scale it better)
It's more generic on how you create the subscription. You just create a queue within an address. Very simple.
We have a nice API for asynchronous confirmations. No need to block ever if you use this feature.
The advantage on the JMS is portability. Your code stays the same if you decide to move between providers.
The disadvantage on JMS is that it lacks some features and it's a bit verbose, what's under works on JMS 2 JSR right now.
Yes, because as I have said here, Hornetq-core api is a proprietary API, hence it will only work on HornetQ. The same way as some message systems will have a proprietary API.
If you encapsulate your Messaging access you can minimize that a lot though where you could just replace a single class on your system using standard OO techniques.
In most of articles I have seen that the major difference between ESB and EAI is "Single Point Failure in EAI".
My Question here is :
In EAI if Hub fails are we saying that this is single point of failure. In ESB also if Bus fails we can say single point failure. Is this right? If not please briefly explain about this.
The major difference between ESB and EAI is not Single-Point-Of-Failure.
Having said that, if the ESB Bus fails then, yes, it is a point of failure. Ultimately these are just applications in your infrastructure and whether they are a single point of failure or not is dependent on their deployment (eg. clustering) and not on the underlying conceptual integration pattern.
Personally I would classify ESB (Enterprise Service Bus) as a type of EAI (Enterprise Application Integration). Many companies trying to sell you a product instead of a concept would argue differently.
ESB is just the new pattern for EAI instead of Hub-Spoke. I wouldn't get too caught up in the differences. When you dig into it they are few and far between.
Refer this comment
The ESB is the next generation of enterprise integration technology, taking over where EAI(hub-spoke) leaves off.
Smarter Endpoints : The ESB enables architectures in which more intelligence is placed at the point
where the application interfaces with the outside world. The ESB allows each endpoint to present
itself as a service using standards such as WSDL and obviates the need for a unique interface written
for each application. Integration intelligence can be deployed natively on the end-points (clients and
servers) themselves. Canonical formats are bypassed in favor of directly formatting the payload to
the targeted format. This approach effectively removes much of the complexity inherent in EAI
products.
Distributed Architecture : Where EAI is a purely hub and spoke approach, ESB is a lightweight
distributed architecture. A centralized hub made sense when each interaction among programs had
to be converted to a canonical format. An ESB, distributes much more of the
processing logic to the end points.
No integration stacks : As customers used EAI products to solve more problems, each vendor added
stacks of proprietary features wedded to the EAI product. Over time these integration stacks got
monolithic and require deep expertise to use. ESBs, in contrast, are a relatively thin layer of software
to which other processing layers can be applied using open standards. For example, if an ESB user
wants to deploy a particular business process management tool, it can be easily integrated with the
ESB using industry standard interfaces such as BPEL for coordinating business processes.
The immediate short-term advantage of the ESB approach is that it achieves the same overall effect
as the EAI(hub-spoke) approach, but at a much lower total-cost-of-ownership. These savings are realized not
only through reduced hardware and software expenses, but also via labor savings that are realized by
using a framework that is distributed and flexible.
We need to avoid it becoming a single point of failure with a clustered set up - it can be a HA cluster or a FO cluster.