I need some help to decide which FIWARE context broker (Orion-LD, Scorpio, Stellio) I should choose for a smart city architecture. There is no existing component which uses NGSI-v2.
Is there any other reason why you should choose the Orion-LD context broker besides the fact that it is the only one that supports NGSI-v2?
Is there an advantage that the Orion broker is the main component of FIWARE?
The paper "Open-Source Publish-Subscribe Systems: A Comparative Study" says the scorpio broker is the most complete system overall.
The paper "Enabling Context-Aware Data Analytics in Smart Environments: An Open Source Reference Implementation" says the Orion-LD context broker is the most extended GE.
I also saw the performance comparison where Orion-LD is way faster at small batches of messages and slower than Scorpio and Stellio with larger batches.
Any suggestions?
Thanks!
My understanding is that Orion-LD is easy to operate, only the service and MongoDB. Scorpio uses Kafka which adds an extra complexity layer, operations-wise. I don't have any reference on Stellio.
Orion-LD supports NGSIv2 as it is a fork of Orion but it is not its main focus. If you are starting fresh, you can adopt directly LD, and you are done.
Related
I hope I'll find a reply to a bad (for me) question for which i don't find an answer at the moment.
I have to synchronize information between an enterprise asset management software, based on IBM Maximo, and an external data lake. In order to achieve this target I can develop a custom API gateway useful to receive any update/insert of data in Maximo. Or, I could use FIWARE Context Broker developing a custom data model for my asset management entities (and relationship) in order to store data having benefits in terms of NGSI protocol.
Both FIWARE Context Broker or API Gateway can be extended in order to provide public services to an IoT platform or other systems of third parties. So, why shall I use FIWARE Context Broker or a custom API Gateway? What's your advice and considerations?
I put below an example of a generic architecture
Thank you so much for any reply.
Back in the days, I used to be a IBM Integration Bus (IIB) - then known as IBM WebSphere Message Broker - developer. I would develop message flows to connect various input, output and processing nodes. This development style, of course, extends to other ESB vendors too; so, this question does not lose generality.
The messaging engine for IIB is WebSphere MQ (WMQ) that provides communication in the form of messages on a queue or as topics. Together with internal logic in IIB, the nodes communicate with each other passing on messages.
A typical IIB/WMQ has well-documented HA installation mechanism too. Besides, if a message flow exposes a HTTP(S) end-point, it could do so behind a load balancer too.
Similarly, one can speak about other technologies that comprised the SOA era. Therefore, my question is, if I
develop micro-services that communicated with say, WMQ
deployed each micro-service to a container
used an ESB to orchestrate these micro-services
relied on ESB (and its ancillary technologies) for access control, traffic management, etc.
then, what do I need Istio for - apart from a 'pure containers based architecture'?
https://developer.ibm.com/integration/blog/2014/07/02/ibm-integration-bus-high-availability-overview/
https://developer.ibm.com/integration/docs/ibm-integration-bus/learn-play/an-introduction-to-ibm-integration-bus/
Istio implements the side-car pattern to be coupled to each microservice. The microservices (not necessarily but usually) are deployed in infrastructures that allow elastic scaling, in which the system is delegated the task of adjusting the number of instances of each microservice based on the scaling strategy configured. This means that the number of containers at any given moment is predictable and at the same time unknown in the short term.
Istio solves the problem of abstracting microservices from purely infrastructure tasks and leaving them to focus solely on the functional plane, and at the same time it is able to elastically scale together with the containers to which it is attached.
Delegating this task to an ESB is not impossible, but in my opinion it would introduce a fairly high complexity factor. Maybe you've found a business opportunity ;-)
The TLDR answer is that istio is more flexible and not trying to get the microservices fully dependent on istio, while the IIB stack was mostly "once you go in, you can't go out without a migration project".
IIB previously had a monolithic architecture and your IIB related links provided would help in creating a High Availability architecture. The recent offerings of ESB(any vendor) has been to deploy the ESB as a microservices. Specifically, with respect to IIB, we can run each execution group(Integration server) as a container. With this you have all kinds of advantages of a microservice architecture. Of course as mentioned, you can have these ESB microservice to do orchestration as well.
But for any Enterprise that has microservices based architecture across its various applications and not just ESB as containers, its very difficult to manage, secure, observe etc. Specially when microservices keep growing with thousands of it running in an enterprise. This is where Istio would help.
https://istio.io/docs/concepts/what-is-istio/
Can someone please post a link to examples of bigger Fiware deployments in IoT domain?
Is Santander the biggest deployment of this kind, as it has 12k sensors - which is big, but not impressive - impressive would be 12M as then you would need clustered broker to accept all these connections. I guess 12k connections can be handled with a single PC machine (no need for clustering).
I am interested in benchmarks (latency and throughput) and stability of Orion and other Fiware components, and want to know if some bigger commercial system is deployed on Fiware, or is it just in experimental phase and not suitable for professional deployments yet.
BR,
Drasko
In the FIWARE LAB Global instance of Orion we are currently processing about 160,000 instances at present.
FIWARE components are also part of private companies commercial portfolios, not only public instances for testing like the FIWARE Lab mentioned above. For instance, Orion CB and the IoT Agents are used by Telefonica Smartcity commercial product at present.
If you are interested in the components performance/scalability more than a existing deployments (that depends more on the current customers of the companies rather than the technology limits) you may check the performance tests that will be published during this year.
Cheers,
I've been reading about how the GemFire distributed data store/management/cache system performs notifications. While this reading, i had this question.
Gemfire seems to be using MBeans to create notifications during events. How different/suitable is using MBeans to create notifications instead of implementing a Listener based aproach ? (not just in GemFire but, generally)
Note: I am very new to the topic of MBean. Just with the understanding that it's main purpose is to expose resources to be managed.
CONTEXT
...topic of MBean... it's main purpose is to expose resources to be managed.
That is correct. (GemFire) Resources exposed as MBeans can both be queried and altered, depending on what the MBean exposes for the resource (e.g. Region, DiskStore, Gateway, AEQ, etc), using JMX.
GemFire's JMX interface can then be consumed by applications and tools that use the JMX API. GemFire's Gfsh (command-line shell and management tool) along with Pulse (web monitoring tool) are both examples of JMX clients and the kinds of applications you could write that use JMX.
You can also use the standard JDK tools like jconsole or jvisualvm to connect to a GemFire Manager (managing node in the cluster that federates the view of all the members in the cluster as well as the ability to control any single member from the Manager). See GemFire's section in the User Guide on Management for more details.
Contrasting that with GemFire Callbacks, callbacks (e.g. CacheListener) can be used by peer/client cache applications to register interests in certain types of events, like Region entry creation/updates, etc. Other callbacks like CacheLoaders can used to read-through to an external data source (e.g. RDBMS) on a Cache miss. Likewise, the CacheWriter can be used to 'write-through' to an external data source on a Cache (Region) create/update, or perhaps asynchronously with a AEQ/AsyncEventListener performing a 'write-behind' to the external data source.
There are many other callbacks and ways in which these callbacks can be used, but nearly all are used programmatically in an GemFire client/peer Cache application to "receive" notifications of some type.
For more details, see the GemFire User Guide on Events and Event Handling.
ANSWER
Now, when it comes to "sending" notifications, GemFire does a fair amount of distribution on your application's behalf. JMX is primarily used to send notifications about management changes... a Region was add, the eviction policy changed, a Function was deployed, etc. In contrast, GemFire sends distribution events when data changes to other members in the cluster that are interested in the event. "Interested" members typically includes other nodes in the cluster that host the same Region and have the same key/values, which need to be updated, and in certain cases atomically (in a TX) for consistency sakes.
Now, if you want to send notifications from your application, then you are better off using Spring and Spring Data GemFire to configure and access GemFire. Spring provides exceptional support for application messaging.
Of course, other options are available including JMS, which Spring provides integration support.
All and all, the events/notifications that are sent and the distribution mechanism used highly depends on the event/notification type. As well, the manner in which to be notified (JMX Notification vs. GemFire Callback) is also dependent on the type of message and purpose.
Sorry for the lengthy explanation; it is loaded/broad question and complex subject that can vary greatly depending on the use case.
Hope this helps (a little ;-)
In most of articles I have seen that the major difference between ESB and EAI is "Single Point Failure in EAI".
My Question here is :
In EAI if Hub fails are we saying that this is single point of failure. In ESB also if Bus fails we can say single point failure. Is this right? If not please briefly explain about this.
The major difference between ESB and EAI is not Single-Point-Of-Failure.
Having said that, if the ESB Bus fails then, yes, it is a point of failure. Ultimately these are just applications in your infrastructure and whether they are a single point of failure or not is dependent on their deployment (eg. clustering) and not on the underlying conceptual integration pattern.
Personally I would classify ESB (Enterprise Service Bus) as a type of EAI (Enterprise Application Integration). Many companies trying to sell you a product instead of a concept would argue differently.
ESB is just the new pattern for EAI instead of Hub-Spoke. I wouldn't get too caught up in the differences. When you dig into it they are few and far between.
Refer this comment
The ESB is the next generation of enterprise integration technology, taking over where EAI(hub-spoke) leaves off.
Smarter Endpoints : The ESB enables architectures in which more intelligence is placed at the point
where the application interfaces with the outside world. The ESB allows each endpoint to present
itself as a service using standards such as WSDL and obviates the need for a unique interface written
for each application. Integration intelligence can be deployed natively on the end-points (clients and
servers) themselves. Canonical formats are bypassed in favor of directly formatting the payload to
the targeted format. This approach effectively removes much of the complexity inherent in EAI
products.
Distributed Architecture : Where EAI is a purely hub and spoke approach, ESB is a lightweight
distributed architecture. A centralized hub made sense when each interaction among programs had
to be converted to a canonical format. An ESB, distributes much more of the
processing logic to the end points.
No integration stacks : As customers used EAI products to solve more problems, each vendor added
stacks of proprietary features wedded to the EAI product. Over time these integration stacks got
monolithic and require deep expertise to use. ESBs, in contrast, are a relatively thin layer of software
to which other processing layers can be applied using open standards. For example, if an ESB user
wants to deploy a particular business process management tool, it can be easily integrated with the
ESB using industry standard interfaces such as BPEL for coordinating business processes.
The immediate short-term advantage of the ESB approach is that it achieves the same overall effect
as the EAI(hub-spoke) approach, but at a much lower total-cost-of-ownership. These savings are realized not
only through reduced hardware and software expenses, but also via labor savings that are realized by
using a framework that is distributed and flexible.
We need to avoid it becoming a single point of failure with a clustered set up - it can be a HA cluster or a FO cluster.