How to design extensible software (plugin architecture)? [closed] - language-agnostic

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I need some resources that talk about how to design your software to be extensible, i.e. so that other people can write add-ons/plug-ins that adds functionality to it.
What do you recommend? Any books out there that discuss the subject?
I would prefer something that's short and to the point; a bit of theory and a bunch of concrete examples.
I'm not targeting a specific language, I want to be able to understand the core idea so that I can implement it in any language.
And for the same reason, I prefer not to do it using a framework that someone else built (unless the framework is not very high-level, i.e. doesn't hide too much), at the moment I only want to educate myself on the subject and experiment with various ways to implement it. Plus, a framework usually assumes user's knowledge about the subject.
UPDATE
I'm not asking about OOP or allowing my classes to be inherited. I'm talking about designing an application that will be deployed on a system, such that it can be extended by third-party add-ons AFTER its been deployed.
For example, Notepad++ has a plug-in architecture where you can place a .dll file in the plugins folder, and it adds functionality to the application that wasn't there, such as color-picking, or snippet insertion, or many other things (a wide range of functionality).

IF we're talking .NET, try Scripting .NET applications with VBScript over on CodeProject. Lots of concrete examples there.
Below are sites implementing various application extension techniques
ClearScript - Makes V8, VBScript and JScript available to .NET apps
CS-Script - The C# Script Engine
Plugin Architecture using C#
Opinio plugin architecture
Notes on the Eclipse Plug-in Architecture
Plug-in Architecture Framework for Beginners
Gecko plugin architecture
Fungimol plugin architecture

OSGI is a good practical example of a technical framework allowing to do what you are after.
The theory is here.
The (free!) book is there.
Extensibility and the ability to write plugin must deal with service lifecycle
adding / removing service/plugin on the spot
managing dependencies between services
managing states of services (declared, installed, started, stopped,...)
What is OSGI for ?
One of the main functions of a module is as a unit of deployment… something that we can either build or download and install to extend the functionality of our application.
You will find a good introduction here, on the central notion of service (which is related to your question, and which explain some problems around services, key component for extensibility).
Extract:
Why are services then so important if so many applications can be built without them? Well, services are the best known way to decouple software components from each other.
One of the most important aspects of services is that they significantly minimize class loading problems because they work with instances of objects, not with class names. Instances that are created by the provider, not the consumer. The reduction of the complexity is quite surprising
Not only do services minimize configuration, they also significantly reduce the number of shared packages.

Implement SOLID principles in your application.
1. Single responsibility principle: A class should have only a single responsibility (i.e. only one potential change in the software's specification should be able to affect the specification of the class
2.Open/closed principle: Software entities … should be open for extension, but closed for modification
3. Liskov substitution principle: Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program
4. Interface segregation principle: Many client-specific interfaces are better than one general-purpose interface
5. Dependency inversion principle: One should Depend upon Abstractions. Do not depend upon concretions
Stackoverflow questions:
Example of Single Responsibility Principle
Is the Open/Closed Principle a good idea?
What is the Liskov Substitution Principle?
Interface Segregation Principle- Program to an interface
What is the Dependency Inversion Principle and why is it important?

You try to reach two competing goals:
The components of your software must expose a lot of themselves, so they can be reused
The components of your software must expose very little of themselves, so they can be reused
Explanation: To encourage code reuse, you should be able to extend existing classes and call their methods. This isn't possible when the methods are declared "private" and the classes are "final" (and can't be extended). So to meet this goal, everything should be public and accessible. No private data or methods.
When you release the second version of your software, you will find that many of the ideas of version 1 were plain wrong. You need to change many interfaces or your code, method names, delete methods, break the API. If you do this, many people will turn away. So in order to be able to evolve your software, the components must not expose anything that is not absolutely necessary - at the cost of code reuse.
Example: I wanted to observe the position of the cursor (caret) in an SWT StyledText. The caret is not meant to be extended. If you do it, you'll find that the code contains checks like "is this class in the package org.eclipse.swt" and a lot of methods are private and final and whatnot. I had to copy about 28 classes out of SWT into my project just to implement this feature because everything is locked down.
SWT is a nice framework to use and hell to extend.

Of course there is the famous Open Closed Principle - http://en.wikipedia.org/wiki/Open/closed_principle

Well it depends on the language.
In C/C++ I'm pretty sure there is a loadlibrary function that allows you to open a library at runtime and invoke it's exported functions. This is typically how it's done in C/C++.
In .NET, there is Reflection, which is offers similar (but more broad) to loadlibrary. There is also entire libraries built on Reflection like Managed Extension Framework, or Mono.Addins that does most of the heavy lifting for you already.
In Java, there is also Reflection. And there is the JPF (Java Plugin Framework) which is used in stuff like Eclipse IIRC.
Depending on what language you use I could recommend some tutorial/books. I hope this was helpful.

Plugin architecture is becoming very popular for its extensibility and thus flexibility.
For c++, Apache httpd server is actually plugin based, but a concept of module is used instead. Most of apache features are implemented as modules, like cache, rewrite, load balancing, and even threading model. It is a very modular software I ever saw.
And for java, Eclipse is definitely plugin based. The core of Eclipse is an OSGI module system which manage bundles, another concept for plugin. Bundle can provide extension points on which we can build modules with less efforts. The most intricate thing in OSGI is its dynamic characteristic, which means bundles can be installed or uninstalled at runtime. No stop-the-world syndrome any more!

Since I dont have enough rep points to leave a comment, I am posting this as an answer. SharpDevelop is an IDE for developing applications in C#/VB.NET/Boo. It has a pretty impressive architecture that allows itself to be extended in a number of ways - right from new menu items to development support for whole new languages.
It uses a bit of XML configuration to act as a glue layer between a core of the IDE and the plugin implementation. It handles locating, loading and versioning of plugins out of the box. Deploying new plugins is matter of simply copying in the new xml configuration file and the required assemblies (DLLs) and restarting the application. You can read more on this in the book "Dissecting a csharp application" by the original author(s) - Christian Holm, Mike Krüger, Bernhard Spuida of the application from here. The book doesnt seem to be available on that site, but i found a copy that might still be around here
Also found a related question here

Checkout "CAB" - Microsoft's Composition Application Building blocks Framework. I think they've got a "web version" of that too...

I have just started to develop a smart client application. These are two options I am considering.
Using Microsoft's System.AddIn namespace. Looks very promising, however it may be a little complex for our end solution.
Or the Smart Client - Composite UI Application Block from Microsoft
Recently, i have looked at taking components both the Composite UI Application Block and the System.AddIn namespace to build my own. Since source code is available for the CAB it is easy to extend. I think our end solution will be a light weight version of the CAB, definatly using the Unity Application Block

If you work with .Net, our research yielded two approaches: scripting and composition.
Scripting
You extend the functionality of what your classes can do by orchestrating them using scripts. That means exposing what is compiled in your favorite .Net language in a dynamic language.
Some options we found worth exploring:
IronPython
IronRuby
JavaScript: Jint, Jurassic and JavaScript .Net are good starting points.
Script.Net -> this one was the first one to call our attention.
Composition
If you start a project with .Net 4 or above, you must take a good look at the Managed Extensibility Framework (MEF). It allows you to extend the functionality of your apps in a plugin way.
The Managed Extensibility Framework (MEF) is a composition layer for
.NET that improves the flexibility, maintainability and testability of
large applications. MEF can be used for third-party plugin
extensibility, or it can bring the benefits of a loosely-coupled
plugin-like architecture to regular applications.
Managed Add-in Framework is also a good read.
MSDN: http://msdn.microsoft.com/en-us/library/dd460648.aspx
Codeplex: http://mef.codeplex.com/

Rather than re-inventing the wheel, use the frameworks in hand. Eclipse and Netbeans both support plugin based extensions. You have to work in Java though.

Related

Tools for building a Graph/Node based user interface in a webapp

This a somewhat random question, but I wanted to figure out what would be the best tools to create a web app which uses node based connections similar to that of Blender's Shader nodes and Unreal Engines Blueprint system. I am new to web development and didn't know what tools to use to create such an UI system. Any help would be appreciated.
I was wondering about the same thing today and seeing the most relevant SO question being unanswered was a little discouraging.
Anyways, I took a little dive into the subject myself and came up with plenty of options. Hopefully, this list provides a good starting point.
Note: While the boundaries of "web app" are pretty extensive, I mainly focused on Javascript libraries.
React Flow
https://reactflow.dev/
Highly customizable library for building an interactive node-based UI, workflow editor, flow chart or static diagram
Rete.js
https://rete.js.org
Rete.js is a modular framework for visual programming. Rete allows you to create node-based editor directly in the browser. You can define nodes and workers that allow users to create instructions for processing data in your editor without a single line of code.
Baklava.js
https://github.com/newcat/baklavajs
Graph / node editor in the browser using VueJS, less than 60kb gzipped
Flume
https://flume.dev/
A React-powered node editor and runtime engine
CodeWire
https://github.com/ayushk7/CodeWire
CodeWire is a node based editor inspired by UE4 Blueprints which helps in better visualization of the code, and faster scripting of complex and repetitive tasks. It doesn't bind to any particular language. Multiple target languages can be added to added in the future.
Wire
https://github.com/emilwidlund/wire
Wire is a monorepo containing a set of libraries (artifacts) that makes it super easy to build blazing fast & reactive visual programming tools for the web. It offers a core data processing framework and a library with React-components that will get you started. Take a look at the underlying artifacts to learn more about Wire and its offerings.

TVML vs Custom App for Apple TVOS

Are there any advantage of using TVML over custom App (media based App not a game) for Apple TVOS? From what I read from Apple's documentation I could not figure out any such advantage (though it mostly talks about TVML/TVJs).
For iOS (Swift, Objective C) developers, using custom seems like easier, faster and with more possibilities.
I think whatever templates are provided by Apple can be build using UIKit because internally it is UIKit. Right?
[Update]
Let me rephrase my original question as the answers so far are not given any points in favour of custom App. I agree that #shirefriendship has some real valid points in favour of TVML (biggest one is- > App changes possible w/o an App update). Are there any advantages of using custom over TVML? Like anything not in reach of TVML?
Having already built an application using TVML/JS and now working on a native application I would say that the Javascript option is aimed more towards beginners or basic applications.
TVML applications provide an easier, faster way to build standard applications with common features/user interface. We used it to rapidly build a prototype of an application. Now that we've launched the application and it's getting some uplift (featured in What to Watch and Top Free Apps UK) we are now rebuilding it natively to add additional features such as analytics, crash reporting, custom interfaces and other bits to increase re-circulation and custom recommendations.
#shirefriendship's answer provides a good explanation to the benefits of TVML applications and I think this is a good option for beginners, prototype applications or even basic applications.
Native applications provide alot more control over how the application looks, works and feels. Having access to most of the iOS SDK's allows you to do more such as integrating Cloudkit, develop 2D & 3D games and more.
Once we have finished development of our native TV application I'll update the answer with some more information such as development time between the two different versions, features in the application etc.
UPDATE: To answer your second question, yes you can pretty much develop ANY user interface / template using UIKit, the TVML templates provide user interfaces that would be commonly used / quite handy. Using UIKit you could replicate or customize any of the pre-defined templates or build something completely new.
It is true that you have more flexibility with custom UIKit code. With enough time, you could potentially create an app that behaves similarly to a TVML/TVJS app...but why reinvent the wheel?
There are certain advantages to using TVML Templates, even if you are primarily a Swift/Obj-C developer.
TVML templates provide a familiar experience for the user. Have you
ever used the Netflix TV app, or HBO GO, or iTunes? They all use
TVML Templates and have a distinct look and feel to them. This cuts down drastically on design time.
Templates are incredibly easy to implement, even with very little
JavaScript experience. Auto-layout is taken care of for you. Lazy
loading images occurs automatically. The behavior of every UI
element on the screen has already been taken care of by Apple.
The templates are structured yet customizable. Templates are plug
and play, but you can still customize certain aspects of the UI
elements Styles and Attributes
You can host your TVML/TVJS files on a web server. This allows you
to make changes to your app without your user needing to download an
update to your app.
You can mix and match TVML with custom UIKit. Want Custom behavior with UIKit sometimes, but for standard views, you'd prefer a template? Why choose? You can have BOTH in the same app. Check out my answer here that describes how: How To Mix UIKit and TVML Within One App
When in doubt, try it out! Mess around with the Catalog App from Apple and see if you like TVML templates. You might surprise yourself.
MSK
The primary advantage of TVML is that it is all javascript which appeals to a broader base of developers.
Not sure about your second question regarding developing templates via UIKit

What Haskell web framework would one use for an HTTP/Websocket data and messaging platform?

Just looking at Haskell and web frameworks and wondering if it would make sense to use Haskell's great threading/event functionality to power a platform for writing HTML5 and REST apps that expose an HTTP API for data and a WebSocket (with maybe SockJS support for appropriate fallback) API for events? It doesn't seem like the "big" web frameworks support WebSockets as a first-class citizen, though they seem to have a lot of other things going for them.
My concern is making use of available cores, which Haskell can do well, but also providing easy user integration on the server side for validation and server-side logic (maybe by embedding Lua or similar?). If one wrote this on the JVM, one could make use of multiple server-side language support and lots of libraries for this sort of thing.
I'm sure people are doing things like this in a one-off solution for their own applications but I'm thinking along the lines of a PaaS-type approach where one can write HTML5 apps with data (including proper synchronization for offline use) and eventing "for free" as a fundamental part of the platform. Most logic would reside in the browser but some could be run on the server with the appropriate hooks and a reasonable embeddability (JavaScript seems out of the question and not sure about embedding interpreters in Haskell as I'm only dangerously familiar with Haskell in general).
Part of the problem I've had with Haskell so far is that I'm not a Math guy. I didn't study CS in college and I'm a creative-type thinker. So a lot of the tutorials and documentation get me pretty lost, especially when dealing with the mathematical stuff.
Has anyone trod this path already? Am I late to the party? :)
Gregory Collins gave a tutorial at CUFP last year about using Snap to build an interactive chat website using long polling (not websockets). The source code is here.
In the websockets department, Jasper Van der Jeugt wrote a Haskell websockets library. It is available on hackage and comes with websockets-snap, which provides Snap framework support. There's also wai-websockets which provides integration with Warp.
I believe all of the major frameworks have some level of websockets support, so they should all be a fair choice based on your requirements. For Yesod, there's an example of creating a chat system (using eventsource, not websockets) available in the book:
http://www.yesodweb.com/book/wiki-chat-example

Language Agnostic API Generation

I am currently working on a design for a collection of subsystems, and I would like to be able to offer the API's exposed by a given subsystem for use by other subsystems.
In the past, I have used SWIG to expose C api's to a variety of other languages. This has worked well for me, but ultimately the API is defined in C. So basically one side of the API is language agnostic, and the other isn't.
What I would really like is to have something similar to SWIG that could generate the interface between 2 arbitrary languages based on some description of the API.
I don't want to use web services.
For example, I would like to call a 'function' from java, and implement the 'function' in Python. I'd like to be able to generate the language interop using a code generator.
Is there anything that exists which can do this today? At least for simple 'function' calls - ignoring the more complex cases like callbacks and situations where you need to maintain references outside of the 'function' call itself.
I found this after following gooli's link to Protocol Buffers .
This looks so compelling I'm going to propose it as an answer to my own question.
http://incubator.apache.org/thrift/
Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.
From the whitepaper:
...we were presented with the challenge of building a transparent,
high-performance bridge across many
programming languages. We found that
most available solutions were either
too limited, did not offer sufficient
datatype freedom, or suffered from
subpar performance
Also from the whitepaper:
A. Similar Systems The following are software systems similar to Thrift.
Each is (very!) briefly described:
SOAP. XML-based. Designed for web services via HTTP, excessive XML
parsing overhead.
CORBA. Relatively comprehensive, debatably overdesigned and
heavyweight. Comparably cumbersome
software installation.
COM. Embraced mainly in Windows client softare. Not an entirely open
solution.
Pillar. Lightweight and high-performance, but missing
versioning and abstraction.
Protocol Buffers. Closed-source, owned by Google. Described in Sawzall
paper.
The last part about Protocol Buffer's is out-of-date - its been open sourced
COM (and to a lesser extent Firefox's XPCOM) might be what you are looking for. In COM, you define the API using a language called IDL which can then be compiled into different languages. When using the same language the calls usually degenerate into efficient function calls.
However, COM is a very complex and dying piece of technology that I would never use for a new project.

What are some viable alternatives to BizTalk Server? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
In evaluating different systems integration strategies, I've come across some words of encouragement, but also some words of frustration over BizTalk Server.
What are some pros and cons to using BizTalk Server (both from a developer standpoint and a business user), and should companies also consider open source alternatives? What viable alternatives are out there?
EDIT: Jitterbit seems like an interesting choice. Open Source and seems to be nicely engineered. Anyone on here have any experience working with it?
BizTalk Server's key benefit is that it provides a lot of 'plumbing' around deployment, management, performance, and scalability. Through Visual Studio, it also provides a comprehensive framework for developing solutions, often with relatively little code.
The frustration and steep learning curve that others mention often comes from using BizTalk for the wrong purpose and from a misunderstanding about how to work with BizTalk and message-oriented systems in general. The learning curve is not as steep as most people suggest - the essential part of the underlying learning actually focuses on changing thinking from a procedural approach to a stateless message-based approach.
A drawback people often cite is cost. The sticker price can seem to be quite high; however, this is cheap in comparison to the amount you'd spend on developing and supporting features on your own.
Before you consider alternatives, or even consider BizTalk server, you should consider your organization's approach to integration and it's long term goals. BizTalk Server is great in cases where you want to integrate systems using a hub and spoke model where BizTalk orchestrates the activities of many applications.
There are other integration models too - one of the more popular ones is a distributed bus (don't confuse this with the term "Enterprise Service Bus" or ESB). You can also get BizTalk to work as a distributed bus and there are alternative solutions that provide more direct support. One of the alternate solutions is an open source solution called nServiceBus.
When considering whether to use a commercial product like BizTalk, verses something else (open source or developed in house), also consider maintenance and enhancements and the availability of the necessary skill-set in the marketplace.
I wrote some articles that go into more detail about the points I discussed here - here are the links:
Why BizTalk?
Top 10 BizTalk Mistakes
Extensibility Features in BizTalk Server
Open Source Integration with nServiceBus
My experience with BizTalk was basically a frustrating waste of time.
There are so many edge cases and weird little business logic tweaks you have to make when you are doing B2B data integration (which is probably the hardest part of any enterprise application) that you just need to roll your own solution.
How hard is it to parse data files and convert them to a different format? Not that hard. Unless you're trying to inject a bloated middleware system like Biztalk into the middle of it.
As a BizTalk consultant I have to agree at least partly with Eric Z Beard, there are a lot of edge cases that take up alot of time. But quite a few scenarios are handled extremly smooth as well, so it all depends IMO. But when you (Eric) call BizTalk bloated I have to disagree! We've found that the performance and reliability is excellent, it's flexible and comes with a lot of good adapters out of the box.
BizTalk needs to be used correctly,
I am a BizTalk developer and my experience with BizTalk is quite good.
Its reliable, performant, scalable, contains a lot of built in architectural patterns and build in components to make integration easy and fast, you get security, retries, secondary transports, validation, transformation etc... and what ever you dont have build in with BizTalk you can easily customized with .NET code, its basically a hard earned integration system and you get all this in one box.
BUT you need to know how to implement BizTalk correctly, not once I came across solutions that where implemented and often also architected incorrectly.
but the real benefit of BizTalk is that you can implement small solutions and scale up whilst most other integration systems from big vendors will only sell a whole integration pack which can cost much more.
BizTalk is considered the most complicated server from the house of Microsoft.
So any body saying BizTalk is not good dosent know BizTalk period.
We evaluated BizTalk at our company and were really disappointed.
We are using IBM WebSphere Transformation Extender (which has lots of (other) problems, too) and the mapping tool of BizTalk is a joke in comparison to WTX.
The graphical tool is not really usable for complex mappings (we have schemas with a few hundred fields in repeating groups) and if you do more than the usual "concat first name and last name to name" mappings, you will be tired of the graphical approach (for example the arguments of the functoids in the graphical mapper are not labeled and the order in which you connect fields to these arguments is important).
The XSLT-Mapper was usable but not really convincing, and even the microsoft rep told us to use a tool like XMLSpy for XSLT and load the resulting XSL file into BizTalk.
A third approach to mapping is to use C#-Code for the mapping, which was not acceptable for us as a general approach (we don't want to teach everyone C#).
In addition to the mapping tool we did not like the deployment in BizTalk. In order to deploy your process, you need to make lots of settings in different tools and places. We had hoped to find a mechanism like a WAR file for Java Web Applications in BizTalk, so that you can give one archive for your whole process solution to your administrator and he can deploy it.
We've been using BizTalk since version 2004, and now have a mix of versions 2006 R2 and 2004 running. I found that the learning curve was quite severe, and development time for solutions is not always quick. Those are definitely shortcomings. Where BizTalk really excels is in its fault tolerance, gauranteed delivery, and performance. You can rest assured that data will not get lost. Retry functionality and fault tolerance robustness is baked in so generally speaking if systems are down BizTalk will handle that and successful delivery will occur once systems come back on line. All these issues such as downtime, etc that are important in an integration scenario are handled by BizTalk.
Further, generally speaking when developing solutions BizTalk abstracts the communication protocols and data formats of the native systems by dealing with everything as xml, so when developing solutions, you typically don't have to wrote code specific to those systems, you use the BizTalk xml framework.
In the last year, we've implemented a java open source engine called Mirth for our HL7 routing. I found that for HL7 purposes, the HL7 adaptor for BizTalk is a challange to work with. Management dicated that we use Mirth for HL7 routing. Where BizTalk falls down in terms of learning curve, Mirth makes up. It is far easier to develop a solution. The problem with mirth is that it doesn't really have any gauranteed delivery. Most of the adaptors (except for hl7) have no retry functionality so if you wanted that you'd have to write your own. Second, Mirth can lose date if it goes down. I would call it very easy to use (although there is no documentation) but I'd be hard pressed to call it an enterprise solution. I'm going to check out jitterbit which was mentioned by someone else.
We used BizTalk for a couple of years, but gave it up for our own custom framework that allowed more flexibility.
There is always Sun's (now Oracle) OpenESB framework. Its generally speaking a smaller, lighter version of Biztalk but with roughly all the same features.
You do get to write more code with it, though.
Its Open Source as well.
In the OSS space (though I've never used them as a BizTalk replacement personally - this is anecdotal) you can use one of the Java/J2EE Messaging engines such as OpenMQ (which is the Sun enterprise one rebadged and without support). If you need Orchestration / Choreography (i.e. SOA/ESB pieces) on top of this, you could look into something like Apache Mule
My experience with BizTalk and doing B2B integrations is that most organizations do not truly do schema first design or fully understand xml standards for that matter. Most tend to weave objects and hope they materialize into meaninful schemas. In an enterprise environment, this is backwards.
BizTalk does have a learning curve, but once you get it you are rewarded with durability, performance, true scalability, and extensibility. Like most have said though, it best to make sure it meets your needs and contort your needs to BizTalk.
In the past I have worked with BizTalk 2004 through 2009, and another product called webMethods.
I have no direct experience with JitterBit, but I have heard very good things from coworkers.
I came across Apatar (unable to post url, but Google finds it) while looking for a solution cheaper than BizTalk. I have yet to try this out.
My last company had many problems with BizTalk being too complex and ridged, but I can’t help but think this was mainly down to the implementation the consultant did.