Related
For example, when discussing a build or deploy process, and making sure it is independent of the IDE. Is this "coupling", or is that considered Separation of Concerns, or something completely different? The general concept is to introduce the least number of variables into a process or architecture, so that when a failure occurs, the difficulty in identifying the possible points of failure are reduced significantly. Is there another definition for that?
Something completely different.
Coupling is code.
Independent tools are just independent tools.
Microsoft has lead me to believe that independent tools are a bad idea. They tell me that one vendor's integrated tool suite is a good thing.
I would characterise coupling as being something which relates to runtime system stability or system change.
When considering stability, coupling comes into the picture if the failure of one component causes the failure of other components. For example if two software components are communicating directly over a TCP connection, then the failure of one component means the other cannot do its work. The whole system is down. If I decouple by allowing the two components to communicate via a message queue (for example) then each component might continue to work independently in the absence of the other (assuming this makes sense in the application).
When considering system change, coupling comes into the picture when a code change in one module means I have to go and change code in a large number of other modules.
The example you give with build/deploy tools is a form of coupling, but not really what people have in mind when they consider architectural issues such as runtime and code coupling.
i have always wanted to make a private server but i don't know how i would do this.
i know how a private server works, the game sends data packets to the server. the server will take the data and process it and send data to the other games connected.
my questions are,
how do you edit the game so it will go to your server/change game data.
how do you find what packets do what.
the game will be something like WOW, i have not desided yet.
If you are hoping to embark on creating your own MMORPG then you have a huge task ahead of you, and unfortunately to put it nicely you are probably being too ambitious especially if you are asking these sorts of questions.
You should probably read up on client server architecture.
Also, in answer to your questions about the structure of the data being sent and how it is interpreted, well, that's 100% up to the people that design the system. You will want to simulate the entire game on the server(s) and don't trust the clients at all.
For something as complex as a MMORPG it is really important to create a solid design for the system before anything else, this is very important.
Just to be clear your intent is to create an emulated MMO server to the effect of WOW?
That's not really a trivial task and carries with it its own ethical implications.
Just to get started will require a ton of research, inspection, decoding, an extreme attention to detail.
If you are serious about it, then I would suggest looking up networking tools that can help you inspect traffic across the network and creating a scientific process for operation inspection.
Again, it should be noted this is by no means a trivial task.
This will be fairly difficult as you do not have the communicaton protocol specification for the game's client/server communication.
If you want to start this, then create a server that is simply a pass through. That is, all client requests are forwarded to the particular server. Once you have generated a large enough sample size of packets to study, then you can begin to dissect the meaning of each byte (possibly). Of course, if the packets are encrypted in any way (even a simple XOR encryption) then you will have an even harder time trying to figure out what each byte means. You should capture a sample set using two clients running sniffers so you can see what happens when one client does something and it needs to be sent to all clients.
But if I were you, I would just abandon the idea and work on something else. My two cents..
If you'd like an inside look at how games do networking, there's always Ryzom, which went open-source earlier this year. If you're creating your own MMO you can begin right there, and if you're looking to reverse-engineer one you can practice with your own client and server.
Some of our partners are telling us that our software needs to interact with an Enterprise Service Bus. After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth. I'm just trying to get a feel for what our partners are telling us. Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
Although ESB is based on messaging, it is not "just" messaging and not just a buzzword.
So if you start with plain old async messaging, the early networks tended to be very point-to-point. You had to wire up (i.e. configure through some admin interface) each connection and each pair of destinations and if you dared to move anything around invariably something broke. Because the connection points were wired by hand these networks never achieved high connection density. The incremental cost was too high and did not scale. There was also a lot of access control and policy embedded in the topology. The lack of connection density actually favors this approach to security, even though it inhibits flexibility.
The ESB attempts to address these issues with...
Run-time resolution of destinations/services/resources
Location transparency
Any-to-any connectivity and maximum connection density
Architected for redundancy, horizontal scalability, failover
Policy, access control, rules externalized from topology
Logical messaging network layer implemented atop the physical messaging network layer
Common namespace
So when your customer asks for ESB compatibility, they want things like the above. From an application standpoint, this also implies...
Avoiding message affinities such as requirements to process in strict sequence or to address requests only to specific nodes instead of to a generic network destination
Ability to resolve destinations dynamically at run time (i.e. add another instance of a queue and it automatically starts getting traffic, delete one and traffic routes to the remaining nodes)
Requestor and provider apps decoupled from knowing where each other "lives". Requestor makes one connection, regardless of how many services it might need to call
Authorize by policy rather than by topology
Service provider apps able to recognize and handle dupes (as per JMS spec, see "functional duplicate" due to session handling)
Ability to run multiple active instances of a service provider application
Instrument the service provider applications so you can inquire on the status of the network or perform a test without sending an actual transaction
On the other hand, if your client cannot articulate these things then they may just want to be able to check a box that says "works with the ESB."
I'll try & keep it buzzword free (but a buzz acronym may creep in).
When services/applications/mainframes/etc... want to integrate (so send messages to each other) you can end up with quite a mess. An ESB hides that mess inside of itself (or itselves) so that an organisation can pretend that there isn't a mess and that it has something manageable. It then wraps a whole load of features around this to make this box even more enticing to the senior people in an organisation who'll make the decision to buy such an expensive product. These people typically will want to introduce a large initiative which costs a lot of money to prove that they are 'doing something' and know how to spend large amounts of money. If this is an SOA initiative then vendors various will have told them that an ESB is required to make the vendors vision of what SOA is work (typically once the number of services which they might want passes a trivial number).
So an ESB is:
A vehicle for vendors to make lots of money;
A vehicle for consultants to make lots of money;
A way for senior executives (IT Directors & the like) to show they can spend lots of money;
A box to hide a mess in;
A total PITA for a technical team to work with.
After researching this a bit, my
instinct is to say that this is just
buzz speak for saying that we need to
have a platform-indpendent way to pass
messages back and forth
You are correct, partially because the term ESB is always nice word that fits well with another buzzword, legitimate or not - which is governance (i.e. helps you manage who is accessing your endpoints and reporting metrics - Metrics btw is what all the suits like to see, so that may be a contributor)
Another reason they might want a platform neutral device is so that any services they consume are always exposed as endpoints from a central location, instead of a specific machine resource. The ESB makes the actual physical endpoints of your services irrelevant to them, which they shouldn't care much about anyway, but that enables you to move services around however they will only consume the ESB Endpoint.
Apart from a centralized repository for Discovery, an ESB also makes side by side versioning of services easier. If I had a choice and my company had the budget, we would have purchased IBM's x150 appliance :(
Thirdly, a lot of more advanced buses, like SoftwareAG's product if I recall, is natively able to expose legacy data, like from data sitting on main frames as services without the need for coding via adapters
I don't know if their intent is to leverage all the benefits an ESB provides, or as you said, make it buzzword compliant.
After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth
That's about right. Sometimes an ESB will go a little bit further and include additional features like message delivery guarantees, confirmation/acknowledgement messages, and so on. The presence of an ESB also usually explicitly or implicitly creates a new protocol where none previously existed, which is another important consideration. (That is, some sort of standard or interface has to be set regarding the format of the messages.)
Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
You should always listen to your customers, even if it initially sounds silly. It's usually worth at least spending the effort to decide what's going on. Reading between the lines, what your partners probably mean is that they want a way for your service to integrate more easily with their own services and products.
An enterprise service bus handles the messaging between systems in a standard way. This allows you to communicate with the bus in the same exact way across all your platforms and the bus handles the actual translating to individual communication mechanism needed for the specific endpoint. This means you write all your code to talk to the bus using a common messaging scheme and the bus handles taking your common scheme and translating it so the endpoint understands it.
The simplest explanation is to explain what it provides:
For many years companies acquired different platforms and technologies to achieve specific functions in their business from Finance to HR. These systems needed to talk to each other to share data so middleware became the glue that allowed them to connect. Before the business knew it, they were paying for support and maint on each of these systems and the middleware. As needs in the business changed departments decided to create their own custom solutions to address special needs rather than try to make the aging solutions flexible enough to meet their needs. Before they knew it, they were paying to support and maintain the legacy systems, middleware, and custom solutions. With new laws like Sarbanes Oxley, companies need to have better information available for reporting purposes. A single view requires that they capture data from all of the systems. In addition, CIOs are now being pressured to lower costs and increase customer service. One obvious solution is the eliminate redudant systems, expensive support and maint contracts, and high cost legacy solutions which require specialists to support. Moving to a new platform allows for this, but there needs to be a transition. There are no turnkey solutions that can replicate what the business does. To address the needs for moving information around they go with SOA because it allows for information access through a generic entity. If I ask for AllEmployees from the service bus it gets them whether it is from 15 HR systems or 1. When the 15 HR systems becomes 1 system the call and result does not change, just how it was done behind the scenes. The Service Bus concept standardizes the flow of information and allows IT managers to conduct transitions behind the bus with no long term effect on upstream users.
I have been doing a little reading on Flow Based Programming over the last few days. There is a wiki which provides further detail. And wikipedia has a good overview on it too. My first thought was, "Great another proponent of lego-land pretend programming" - a concept harking back to the late 80's. But, as I read more, I must admit I have become intrigued.
Have you used FBP for a real project?
What is your opinion of FBP?
Does FBP have a future?
In some senses, it seems like the holy grail of reuse that our industry has pursued since the advent of procedural languages.
1. Have you used FBP for a real project?
We've designed and implemented a DF server for our automation project (dispatcher, component iterface, a bunch of components, DF language, DF compiler, UI). It is written in bare C++, and runs on several Unix-like systems (Linux x86, MIPS, avr32 etc., Mac OSX). It lacks several features, e.g. sophisticated flow control, complex thread control (there is only a not too advanced component for it), so it is just a prototype, even it works. We're now working on a full-featured server. We've learnt lot during implementing and using the prototype.
Also, we'll make a visual editor some day.
2. What is your opinion of FBP?
2.1. First of all, dataflow programming is ultimate fun
When I met dataflow programming, I was feel like 20 years ago, when I met programming first. Altough, DF programming differs from procedural/OOP programming, it's just a kind of programming. There are lot of things to discover, even sooo simple ones! It's very funny, when, as an experienced programmer, you met a DF problem, which is a very-very basic thing, but it was completely unknown for you before. So, if you jump into DF programming, you will feel like a rookie programmer, who first met the "cycle" or "condition".
2.2. It can be used only for specific architectures
It's just a hammer, which are for hammering nails. DF is not suitable for UIs, web server and so on.
2.3. Dataflow architecture is optimal for some problems
A dataflow framework can make magic things. It can paralellize procedures, which are not originally designed for paralellization. Components are single-threaded, but when they're organized into a DF graph, they became multi-threaded.
Example: did you know, that make is a DF system? Try make -j (see man, what -j is used for). If you have multi-core machine, compile your project with and without -j, and compare times.
2.4. Optimal split of the problem
If you're writing a program, you often split up the problem for smaller sub-problems. There are usual split points for well-known sub-problems, which you don't need to implement, just use the existing solutions, like SQL for DB, or OpenGL for graphics/animation, etc.
DF architecture splits your problem a very interesting way:
the dataflow framework, which provides the architecture (just use an existing one),
the components: the programmer creates components; the components are simple, well-separated units - it's easy to make components;
the configuration: a.k.a. dataflow programming: the configurator puts the dataflow graph (program) together using components provided by the programmer.
If your component set is well-designed, the configurator can build such system, which the programmer has never even dreamed about. Configurator can implement new features without disturbing the programmer. Customers are happy, because they have personalised solution. Software manufacturer is also happy, because he/she don't need to maintain several customer-specific branches of the software, just customer-specific configurations.
2.5. Speed
If the system is built on native components, the DF program is fast. The only time loss is the message dispatching between components compared to a simple OOP program, it's also minimal.
3. Does FBP have a future?
Yes, sure.
The main reason is that it can solve massive multiprocessing issues without introducing brand new strange software architectures, weird languages. Dataflow programming is easy, and I mean both: component programming and dataflow configuration building. (Even dataflow framework writing is not a rocket science.)
Also, it's very economic. If you have a good set of components, you need only put the lego bricks together. A DF program is easy to maintain. The DF config building requires no experienced programmer, just a system integrator.
I would be happy, if native systems spread, with doors open for custom component creating. Also there should be a standard DF language, which means that it can be used with platform-independent visual editors and several DF servers.
Interesting discussion! It occurred to me yesterday that part of the confusion may be due to the fact that many different notations use directed arcs, but use them to mean different things. In FBP, the lines represent bounded buffers, across which travel streams of data packets. Since the components are typically long-running processes, streams may comprise huge numbers of packets, and FBP applications can run for very long periods - perhaps even "perpetually" (see a 2007 paper on a project called Eon, mostly by folks at UMass Amherst). Since a send to a bounded buffer suspends when the buffer is (temporarily) full (or temporarily empty), indefinite amounts of data can be processed using finite resources.
By comparison, the E in Grafcet comes from Etapes, meaning "steps", which is a rather different concept. In this kind of model (and there are a number of these out there), the data flowing between steps is either limited to what can be held in high-speed memory at one time, or has to be held on disk. FBP also supports loops in the network, which is hard to do in step-based systems - see for example http://www.jpaulmorrison.com/cgi-bin/wiki.pl?BrokerageApplication - notice that this application used both MQSeries and CORBA in a natural way. Furthermore, FBP is natively parallel, so it lends itself to programming of grid networks, multicore machines, and a number of the directions of modern computing. One last comment: in the literature I have found many related projects, but few of them have all the characteristics of FBP. A list that I have amassed over the years (a number of them closer than Grafcet) can be found in http://www.jpaulmorrison.com/cgi-bin/wiki.pl?FlowLikeProjects .
I do have to disagree with the comment about FBP being just a means of implementing FSMs: I think FSMs are neat, and I believe they have a definite role in building applications, but the core concept of FBP is of multiple component processes running asynchronously, communicating by means of streams of data chunks which run across what are now called bounded buffers. Yes, definitely FSMs are one way of building component processes, and in fact there is a whole chapter in my book on FBP devoted to this idea, and the related one of PDAs (1) - http://www.jpaulmorrison.com/fbp/compil.htm - but in my opinion an FSM implementing a non-trivial FBP network would be impossibly complex. As an example the diagram shown in
is about 1/3 of a single batch job running on a mainframe. Every one of those blocks is running asynchronously with all the others. By the way, I would be very interested to hearing more answers to the questions in the first post!
1: http://en.wikipedia.org/wiki/Pushdown_automaton Push-down automata
Whenever I hear the term flow based programming I think of LabView, conceptually. Ie component processes who's scheduling is driven primarily by a change to its input data. This really IS lego programming in the sense that the labview platform was used for the latest crop of mindstorm products. However I disagree that this makes it a less useful programming model.
For industrial systems which typically involve data collection, control, and automation, it fits very well. What is any control system if not data in transformed to data out? Ie what component in your control scheme would you not prefer to represent as a black box in a bigger picture, if you could do so. To achieve that level of architectural clarity using other methodologies you might have to draw a data domain class diagram, then a problem domain run time class relationship, then on top of that a use case diagram, and flip back and forth between them. With flow driven systems you have the luxury of being able to collapse a lot of this information together accurately enough that you can realistically design a system visually once the components are build and defined.
One question I never had to ask when looking at an application written in labview is "What piece of code set this value?", as it was inherent and easy to trace backwards from the data, and also mistakes like multiple untintended writers were impossible to create by mistake.
If only that was true of code written in a more typically procedural fashion!
1) I build a small FBP framework for an anomaly detection project, and it turns out to have been a great idea.
You can also have a look at some of the KNIME videos, that give a good idea of what a flow based framework feels like when the framework is put together by a great team. Admittedly, it is batch based and not created for continuous operation.
By far the best example of flow based programming, however, is UNIX pipes which is one of the oldest, most overlooked FBP framework. I don't think I have to elaborate on the power of nix pipes...
2) FBP is a very powerful tool for a large set of problems. The intrinsic parallelism is a great advantage, and any FBP framework can be made completely network transparent by using adapter modules. Smart frameworks are also absurdly fault tolerant, and able to dynamically reload crashed modules when necessary. The conceptual simplicity also allows cleaner communication with everybody involved in a project, and much cleaner code.
3) Absolutely! Pipes are here to stay, and are one of the most powerful feature of unix. The power inherent in a FBP framework compared to a static program are many, and trivialise change, to the point where some frameworks can be reconfigured while running with no special measures.
FBP FTW! ;-)
In automotive development, they have a language agnostic messaging protocol which is part of the MOST specification (Media Oriented Systems Transport), this was designed to communicate between components over a network or within the same device. Systems usually have both a real and visualized message bus - therefore you effectively have a form of flow based programming.
That was what made the light bulb go on for me several years ago and brought me here. It really is a fantastic way to work and so much more fun than conventional programming. The message catalog form the central specification and point of reference. It works well for both developers and management. i.e. Management are able to browse the message catalog instead of looking at source.
With integrated logging also referencing the catalog to produce intelligible analysis things can get really productive. I have real world experience of developing commercial products in this way. I am interested in taking things further, particularly with regards to tools and IDEs. Unfortunately I think many people within the automotive sector have missed the point about how great this is and have failed to build on it. They are now distracted by other fads and failed to realize that there was far more to most development than the physical bus.
I've used Spring Web Flow extensively in Java Web applications to model (typically) application processes, which tend to be complex wizard-like affairs with lots of conditional logic as to which pages to display. Its incredibly powerful. A new product was added and I managed to recut the existing pieces into a completely new application process in an hour or two (with adding a couple of new views/states).
I also looked into using OS Workflow to model business processes but that project got canned for various reasons.
In the Microsoft world you have Windows Workflow Foundation ("WWF"), which is becoming more popular, particularly in conjunction with Sharepoint.
FBP is just a means of implementing a finite state machine. It's nothing new.
I realize that it is not exactly the same thing, but this model has been used for years in PLC programming. ISO calls it Sequential Flow Chart, but many people call it Grafcet after a popular implementation. It offers parallel processing and defines transitions between states.
It's being used in the Business Intelligence world these days to mashup and process data. Data processing steps like ETL, querying, joining , and producing reports can be done by the end-user. I'm a developer on an open system - ComposableAnalytics.com In CA, the flow-based apps can be shared and executed via the browser.
This is what MQ Series, MSMQ and JMS are for.
This is cornerstone of Web Services and Enterprise Service Bus implementations.
Products like TIBCO and Sun's JCAPS are basically flow-based without using this particular buzz-word.
Most of the work of the application is done with small modules that pass messages through a processing network.
Can we achieve 100% decoupling between components of a system or different systems that communicate with each other? I don't think its possible. If two systems communicate with each other then there should be some degree of coupling between them. Am I right?
Right. Even if you write to an interface or a protocol, you are committing to something. You can peacefully forget about 100% decoupling and rest assured that whatever you do, you cannot just snap out one component and slap another in its place without at least minor modifications anyway, unless you are committing to very basic protocols such as HTTP (and even then.)
We human beings, after all, just LOOVE standards. That's why we have... well, nevermind.
If components are 100% decoupled, it means that they don't communicate with each other.
Actually there are different types of coupling. But the general idea is that objects are not coupled if they don't depend on each other.
You can achieve that. Think of two components that communicate with each other through network. One component can run on Windows while other on Unix. Isn't that 100% decoupling?
At minimum, firewall protection, from a certain interface at least, needs to allow the traffic from each machine to go to the other. That alone can be considered a form of 'coupling' and therefore, coupling is inherent to machines that communicate, at least to a certain level.
This is achievable by introducing a communication interface or protocol which both components understand and not passing data directly between the components.
Well two webservices that don't reference each other might be a good example of 100% decoupled.
The coupling would then arrive in the form of an app util that "couples" them together by using them both.
Coupling isn't inherently bad but you do have to make solid judgement calls about when to do it (is it only at Implementation, or in your framework itself?) and if the coupling is reasonable.
If the components are designed to be 100% orthogonal, it should be possible. A clear separation of concerns can achieve this. All a component needs to know is the interface of its input.
The coupling should be one-directional: components know the semantics of their parameters, but should be agnostic of each other.
As soon as you have 1% coupling between components, the 1% starts growing (in a system which lasts a little longer)
However, often knowledge is injected in peer components to achieve higher performance.
Even if two components do not comunicate directly, the third component, which uses the other two is part of the system and it is coupled to them.
#Vadmyst: If your components communicate over network they must use some kind of protocol which is the same as the interface of two local components.
That's a painfully abstract question to answer. If said system is the components of a single application, then there are various techniques such as those involving MVC (Model View Controller) and interfaces for IoC / Dependency Injection that facilitate decoupling of components.
From the perspective of physically isolated software architectures, CORBA and COM support local or networked interop and use a "common tongue" of things like ATL. These have been deprecated by XML services such as SOAP, which uses WSDL for performing coupling. There's nothing that stops a SOAP client from using a WSDL for run-time late coupling, although I rarely see it. Then there are things like JSON, which is like XML but optimized, and Google Protocol Buffers which optimizes the interop but is typically precompiled and not late-coupled.
When it comes to IPC (interprocess communications), two systems need only to speak a common "protocol". This could be XML, it could be a shared class library, or it could be something proprietary. Even at the proprietary level, you're still "coupled" by memory streams, TCP/IP networking, shared file (memory or hard disk), or some other mechanism, and you're still using bytes, and ultimately 1's and 0's.
So ultimately the question really cannot be answered fairly; strictly speaking, 100% is only attained by systems that have zilch to do with each other. Refine your question to a context.
It's important to distinguish between direct, and indirect components. Strive to remove direct connections (one class referencing another) and to use indirect connections instead. Bind two 'ignorant' classes with a third which manages their interactions.
This would be something like a set of user controls sitting on a form, or a pool of database connections and a connection pooling class. The more fundamental components (controls and connections) are managed by the higher piece (form and connection pool), but no fundamental component knows about another. The fundamental components expose events and methods, and the other piece 'pulls the strings'.
No, we can't. Read Joel's excellent article The Laws of Leaky Abstraction, it is an eye-opener for many people. However, this isn't necessarily a bad thing, it just is. Leaky abstractions offer great opportunity because they make the underlying platform exploitable.
Think of the API very hard for a very long time, then make sure it's as small as it can possibly be, until it's at the place where it has almost disappeared...
The Lego Software Process proposes this ... :) - and actually quite well achieves this...
How "closely coupled" are two cells of an organism...?
The cells in an organism can still communicate, but instead of doing it by any means that requires any knowledge about the receiving (or sending) part, they do it by releasing chemicals into the body ... ;)