Moving on from Hello world style simple programs to big / real world projects : A guide to real world projects - language-agnostic

Somebody please re-tag with appropriate tags
Hello,
This is my story but I guess it holds true for all programmers.
We begin programming with some simple Hello World program. We practice & add functions/classes to the program. But they still maintain the Hello World style. function calling some other functions standard library.
But when it comes to real world projects(I'm just familiar with OpenSource). Lot more other things come into picture. Then begins the hardships of this newbie programmer.
Project Flow:
Program is not running as expected. Make use of Debugger
Making use of third party libraries. Today, we have
library in every popular language for
almost everything we need.
Multiple persons working on same project. Using Version Control
Systems.
Project is growing big. Build Automation
Lot of people started using your application. You need to port it to
different platforms (operating
systems/architectures). Need for
Cross Compiliation
I don't know why but we need Unit Testing Framework and/or unit tests
What else???
The problem in this is the lack of knowledge of this newbie programmer about existence of these things.
What I mean is when I started looking into some real world projects(Opensource). I didn't know what is this? and why we need to do this?
$./configure
$make
$make install
Recently I became aware of the keyword "Build Automation". I was in need of some library which was available for linux but I needed it in windows. I didn't know that its called "Cross compilation" and tools like MinGW/MSYS exist for this purpose. I had to learn these things in the hard way. I wish some one has told me about existence of such things. That would have saved my lot of time.
Today I ran into performance problem and was feeling the need for something. I guess the thing I'm looking for is Profiler. Thanks to my involvement in opensource projects. Even though I didn't realized/felt the need for this, I'm aware of term Unit Testing.
Though this (hard)way of learning things has some big advantages like now, I'm able to figure out solution or some unknown thing very quickly & unlike my other friends I don't get struck at any point. But I hate the wastage of time involved. You do not believe how much I time I wasted in figuring out the Makefiles & Gnu Build System
So, what am I looking for this in this post?
Please complete the Project Flow. I want to see what all things are involved.
For each of the tasks in the Project Flow list. I want to see following information.
Most popular solutions/tools availabe.
Wikipedia list to all alternatives.
[optional] Suggest some good books/tutorials/guides for learning about this. Or link to relavent SO posts/tags.
I know somethings are language & OS specific. I would say we have only handful of major platforms Linux/Unix, Windows, Java, .NET and handful of major languages C, C++, Java, .NET, Python. Address these languages. Its more than sufficient.
Example:
Making use of libraries:
Libraries are distributed in any of the following forms
Source Distrubtion
Static Libraries(*.lib for windows / *.a for linux)
Dynamic Libraries (.dll for windows /.so for linux)
.NET assemblies
I don't know about java
Resources (Now, once I know the above info. I can search on my own for resources)
http://en.wikipedia.org/wiki/Library_(computing)
How To Write Shared Libraries
http://www.tenouk.com/ModuleBB.html
Note:
Please not that I'm not asking to suggest info on how to learn each of these things. I'm asking about what more such kind of things are involved and alternatives for each of them.

Your question seems to be too broad, but in comments you asked for examples of different methods. This would at least enable you to find some other pointers and make your own decisions.
Rational Unified Process (RUP)/Unified Process (UP)
eXtreme Programming (XP)
Scrum
Test Driven Development (TDD)/Behaviour Driven Development (BDD)
Lean Software development
Kanban
Feature Driven Development (FDD)
Those and many more are software development methods, which comprise of different principles, processes and roles. Most of them also come with some sets of practices for different purposes, e.g. ones you mention in your question. Those are quite widely used in real world, although many companies use different combinations and also some own processes and methods. What is not part of what is used so widely, but might give you some interesting views on development are formal methods, which are precise, but most of the time too unpractical.

Related

I want to write a tool without usage entry barriers. Do I have to write it in C? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I want to write an open-source tool for use by developers. I want to eliminate entry barriers, so if they like the idea, they just get the tool and start playing with it.
In particular, I don't want an "Oh, should I also install 200Mb of ThatLanguage runtime libraries? Oh, so they don't build on my latest version of Linux?" entry barrier.
Should I write this tool in C, then? Or is Python, or Java, or whatever, already sufficiently widespread to not worry about this sort of things altogether (everyone already has them installed)?
Well, of course I know that they are freaking hugely widespread, but still - are there any major benefits to writing a super-lightweight zero-dependency tool, or am I being too much of a perfectionist?
Just write it first. If it is worth it people will use it.
Beyond that, (almost) everyone has Java, Python, and Ruby installed (especially devs). Some languages are still esoteric enough that it might not be worth it for 'that one app' (erlang, haskell, etc.).
Just write it though, that's the important part. From there it can be ported, rewritten, adopted, but none of that can happen if the tool isn't written first.
It won't help if people don't know C.
If you write your own DSL, you can have people use that API and not worry about which language you choose.
Write it in whatever common language you like. Everybody has installed .NET framework or JVM. The only difference between your C approach and Java or C# is, that you would link additional libraries directly to your program (opposed to standard libraries).
On the other hand I would hesitate to write it in some exotic language, for example smalltalk, because normal user does not know what is it squak or smalltalk itself and could be worried about installing the wierd thing :-).
I also think, that you should be concerned more about developers, because you write, you want it to be open source. I dont know anyone, who wants to write his own Swing, Spring or any other framework just to be independent of something. Also its (usually) much faster and easier to write it in JIT Language, than to code it in assembler...
I'm going to suggest what Reese suggested but take a slightly different approach: write it first, preferably in a language that allows you to quickly prototype and develop your program. Then, and this is the most important part, document the protocal you've developed.
I'm giving this advice because you mentioned that your "application" may later have bindings in lots of different languages and it is a client/server architecture. Well, two of the biggest applications in the world started out like this.
Bittorrent started out as Python code. This allowed very quick prototyping of the concept to get it working. The main thing that it had going for it was that the original code was well written and well documented. This later on allowed other people to port the protocol to other languages.
HTTP and HTML is an even bigger success story and started out with an even less popular language at the time it was written: objective-C. Even better than bittorrent, the protocol itself is very simple and very well documented. People didn't care that the original implementation was in a language that they've never seen before that uses square brackets in strange ways on a NeXT cube. The concept and execution was good and people quickly ported it to their favourite programming languages. Again, objective-C was chosen to aid in quick prototyping. Legend has it that the original implementation was written in just a couple of days.
I would say yes, you have to write it in C. If it were written in any language other than C (except perhaps C++ or Perl), I would definitely stop to consider whether the necessary build tools, runtime tools, and/or interpreter for that language would be available everywhere I might need the tool before getting myself dependent upon it. If the tool were meant for use in build scripts, I would consider it a complete show-stopper, since I can't expect anyone who wants to build my software to have random arbitrary language environments installed.
The reason I mentioned C++ and Perl as exceptions is that they're both largely portable in a formal sense. They have implementations that work without significant ties to the host implementation, and can be built not just on any current popular system but on any system that remotely adheres to standards. Python is quite the opposite, with strong dependencies on the underlying system's dynamic loader; I've been completely unable to get Python to work on various systems that only support static linking.
ocaml is another possible choice that has a very portable implementation, but it's not widely installed and people who aren't familiar with it tend to frown on it for no good reason.
If you write your program in C, then you will have the dependency of the platform (Windows != Linux != AIX, etc). If you are talking only about writing this tool for one OS, or rather THE OS (Linux;-), then I think that you can have a reasonable amount of confidence that your app will work on almost any system, especially if you use an Open Source language. If you want to run the app on Windows, I wouldn't count on any of those languages being installed on the host system. Your highest confidence across platforms will be with Java.
If possible you could use the lightest weight framework possible and put it online, where it can be viewed in a browser. What does your app do? Would it work as a web app?
I would suggest go for Delphi. If you want to make it portable, you can do it since most of the Delphi code is kylix compatible.

Scripting Languages vs. Compiled Languages for web development

Though I come from a purely PHP background on the web development side of programming, I have also spent much time with C# and C++ on the desktop.
I don't really want to spark any flame wars, but:
When should you use scripting languages over compiled languages for website development?
(and vice versa)
Just to clarify, for the sake of this question, I define a "scripting language" to mean an interpreted language like PHP, Python, or Ruby, and a "compiled language" to mean a strongly typed, compiled language like C#, C++, Java, or VB.
It depends :-)
On...
...where and how you want to deploy the application
...the skillsets of the engineers in your organization
...what third-party components you want to integrate with or incorporate
Deployment
If you need to be able to deploy the solution on any of dozens of different possible platforms, you may find that you're better off with PHP than Java (for example). There are hundreds of thousands of Java hosting providers out there, but there are probably millions of PHP hosting providers. (And I say this as a Java-head who finds PHP "so so" at best.)
This goes to OS as well. Mono aside, .Net stuff is going to limit you to Windows-based deployment (or lagging behind the cutting edge and having to very, very rigorously test each and every 3rd party component you bring in, to ensure that it doesn't have Mono...issues).
Skillsets
Coming up to speed in an environment or language is non-trivial. For most of us, picking up the basics is pretty quick, but you may not be making the best architectural/design decisions because you're (comparatively) weak on the environment/language. Skillsets count.
Related to this: Skillset hiring counts. Is it easier (and/or cheaper) to hire PHP devs with 3-4 years of experience, or Java devs with 3-4 years of experience, or C# devs, or...?
Buying/finding/integrating vs. building
In your target area of development, which server-side components or packages will you want to integrate with? PHP has a vast array of things available for it, as does Java, as does C# or ASP.Net. But they're different things (by and large), so you'll want to look at what you actually want to use.
Conclusion
So I think it's less a matter of compiled vs. scripted (in today's world), and more a matter of what's the best fit by other criteria for what you're trying to do.
Addendum: Both/And
And of course, there's always "both/and". For instance, I do work in two main, unrelated environments right now, both using a combination of scripted and compiled resources. (One of them is Java + JavaScript via Rhino on Tomcat, the other is compiled COM objects + JScript [again, server-side] on IIS.)
A programmer can write good/bad fast/slow scalable/unscable code in any language. Although, some language and technologies make it harder to do. In my experience, with scripting languages you can produce a small to medium scale application faster than you can with compiled languages like Java. However, as applications grow in size, compiled languages become more suited to the task I think this comes from strongly typing objects, deeper layers of architecture to manage tasks, and more QA frameworks to verify things are running as they should be as changes occur.
I find it to be mostly a matter of opinion. At first I hated the pre-compiled web applications asp.net provides, but I've gotten used to it so I don't hate it anymore. It has advantages and disantages:
Pro
pre-compiled web applications are easy to deploy, often you'll only have to update the bin-directory
pre-compiled web applications perform well
you don't have to upload source code, which is nice imho.
Con
updating a pre-compiled web app generally means the web application is reset, so unless you've changed the session state, it'll end all sessions and log everyone out
rebuilding a large web application can take some time, which is added to the time it took you to write the changes in the first place. I am sometimes impatient.
I've always liked how easy it is to just update one file in a PHP project without having to rebuild a project or something like that, on the other hand, .net has a nice IDE that allows you to debug everying, from back end (C#, VB.net) to front end (Javascript), in one package.
But again; both have advantages and disadvantages.
I wouldn't draw such a sharp distinction between compiled and interpreted languages - this is really just an implementation detail, and tends to change with time (faster than the languages themselves change.) Case in point - thanks to Facebook, PHP is now a "compiled language" too. Another case in point - I enjoy web development with Scheme - and my preferred Scheme implementation now runs a VM and in that sense is at least as compiled as Java is.
So I think the issues to focus on are the expressiveness of the language, its performance, and its ease of deployment - compiled vs. interpreted is only important insofar as it relates to these things.
I'm a big fan of compiled languages everywhere, if for nothing more than the static typing. On the other hand, scripting languages are very convenient -- no binaries to deal with, only text files, which is a big win for web servers.
In the end, it doesn't really matter -- use whatever language you know and feel most comfortable with for the job.
I think that speed is a key concern in a web application, in particular
how fast is it to write my code
how fast is it to fix my code
how fast is it to refactor my code
how fast is it to test my code
That is, I am concerned about the speed of the slowest link: myself. Anything else is fast enough for Twitter-like loads.
Today, the number one on my evaluation list for a new project would be Tornado and Python.
If I had a choice of platforms, of course.
Ah, Python is among the fastest in scripting languages.
For scripting languages, anyone that has a copy of your software could potentially modify your source code because it's open source.
For programming languages, anyone that has a copy the software cannot simply modify your source code because it is compiled.
So I guess, it depends upon your preferences.

Diversify programming knowledge

I've taken courses, studied, and even developed a little by myself, but so far, i've only worked with Microsoft technologies, and until now I have no problems with it.
I recently got a job in a Microsoft gold partner company for development in C#, VB.net and asp.net.
I'd like tips on how to diversify, learning technologies other than those from Microsoft. Not necessarely for finding another job, I think my job just fits me for my current interests. I think that by learning by myself other languages, frameworks, databases.. I may become a better programmer as a whole and (maybe) at the end of it all having more options of job opportunities, choosing what i'm going to be working with.
What should I start with? how should I do it?
If you're comfortable with C# and VB, learn a language that uses different paradigms. The usual suspects would be Ruby, Erlang, Haskell, Lisp. All of these are available for Windows and other platforms. You might have to get used to different tools to interact with them but that's not necessarily a bad thing.
At the risk of sounding trite, why not install some variant of Linux on a cheap desktop? The mere act of setting up a Linux box is educational.
Once you find your way around it, do some shell scripting and install things like a web server. That should keep you busy for a while. Once you past that, play with some dynamic languages like perl, ruby, python, PHP, etc.
If you're interested in other languages, just pick one and away you go. You sound like you have enough experience to be apt in another language.
If you're looking into a new desktop-development-language then I'd recommend Java or Python, both of which you'd ease into with your C# and VB.NET experience.
If you're looking into web programming, go for PHP?
Browse some source
examples and see what catches your
eye as the most interesting.
Pick up a book on that language.
Ideally, one should know at least one example from each of the major "paradigms":
Assembly (nowadays a dying art, and not that useful)
plain C
one of the OO-variants of C (C++, objective C)
Java or C# (they are very similar, probably no need to learn both)
a scripting language like Ruby or Perl
Javascript (preferrably via Crockford's book)
a non-pure functional language, e.g Scheme (PLT Scheme is a nice learning environment)
a pure-functionalal language like Haskell or OCAML
Erlang (somewhat of a class of its own)
a mathematical/statistical language like R, or J (an APL-successor)
Microsoft technologies aren't bad to start with. My advice would be:
Make sure you aquire sound knowledge about the foundations of programming and the technologies you use. The more basics you know, the more independent you'll be from the latest fads:
Read "Windows Internals" to understand the operating system you're working with. In the process, you will understand other operating systems a lot better.
Toy around with other languages. Learn the differences between statically-typed languages and duct-type languages, functional programming languages, iterative programming languages whatever.
Learn the language you use the best you can. Become John Skeet!
In other words, don't move sideways first. Dig deeper and become better at understanding what you do.
It would be a nice idea to get associated with one the open source programm on http://sf.net. That way you can even have your learning for new platform and also produce some legitimate code. Also you get to look at some good coding practices. Last but not least some giving back to the software community
Maybe think of a project that would be of use to you in your daily life and see if you could develop that in a suitable language. That way you have a goal and at the end of the project you have something useful.
Alternatively why not try learing something not directly programming related, project management might be of use for future roles or do some reading about the history of technology.
These won't add any new languages to your CV but they might add some different aspects to your thinking that might make you a more well rounded potential employee.
I see two main directions to go:
Specific technologies. Select these depending upon how you want to extend yourself, new language (perhaps scripting if you haven't done that, perhaps functional programming), or new techniques (for example, UI programming, or low-level network programming depending upon what you haven't already done), or new OS (Linux if you're a Windows person).
Or, look at higher level problems, for example Design Methods and Team organisation. Read books such as Brooks' Mythical Man Month and Beck's Extreme Pogramming. Consider how to deal with problems bigger that can be solved by one person. Read up on (Rational) Unified Process, UML. Explore revision control systems, Testing techniques, not just Unit Test, but otehr flavours. Think about how you would organise a team if you were the leader. How would the tasks be subdivided, how would communication be managed?

What does it take to make a language successful? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I have an interesting idea for a new programming language. It's based on a new programming paradigm that I've been working out in my head for some time. I finally got around to start working on a basic parser and interpreter for it a few weeks ago.
I want my new language to be successful and I want to eventually create a community around it when it's ready to release. The idea behind it is fairly innovative, so I don't expect it to gain a lot of ground in the business world, but it would thrill me more than anything else to see a handful of start ups use or open source projects use it.
So taking those aims into account, what can I do to help make my language successful? What do language projects do to become successful? What should I avoid at all costs? I'd love to hear opinions or stories about other languages -- successful or not -- so I can think about them as I continue to develop.
So far, the two biggest concerns on my mind are finding a market, access to existing libraries, having amazing tool support. What else might I add to this list?
The true answer is by having a beard.
http://blogs.microsoft.co.il/blogs/tamir/archive/2008/04/28/computer-languages-and-facial-hair-take-two.aspx
Although not specific to new programming languages, the book Producing Open Source Software by Karl Fogel (available to read online) may be contain some hints to the issue of making a community around your new programming language.
In terms of adoption of programming languages in general, it seems like the trend lately has been to have a rich library to make development times shorter.
As there isn't much detail on what your language is like, it's hard to determine whether adoption of the language is going to depend on the availability of a rich library. Perhaps your language will be able to fill a niche that has been overlooked by other languages and be able to gain users. Or perhaps it has a slick name that will draw people in -- there are many factors which can affect the adoption of a language.
Here are some factors that come to mind when thinking about recent successful languages:
Ability to leverage existing libraries in the new language.
Having an adapter to external libraries written in other languages.
Python allows access to code written in C through the Python/C API.
Targeting a platform which already has plenty of libraries available for use.
Groovy and Scala target the Java platform, therefore allowing the use of and interoperation between existing Java code.
Language design and syntax to allow increased productivity.
Many dynamically-typed languages have gained popularity, such as Ruby and Python to name a couple.
More concise and clear code can be written in languages such as Groovy, as opposed to verbose languages such as Java.
Offering features such as functions as first-class objects and closures which aren't offered in more "traditional" languages such as C and Java.
A community of dedicated users who also are willing to teach newcomers on the benefits of a language
The human factor is going to be big in wide-spread support for a language -- if people never start using your language, it won't gain more users.
Also, another suggestion that I could add is to make the development of your language open -- keep your users posted on developments in your language, and allow people to give you feedback. Better yet, let your users take part in the decision-making process, if you feel that is appropriate.
I believe that by offering ways to participate in the bringing up of a language, the more people will feel that they have a stake in the success of the new language, so the more likely it will gain more support.
Good luck!
Most languages that end up taking off rapidly do so by means of a killer app. For C it was Unix. Ruby had Rails. JavaScript is the only available programming system common to most browsers without third-party add-ons.
Another means of success is by fiat. This only works if you have significant clout. For example C#, as nice as a language as it might be, wouldn't be any where near as popular as it is now if Microsoft had not pushed it as hard as it does. Objective-C is the language of MacOS X simply because Apple says so.
The vast majority of languages, though, which lack a single killer app or a major corporate backer have gained success through long term investment of their respective creators. Perl and Python are prime examples. C++ has no single entity behind it, but it has evolved as the needs of developers have changed.
Don't worry about trying to make the language be successful; worry about using it to solve real problems and make real money.
You'll either make lots of money from using this language, or not. Once you have lots of money, others may care how you did it. Or not, either way you have lots of money.
If you don't make lots of money, nobody will want to know how you did it.
Edit based on comment: I define successful as people using it, and people use languages to solve problems, most for profit, thus successful == profitable.
In addition to making the language easy to use (which has several meanings), you should develop a comprehensive library that covers and also provides a good level of abstraction over (the following most important areas):
* Data structures and manipulation
* File I/O support
* XML processing
* Networking (plus web based technologies like HTTP/HTTPS)
* Database support
* Synchronous and asynchronous I/O
* Processes and threads
* Math
A well thought out framework that makes rapid development faster (and easier to maintain) would be a great addition. For this, you should know the currently popular frameworks well.
Keep in mind that it takes a lot of time. I think it took python about 10 years (someone please correct me if I'm wrong).
So even if your community still seems small after say, 5 years, that's not the end of the story.
"It's based on a new programming paradigm that I've been working out in my head for some time."
While laudable, odds are really good that someone has already done something with your "new" paradigm.
To make a language usable, it must build on prior art. Totally new is not a good path to success. My favorite example is Algol 68.
Algol 60 was wildly popular (back in the day, which is a while ago, admittedly).
The experts wanted to build on this success. They proposed some new paradigms, the effort split into factions. The purists put the new paradigms into Algol 68; it disappeared into obscurity. Some folks created a different version of Algol, called PL/I. It did not have any really new paradigms. It actually went somewhere and was used heavily. Another group created Pascal -- it didn't have much that was new -- it discarded things from Algol 60. It actually went somewhere ans was used heavily.
Your new paradigm must have a clear and concise summary so people can fit it into a context of where the language is usable, how it can be used, what the costs and benefits of using it are.
A "new programming paradigm" causes some people to say "why learn a completely new paradigm when the ones I have work so nicely?" You have to be very clear on how it helps to have a new paradigm.
The language and libraries must work, and work very, very well. A language that isn't rock-solid is worthless. In order to be rock-solid it must be very simple.
It has to have a tutorial that will help anyone get started with your language.
Good Framework for Common Tasks
Easy Installation/Deployment
Good Documentation
Debugger/IDE and other Tools
A popular flagship product that uses your language!
Good documentation, including a detailed reference manual as well as simple examples to get people started quickly.
Good library support so that people can actually write useful programs.
Most popular languages seem to be very strong in either or both or both of those.
Use Trojan Horse approach
C++ - The Forgotten Trojan Horse
An interesting article on why C++ can grab the heart of programmers successfully.

Flow Based Programming

I have been doing a little reading on Flow Based Programming over the last few days. There is a wiki which provides further detail. And wikipedia has a good overview on it too. My first thought was, "Great another proponent of lego-land pretend programming" - a concept harking back to the late 80's. But, as I read more, I must admit I have become intrigued.
Have you used FBP for a real project?
What is your opinion of FBP?
Does FBP have a future?
In some senses, it seems like the holy grail of reuse that our industry has pursued since the advent of procedural languages.
1. Have you used FBP for a real project?
We've designed and implemented a DF server for our automation project (dispatcher, component iterface, a bunch of components, DF language, DF compiler, UI). It is written in bare C++, and runs on several Unix-like systems (Linux x86, MIPS, avr32 etc., Mac OSX). It lacks several features, e.g. sophisticated flow control, complex thread control (there is only a not too advanced component for it), so it is just a prototype, even it works. We're now working on a full-featured server. We've learnt lot during implementing and using the prototype.
Also, we'll make a visual editor some day.
2. What is your opinion of FBP?
2.1. First of all, dataflow programming is ultimate fun
When I met dataflow programming, I was feel like 20 years ago, when I met programming first. Altough, DF programming differs from procedural/OOP programming, it's just a kind of programming. There are lot of things to discover, even sooo simple ones! It's very funny, when, as an experienced programmer, you met a DF problem, which is a very-very basic thing, but it was completely unknown for you before. So, if you jump into DF programming, you will feel like a rookie programmer, who first met the "cycle" or "condition".
2.2. It can be used only for specific architectures
It's just a hammer, which are for hammering nails. DF is not suitable for UIs, web server and so on.
2.3. Dataflow architecture is optimal for some problems
A dataflow framework can make magic things. It can paralellize procedures, which are not originally designed for paralellization. Components are single-threaded, but when they're organized into a DF graph, they became multi-threaded.
Example: did you know, that make is a DF system? Try make -j (see man, what -j is used for). If you have multi-core machine, compile your project with and without -j, and compare times.
2.4. Optimal split of the problem
If you're writing a program, you often split up the problem for smaller sub-problems. There are usual split points for well-known sub-problems, which you don't need to implement, just use the existing solutions, like SQL for DB, or OpenGL for graphics/animation, etc.
DF architecture splits your problem a very interesting way:
the dataflow framework, which provides the architecture (just use an existing one),
the components: the programmer creates components; the components are simple, well-separated units - it's easy to make components;
the configuration: a.k.a. dataflow programming: the configurator puts the dataflow graph (program) together using components provided by the programmer.
If your component set is well-designed, the configurator can build such system, which the programmer has never even dreamed about. Configurator can implement new features without disturbing the programmer. Customers are happy, because they have personalised solution. Software manufacturer is also happy, because he/she don't need to maintain several customer-specific branches of the software, just customer-specific configurations.
2.5. Speed
If the system is built on native components, the DF program is fast. The only time loss is the message dispatching between components compared to a simple OOP program, it's also minimal.
3. Does FBP have a future?
Yes, sure.
The main reason is that it can solve massive multiprocessing issues without introducing brand new strange software architectures, weird languages. Dataflow programming is easy, and I mean both: component programming and dataflow configuration building. (Even dataflow framework writing is not a rocket science.)
Also, it's very economic. If you have a good set of components, you need only put the lego bricks together. A DF program is easy to maintain. The DF config building requires no experienced programmer, just a system integrator.
I would be happy, if native systems spread, with doors open for custom component creating. Also there should be a standard DF language, which means that it can be used with platform-independent visual editors and several DF servers.
Interesting discussion! It occurred to me yesterday that part of the confusion may be due to the fact that many different notations use directed arcs, but use them to mean different things. In FBP, the lines represent bounded buffers, across which travel streams of data packets. Since the components are typically long-running processes, streams may comprise huge numbers of packets, and FBP applications can run for very long periods - perhaps even "perpetually" (see a 2007 paper on a project called Eon, mostly by folks at UMass Amherst). Since a send to a bounded buffer suspends when the buffer is (temporarily) full (or temporarily empty), indefinite amounts of data can be processed using finite resources.
By comparison, the E in Grafcet comes from Etapes, meaning "steps", which is a rather different concept. In this kind of model (and there are a number of these out there), the data flowing between steps is either limited to what can be held in high-speed memory at one time, or has to be held on disk. FBP also supports loops in the network, which is hard to do in step-based systems - see for example http://www.jpaulmorrison.com/cgi-bin/wiki.pl?BrokerageApplication - notice that this application used both MQSeries and CORBA in a natural way. Furthermore, FBP is natively parallel, so it lends itself to programming of grid networks, multicore machines, and a number of the directions of modern computing. One last comment: in the literature I have found many related projects, but few of them have all the characteristics of FBP. A list that I have amassed over the years (a number of them closer than Grafcet) can be found in http://www.jpaulmorrison.com/cgi-bin/wiki.pl?FlowLikeProjects .
I do have to disagree with the comment about FBP being just a means of implementing FSMs: I think FSMs are neat, and I believe they have a definite role in building applications, but the core concept of FBP is of multiple component processes running asynchronously, communicating by means of streams of data chunks which run across what are now called bounded buffers. Yes, definitely FSMs are one way of building component processes, and in fact there is a whole chapter in my book on FBP devoted to this idea, and the related one of PDAs (1) - http://www.jpaulmorrison.com/fbp/compil.htm - but in my opinion an FSM implementing a non-trivial FBP network would be impossibly complex. As an example the diagram shown in
is about 1/3 of a single batch job running on a mainframe. Every one of those blocks is running asynchronously with all the others. By the way, I would be very interested to hearing more answers to the questions in the first post!
1: http://en.wikipedia.org/wiki/Pushdown_automaton Push-down automata
Whenever I hear the term flow based programming I think of LabView, conceptually. Ie component processes who's scheduling is driven primarily by a change to its input data. This really IS lego programming in the sense that the labview platform was used for the latest crop of mindstorm products. However I disagree that this makes it a less useful programming model.
For industrial systems which typically involve data collection, control, and automation, it fits very well. What is any control system if not data in transformed to data out? Ie what component in your control scheme would you not prefer to represent as a black box in a bigger picture, if you could do so. To achieve that level of architectural clarity using other methodologies you might have to draw a data domain class diagram, then a problem domain run time class relationship, then on top of that a use case diagram, and flip back and forth between them. With flow driven systems you have the luxury of being able to collapse a lot of this information together accurately enough that you can realistically design a system visually once the components are build and defined.
One question I never had to ask when looking at an application written in labview is "What piece of code set this value?", as it was inherent and easy to trace backwards from the data, and also mistakes like multiple untintended writers were impossible to create by mistake.
If only that was true of code written in a more typically procedural fashion!
1) I build a small FBP framework for an anomaly detection project, and it turns out to have been a great idea.
You can also have a look at some of the KNIME videos, that give a good idea of what a flow based framework feels like when the framework is put together by a great team. Admittedly, it is batch based and not created for continuous operation.
By far the best example of flow based programming, however, is UNIX pipes which is one of the oldest, most overlooked FBP framework. I don't think I have to elaborate on the power of nix pipes...
2) FBP is a very powerful tool for a large set of problems. The intrinsic parallelism is a great advantage, and any FBP framework can be made completely network transparent by using adapter modules. Smart frameworks are also absurdly fault tolerant, and able to dynamically reload crashed modules when necessary. The conceptual simplicity also allows cleaner communication with everybody involved in a project, and much cleaner code.
3) Absolutely! Pipes are here to stay, and are one of the most powerful feature of unix. The power inherent in a FBP framework compared to a static program are many, and trivialise change, to the point where some frameworks can be reconfigured while running with no special measures.
FBP FTW! ;-)
In automotive development, they have a language agnostic messaging protocol which is part of the MOST specification (Media Oriented Systems Transport), this was designed to communicate between components over a network or within the same device. Systems usually have both a real and visualized message bus - therefore you effectively have a form of flow based programming.
That was what made the light bulb go on for me several years ago and brought me here. It really is a fantastic way to work and so much more fun than conventional programming. The message catalog form the central specification and point of reference. It works well for both developers and management. i.e. Management are able to browse the message catalog instead of looking at source.
With integrated logging also referencing the catalog to produce intelligible analysis things can get really productive. I have real world experience of developing commercial products in this way. I am interested in taking things further, particularly with regards to tools and IDEs. Unfortunately I think many people within the automotive sector have missed the point about how great this is and have failed to build on it. They are now distracted by other fads and failed to realize that there was far more to most development than the physical bus.
I've used Spring Web Flow extensively in Java Web applications to model (typically) application processes, which tend to be complex wizard-like affairs with lots of conditional logic as to which pages to display. Its incredibly powerful. A new product was added and I managed to recut the existing pieces into a completely new application process in an hour or two (with adding a couple of new views/states).
I also looked into using OS Workflow to model business processes but that project got canned for various reasons.
In the Microsoft world you have Windows Workflow Foundation ("WWF"), which is becoming more popular, particularly in conjunction with Sharepoint.
FBP is just a means of implementing a finite state machine. It's nothing new.
I realize that it is not exactly the same thing, but this model has been used for years in PLC programming. ISO calls it Sequential Flow Chart, but many people call it Grafcet after a popular implementation. It offers parallel processing and defines transitions between states.
It's being used in the Business Intelligence world these days to mashup and process data. Data processing steps like ETL, querying, joining , and producing reports can be done by the end-user. I'm a developer on an open system - ComposableAnalytics.com In CA, the flow-based apps can be shared and executed via the browser.
This is what MQ Series, MSMQ and JMS are for.
This is cornerstone of Web Services and Enterprise Service Bus implementations.
Products like TIBCO and Sun's JCAPS are basically flow-based without using this particular buzz-word.
Most of the work of the application is done with small modules that pass messages through a processing network.