Related
I'm trying to understand how exceptions are handled in an event driven world using micro-services (using apache kafka). For example, if you take the following order scenario whereby the following actions need to happen before the order can be completed.
1) Authorise the payment with the payment service provider
2) Reserve the item from stock
3.1) Capture the payment with the payment service provider
3.2) Order the item
4) Send a email notification accepting the order with a receipt
At any stage in this scenario, there could be a failure such as:
The item is no longer in stock
The payment information was incorrect
The account the payee is using doesn't have the funds available
External calls such as those to the payment service provider fail, such as downtime
How do you track that each stage has been called for and/or completed?
How do you deal with issues that arise? How would you notify the frontend of the failure?
Some of the things you describe are not errors or exceptions, but alternative flows that you should consider in your distributed architecture.
For example, that an item is out of stock is a perfectly valid alternative flow in your business process. One that possibly requires human intervention. You could move the message to a separate queue and provide some UI where a human operator can deal with the problem, solve it and cause the flow of events to continue.
A similar thing could be said of the payment problems you describe. If an order cannot successfully be settled, a human operator will need to investigate the case and solve it. For that matter, your design must contemplate that alternative flow as part of it, and make it so a human can intervene somehow when the messages end up in a queue that requires a person to review them.
Those cases should be differentiated from errors or exceptions being thrown by the program. Those cases, depending on the circumstance, might in fact require to move the message to a dead letter queue (DLQ) for an engineer to take a look at them.
This is a very broad topic and entire books could written about this.
I believe you could probably benefit from gaining more understanding of concepts like:
Compensating Transactions Pattern
Try/Cancel/Confirm Pattern
Long Running Transactions
Sagas
The idea behind compensating transactions is that every ying has its yang: if you have one transaction that can place an order, then you could undo that with a transaction that cancels that order. This latter transaction is a compensating transaction. So, if you carry out a number of successful transactions and then one of them fails, you can trace back your steps and compensate every successful transaction you did and, as a result, revert their side effects.
I particularly liked a chapter in the book REST from Research to Practice. Its chapter 23 (Towards Distributed Atomic Transactions over RESTful Services) goes deep in explaining the Try/Cancel/Confirm pattern.
In general terms it implies that when you do a group of transactions, their side effects are not effective until a transaction coordinator gets a confirmation that they all were successful. For example, if you make a reservation in Expedia and your flight has two legs with different airlines, then one transaction would reserve a flight with American Airlines and another one would reserve a flight with United Airlines. If your second reservation fails, then you want to compensate the first one. But not only that, you want to avoid that the first reservation is effective until you have been able to confirm both. So, initial transaction makes the reservation but keeps its side effects pending to confirm. And the second reservation would do the same. Once the transaction coordinator knows everything is reserved, it can send a confirmation message to all parties such that they confirm their reservations. If reservations are not confirmed within a sensible time window, they are automatically reversed by the affected system.
The book Enterprise Integration Patterns has some basic ideas on how to implement this kind of event coordination (e.g. see process manager pattern and compare with routing slip pattern which are similar ideas to orchestration vs choreography in the Microservices world).
As you can see, being able to compensate transactions might be complicated depending on how complex is your distributed workflow. The process manager may need to keep track of the state of every step and know when the whole thing needs to be undone. This is pretty much that idea of Sagas in the Microservices world.
The book Microservices Patterns has an entire chapter called Managing Transactions with Sagas that delves in detail on how to implement this type of solution.
A few other aspects I also typically consider are the following:
Idempotency
I believe that a key to a successful implementation of your service transactions in a distributed system consists in making them idempotent. Once you can guarantee a given service is idempotent, then you can safely retry it without worrying about causing additional side effects. However, just retrying a failed transaction won't solve your problems.
Transient vs Persistent Errors
When it comes to retrying a service transaction, you shouldn't just retry because it failed. You must first know why it failed and depending on the error it might make sense to retry or not. Some types of errors are transient, for example, if one transaction fails due to a query timeout, that's probably fine to retry and most likely it will succeed the second time; but if you get a database constraint violation error (e.g. because a DBA added a check constraint to a field), then there is no point in retrying that transaction: no matter how many times you try it will fail.
Embrace Error as an Alternative Flow
As mentioned at the beginning of my answer, not everything is an error. Some things are just alternative flows.
In those cases of interservice communication (computer-to-computer interactions) , when a given step of your workflow fails, you don't necessarily need to undo everything you did in previous steps. You can just embrace error as part of you workflow. Catalog the possible causes of error and make them an alternative flow of events that simply requires human intervention. It is just another step in the full orchestration that requires a person to intervene to make a decision, resolve an inconsistency with the data or just approve which way to go.
For example, maybe when you're processing an order, the payment service fails because you don't have enough funds. So, there is no point in undoing everything else. All we need is to put the order in a state that some problem solver can address it in the system and, once fixed, you can continue with the rest of the workflow.
Transaction and Data Model State are Key
I have discovered that this type of transactional workflows require a good design of the different states your model has to go through. As in the case of Try/Cancel/Confirm pattern, this implies initially applying the side effects without necessarily making the data model available to the users.
For example, when you place an order, maybe you add it to the database in a "Pending" status that will not appear in the UI of the warehouse systems. Once payments have been confirmed the order will then appear in the UI such that a user can finally process its shipments.
The difficulty here is discovering how to design transaction granularity in way that even if one step of your transaction workflow fails, the system remains in a valid state from which you can resume once the cause of the failure is corrected.
Designing for Distributed Transactional Workflows
So, as you can see, designing a distributed system that works in this way is a bit more complicated than individually invoking distributed transactional services. Now every service invocation may fail for a number of reasons and leave your distributed workflow in a inconsistent state. And retrying the transaction may not always solve the problem. And your data needs to be modeled like a state machine, such that side effects are applied but not confirmed until the entire orchestration is successful.
That‘s why the whole thing may need to be designed in a different way than you would typically do in a monolithic client–server application. Your users may now be part of the designed solution when it comes to solving conflicts, and contemplate that transactional orchestrations could potentially take hours or even days to complete depending on how their conflicts are resolved.
As I was originally saying, the topic is way too broad and it would require a more specific question to discuss, perhaps, just one or two of these aspects in detail.
At any rate, I hope this somehow helped you with your investigation.
I am looking for feedback on a certain directory structure for an application. I realize that this does not follow the classical stack overflow format where there is such a thing as "a correct answer", though think it is interesting nonetheless. To provide meaningful feedback, some context first needs to be understood, so please bear with me.
--
Two colleagues of mine and I have created an application that uses the Clean Architecture. HTTP requests to routes get turned into request models, which gets handed to use cases, which then spit out a response model that gets handed to a presenter.
The code is fully open source and can be found on GitHub. We also have some docs describing what the main directories are about.
We are thinking about reorganizing our code and would like to get feedback on what we've come up with so far. Primarily amongst the reasons for this reorganization are:
Right now we do not have a nice place to put things that are not part of our domain, yet somehow bind to it. For instance authorization code, which knows about donation ids (with authorization not being part of the core domain, while donation ids are).
It's nice to group cohesive things together. Our Donation code is cohesive and our Membership Application code is cohesive, while both don't depend on each other. This is closely related to the notion of Bounded Contexts in Domain Driven Design. Right now these contexts are not explicitly visible in our code, so it is easy to make them dependent on each other, especially when you are not familiar with the domain.
These are the contexts we have identified so far. This is a preliminary list and just to give you an idea, and not the part I want feedback on.
Donation
Membership
Form support stuff (validation of email, generation of IBAN, etc)
The part I want feedback on is the directory structure we think of switching to:
src/
Context_1/
DataAccess/
Domain/
Model/
Repositories/
UseCases/
Validation/
Presentation/
Authorization/
Context_2/
Factories/
Infrastructure/
tests/
Context_1/
Unit/
Integration/
EdgeToEdge/
System/
TestDoubles/
Context_2/
The Authorization/ folder directly inside of the context would provide a home for our currently oddly placed authorization code in Infrastructure. Other code not part of our domain, yet binding to it, can go directly into the context folder, and gets its own folder if there is a cohesive/related bunch of stuff amongst it, such as authorization.
I'm happy to provide additional information you need to provide useful feedback.
Right now we do not have a nice place to put things that are not part of our domain, yet somehow bind to it.
Right now these contexts are not explicitly visible in our code, so it is easy to make them dependent on each other, especially when you are not familiar with the domain.
There are both technical and non-technical ways to address this issue:
You can enforce stricter separation through class libraries. It is more obvious you are taking a dependency on something if you have to import a dll / reference another project. It will also prevent circular dependencies.
Code reviews / discipline is a non-technical way to handle it.
I've been using Hexagonal Architecture with DDD where the domain is in the middle. Other concerns such as repositories are represented by interfaces. Your adapters then take a reference to the domain, but never in the other direction. So you might have an IRepository in your domain, but your WhateverDatabaseRepository sits in it's own project. It is then the responsibility of the application services / command handlers to co-ordinate your use cases and load the adapters. This is also where you would apply cross-cutting concerns such as authorization.
I'd recommend watching Greg Young videos (try this one) and reading Vaughn Vernon's IDDD as it goes into how to structure applications and deals with questions like yours. (sorry that my answer is basically watch a 6hr video and read a 600+ page book, but they both really helped clarify some of the more "wooly" aspects of DDD for me)
As an example, see https://github.com/gregoryyoung/m-r/blob/master/SimpleCQRS/CommandHandlers.cs
These days I've read several articles about BDD to find what it is talking about. Now I get a basic understandings, but still not clear about the whole process.
The following is what I think to do in a BDD process:
All the stakeholders(BA, customer, Dev, QA) are sitting together to discuss the requirements, and write the agreed features on story cards. Here I take a "user registeration" feature as example:
As a user,
I want to register on the system,
so that I can use its services
Create several scenarios in Given/When/Then format, and here is one of them:
Scenario: user successfully register
Given an register page
And an un-registered user
When the user fills username "Jeff" and password "123456"
And click on "Register"
Then the user can see a "Success" message
And the user "Jeff" is created in the system
Implement this scenario with some BDD testing framework, say cucumber-jvm, like:
import cucumber.api.java.en.Given;
public class Stepdefs {
#Given("an register page")
public void an_register_page throws Throwable {
// ...
}
#Given("an un-registered user")
public void an_register_page throws Throwable {
// ...
}
// ...
}
for the steps one by one.
But I soon find myself in trouble: there are pages, models, maybe databases need for this scenario, seems a lot of thing to do.
What should I do now?
What should I do now? Do I need to discuss this scenario with all the stakeholders? For BA/Customer/QA, I don't think they really care about the implementations, is it a good idea to discuss it with some other developers?
Suppose after I discuss it with some other developers, we agree to split it to several small parts. Can we make these small parts as "scenario"s with Scenario/Given/When/Then format as what we just did with cucumber-jvm, or can we use JUnit as we normally do in TDD?
1. If choose "cucumber-jvm", it seems a little heavy for small part
2. If choose JUnit, we need to involve more than one testing framework in a project
3. Is it the best if there is a single testing framework to do both things (not sure if there is)
Suppose I choose option 2, using JUnit for the small tasks
The following is what I will do after this decision:
Now we create new small tests to drive implementations, like creating user in database, as we normally do in TDD. (red->green->refactoring). And we don't care the cucumber test Scenario: user successfully register (which is failed) for now, just leave it there. Right?
We develop more small tests with JUnit, made them red -> green -> refactored. (And the imcomplete cucumber test is always failed)
Until all the small tests are passed, we turn to the cucumber test Scenario: user successfully register. Completed it and make sure it turn green at last.
Now develop another scenario, if it's easy, we can implement it just with cucumber, otherwise we will have to split it and write several jUnit test
There are definitely a lot of mis-understandings, even very basic ones. Because I don't find myself gain much value from BDD except the "discuss with all stakeholders" part.
Where is my mistake? Appreciate for any sugguestions!
Don't start with logging in; start with the thing that's different to the other systems out there. Why is someone logging in? Why do they want to use the service? Hard-code a user, pretend they're logged in, focus on the value.
If you focus on UI details you tie yourself to the UI very strongly, and it makes the UI hard to change. Instead, look at what capabilities the system is delivering. I don't recommend using a login scenario anyway, but if I did, I'd expect it to look more like:
Given Jeff isn't registered with the site
When he registers with the username "Jeff" and password "123456"
Then his account creation should be confirmed
And he should be invited to log in for the first time.
Look up "declarative vs. imperative" here to see more on this.
If your UI is really in flux, try out the scenario manually until the UI has settled down a bit. It will be easier to automate then. As you move into more stable scenarios it will be better to automate first (TDD-style).
What should you do now? Well, most people don't write class-level tests for the UI, so don't worry about it until you start driving out the controller and presenter layers. It's normally easier to have frameworks in the same language, but two different frameworks is fine. Cucumber / RSpec, JBehave / JUnit, SpecFlow / NUnit are pretty typical combinations. Do the smallest amount you need to get that first scenario working. It won't be much, because you can hard-code a lot of it. The second scenario will start introducing more interesting behaviour, and then you'll start to see your class-level tests emerge.
BTW, BDD started at a class level, so you can do the same thing with the classes; think of an example of how you might use it, write "Given, When, Then" in comments if your framework doesn't work that way already, and then fill in the gaps!
Yes, your Cucumber scenario will be red throughout, until it isn't.
Ideally you'll be making one last unit test and the Cucumber scenario pass at the same time, rather than just writing a bit of extra code. It's very satisfying to see it finally go green.
The original point of BDD was to get rid of the word "test", since it causes people to think of things like TDD as being about testing. TDD's really about clean design; understanding the responsibilities and behaviour of your code, in the same way that scenarios help you understand the capabilities and behaviour of your system. It should be normal to write both system-level scenarios and class-level tests too.
You're already ahead of all the people who forget to discuss the scenarios before they start coding, though! The conversations with stakeholders are the most important part. You might get value out of including a tester in those conversations. Testers are very good at spotting scenarios that other people miss.
It looks like you're pretty much on the right track where the rest of the process is concerned. You might find some of the other BDD answers in my profile helpful for you too. Congrats and good luck!
I think doing registration/sign_in first is a really good thing to do when you are learning the mechanics of doing BDD. Pretty much everyone understands why you would want to sign into a system, and everyone understands that the system has to know who you are before you can do this, so you have to register first.
Doing this simple task allows you to concentrate on a smaller subset of BDD. By narrowing your focus you can improve quality, whilst being aware that there is much more to learn a little later on.
To write your sign in scenarios you need to focus on two things:
writing scenarios
implementing step definitions
These are the basic mechanics of BDD, but they are only a small part of the overall process. Still I think you'd benefit from working on them because at the moment you are not executing the mechanics very well, which is to be expected because you are new to this.
When you write scenarios you should concentrate on 'what' you are doing and 'why' you are doing it. Scenarios have no need to know anything about 'how' you do things. Anything to do with filling in stuff, clicking on stuff etc. is a smell. When your scenarios only deal with the what and why they become much simpler.
Feature: Registration
A pre-requistite for signing in, see sign_in.feature
Scenario: Register
Given I am a new user
When I register
Then I should be registered
Feature: Sign in
Dependant on registration ...
I want to sign in so I can have personalised content and ...
Scenario: Sign in
Given I am registered
When I sign in
Then I should be signed in
You really don't much more than this to drive the development of a simple sign_in system. Once you have that running you can deal with some sad paths e.g.
Scenario: Sign in with bad password
Given I am registered
When I sign in with a bad password
Then I should not be signed in
And I should be told ...
If you implement things nicely this sad path scenario should be trivial to implement as all the infrastructure is already in place to sign in, all that is different is you are using a bad password.
You can see an example of this at https://github.com/diabolo/cuke_up. The way to use this example is follow the commit history, and in particular notice how I am using the extract_method refactor to take all the code out of the step definitions. Each method I extract is a tool to reused when writing subsequent scenarios. Making an effective set of tools is the key to productivity when implementing scenarios.
Nowadys sign_up is so simple because we can rely on a 3rd party library and their unit tests. This means we can get pretty good results without ever having to worry about the transition to our own code and doing bits of TDD. So for now there really is no need to think about TDD.
So long as you are aware that you are only doing a small subset of BDD, I think you can successfully use this approach to provide foundations for all the extra stuff you have to deal with when working with the things that differentiate your system from others.
To summarize, just focus on
writing simple scenarios
making your step definitions elegant
creating tools (extracted methods) that can be used in the next scenario you write
You have plenty of time to learn the other stuff, and it will be much easier if your basic mechanics are better developed.
When using distributed and scalable architecture, eventual consistency is often a requirement.
Graphically, how to deal with this eventual consistency?
Users are used to click save, and see the result instantaneously... with eventual consistency it's not possible.
How to deal with the GUI for such scenarios?
Please note the question applies both for desktop applications and web applications.
PS: I'm working with the Microsoft platform, but I imagine the question applies to any technology...
A Task Based UI fits this model great. You create and execute tasks from the UI. You can also have something like a task status monitor to show the user when a task has executed.
Another option is to use some kind of pooling from the client. You send the command, and pool from the client until the command completed and the new data is available. You will have a delay in some cases from when the user presses save to when he will see the new record, but in most cases it should be almost synchronous.
Another (good?) option is to assume/design commands that don't fail. This is not trivial but you can have a cache on the client and add the data from the command to that cache and display it to the user even before the command has been executed. If the command fails for some unexpected situation, well then just design a good "we are sorry" message for misleading the user for a few seconds.
You can also combine the methods above.
Usually eventual consistency is more of a business/domain problem, and you should have your domain experts handle it.
I think that other answers mix together CQRS in general and eventual consistency in particular. Task-based UI is very suitable for CQRS but it does not resolve the issue with eventually consistent read model.
First, I would like to challenge your statement:
Users are used to click save, and see the result instantaneously... with eventual consistency it's not possible.
What do you by this? Why is it not possible to see the result immediately? I think the issue here is your definition of result.
The result of any action is that that action has been performed. There are numerous of ways to show this! It depends on what kind of action do you want to complete. Examples:
Send an email: if user has entered a correct email address, it is almost guaranteed that the action will complete successfully. To prevent unexpected failures one might use durable queues since this kind of actions do not need to be done synchronously. So you just say "email sent". Typically you see this kind of response when you ask to reset your password.
Update some information in a user profile: after you have validated the new data on the client, most probably the command will succeed too since the only thing that could happen is the database error (if you use database). Again, even this can be mitigated by using durable queues. In this case you just show the updated field in the same form. The good practice for SPA is to have a comprehensive data store on the client side, like Redux does. In this case you can safely update the server by sending a command and also updating the client-side store, which will result in UI to shows the latest data. Disclaimer: some answers refer to this technique as "tricking the user", but I disagree with this definition.
If you have commands that are prone to error, you can use techniques that are already described in other answers like Websockets or Server-side events to communicate errors back. This requires quite a lot of additional work. You can also send a command and wait for reply or execute commands synchronously. Some would say "this is not CQRS" but this would be just another dogma to be challenged. Ensuring the command has completed the execution in combination with the previous point (client-side data store) will be a good solution.
I am not sure if there is any 100% bullet proof technique that allows you to always show non-stale data from the read model. I think it goes against the principles of CQRS. Even with real-time events you will only get events that indicate that you write model has been updated. Still, your projections could have failed and reacting on this is a whole other story.
However, I would not concentrate that much on this issue. The fact is that well-tested projections and almost-guaranteed commands will work very well. For error handling in 90% of situations it is enough to have some manual or half-manual process to recover from those errors. For the last 10% you can combine generic "error" messages pushed from the server saying "sorry, your action XXX has failed to execute" and the top priority actions could have some creative process behind them but in reality those situations would be very very rare.
There are 2 ways:
To trick a user (just to show that things has happened then they
really hasn't happened yet)
Show that system is processing request
and use polling in background (not good) or just timer with value of
your SLA.
I prefer the 1st option.
As someone has already mentioned, task based UI's fit well for this, and what I would do is employ a technique that 'buys you time' for the command to propagate.
For example, imagine we are on a list screen, where the user can perform various actions, one of which being to add a new item to the list. After choosing to add an item you could display a "What would you like to do next?" which could have 'Add another item', 'Do this task', 'Do some other task', 'Go back to list'.
By the time they have clicked on an option, the data would have hopefully been refreshed.
Also, if you're using a task based UI, you can analyse the patterns of task execution and use these "what would you like to do next" screens to streamline the UI. Similar to amazon's "other people also bought these items".
As previously stated, it is fine to tell the user that the request (command) has been acknowledged (successfully issued). In case of some failure, the system should communicate this to the requester, by means of:
email;
SMS;
custom inbox (e.g. like the SO inbox);
whatever.
E.g., mail client / service:
I am sending a mail to a wrong address;
the mail service says: "email sent successfully :)";
after few minutes, I receive a mail from the service: "email could not be delivered".
I believe a great way to inform the user about a recent failure is to present him an error panel while he's navigating through the application. A user gesture might be required in order to dismiss that alert etc.
For example:
I wouldn't go with tricking the user or blocking him from committing some other actions. I would rather go for streaming data toward UI after they are being acknowledged by a read side. Let's consider these two cases:
Users saves data and expects result. Connection is established toward server. After they are being acknowledged by a read side, they are streamed toward UI and UI is being updated.
User saves data and refreshes web page. Upon reload, data are being fetched from data store and connection for streaming is established. If read side didn't update the data store in the meantime, there's still an opened stream and UI should be updated after data reaches the read side.
Why streaming from read side and not directly from write side? Simply, that would be a confirmation that read side has been reached.
From technical aspect, Server-Sent Events could be used.
Disadvantage:
Results will still not be reflected immediately by a read side. But at least, in most cases, user will be able to continue with his work without being blocked by a UI.
There are several ways to handle eventual consistency. All of them are really to occupy the time from the User's action until the backend refresh.
User Reads A given user can only read from the same database node that they write to. Other users read from the replicated nodes. PROS: UI is quick enough, and application stays in sync. CONS: Your service architecture has to track and route Users to specific database nodes.
Disable the UI until the action has completed, and refresh it. Java Server Faces has a classic example of this. One could create a modal with a loading spinner to cover the UI until the refresh was completed. PROS: UI stays in sync with application state. CONS: Most every action creates a blocked UI. Users get very frustrated by the restricted UI, and will complain of application slowness.
Confirmation Immediately thank the user for their submission. Then let them know later (email, SMS, in-app notification) whether or not the action was completed. PROS: It's fast up front. CONS: UI lags behind system until refresh. Even with a notice, the User may get confused that they don't see the updates. It also requires integration of various communication channels. Users won't see their changes right away. If the action fails, they may not know until it's too late.
Fake it Optimistically assume that the action will complete. Show the User the resulting UI (upvote, comment, credit card confirmation, etc) and allow them to continue as if it succeeded. If there were failures, immediately show them as contextual errors: alerts next to the undone upvotes, in-app alert on the post with the failed comment, email for the declined credit card. PROS: UI feels much faster. CONS: UI is temporarily out of sync with application state, and you must resolve that. One case: you might fake creation of content with temp IDs. But after content is created, then the temp IDs will be wrong until the refresh. Second case, you might need to store all state changes on the UI after the action until the refresh. Then you need some Resolver to apply all the local state changes since the action was issued. This is resolution is non-trivial.
Web Sockets Subscribe the UI to an event stream so that when the action is completed on the backend, it is pushed to the front end. Is it one-way or two-way streaming? PROS: UI feels fast, and it's in sync with the application state. CONS: Consistent browser support, need a backend source of streaming events, and socket server scalability.
Some of my recent web projects that I worked on, use a flow engine as the central abstraction in the presentation and/or (more or less the) business layer. Reflecting on my experiences, I can honestly say that I am not a fan of the flow-centric approach. On the contrary even. I see the same symptoms pop up in projects that use flows as central abstraction.
Everything is a flow. You don't just start an application, no, you "enter the main flow" even if it is just to show a menu with a huge dispatcher behind it. I am not against flows as such. Some use cases keep popping up everywhere and need to be included at various points in other use cases (LookupCustomer, ...), but for flow-centric people everything is a flow, even things that are... not flows.
Fragmentation. Flow-based applications tend to have many pieces of logic (actions, commands, fragments of code to prepare the view...) dispersed throughout the code. Mapping in and out of these actions adds overhead, is tedious and bloats the code. Although it is easy to follow the abstract flow, actually figuring out what is happening inside these little (or big) chunks of code is another thing. While every style of application allows people to write bad and inconsistent code, flow-centric applications make it particularly easy to do so.
Config files. Most applications use some XML format to declare flows and actions that accompany state changes. The language in which the application is written (say Java, C#, Ruby, ...) is probably far more richer and expressive than the XML format ever will be. Why bother?
Flows break encapsulation. If you give me a component that has a certain embedded flow logic, then the flow should be part of the component, and should not be an external abstraction. In other words: the flow is part of the component and the component is self-contained. It is a detail. Sure, it can be parameterized and stuff, but a component should "just work". People writing a Swing, GWT, or whatever fat or rich interface application, don't bother with explicit flow abstractions. Why should my web application have one? Give me the flow diagram of GMail.
(Edit) Flows are procedural. If you look at "rich" patterns like MVC with events and everything, flows really pale in comparison. You are using an modern and expressive language to implement your application, right? So you can do better than the rigid "do this, then that, and that, and ..." way from the time when punchcards and assembler were in fashion.
Examples of frameworks that promote flow-centric development are Struts, BTT, Spring Webflow, and JSF. I've also come across homegrown flow engines built on top of ordinary servlets.
This is also an interesting article: http://chillenious.wordpress.com/2006/07/16/on-page-navigation/
Do you (still) think a flow-centric approach for (the front-end of) a web-application is a good idea?
In general, flows seem to be an unnecessarily enterprisey approach to what should be a relatively simple problem: we would like to ensure that users take one of several particular paths through our application. What's more instructive and insightful is to examine why we need this path to occur. Is it because...
... we don't want them to interact with our application except in rigidly predefined ways? Then we've limited the utility of our application, and we make our application much harder to change and use.
... we're worried about the ability of our application to handle unexpected input or deal with states we haven't anticipated if people stray off the beaten path? Then that says a lot about our technical choices for a validation framework.
... we can't envision a scenario other than the predefined ones under which someone would use the site? Then we are implicitly assuming that only we know how best to use it; we limit the ability of the user to control their interaction.
Notice how each of these underscores an issue intrinsic to the application's development and team members, and one that's not the fault of a user. So I support your general premise that flow-based approaches tend to have a number of issues.
The primary problem is that flows unnecessarily increase brittleness that is already better abstracted by other mechanisms. For example, to achieve a rule like "you need to fill out your order form before you confirm checkout", don't make a workflow; have a better CustomerOrder model that knows when it doesn't have all the information necessary to allow an OrderConfirmation. If you try to skip ahead, your model and controller should take care of failing validation on the next POST.
Essentially, flows extract out disparate fragments of each participating controller and collect them into a new "flow controller" that's specific to each flow. That's not necessarily a bad idea, but it suggests that the original controllers may have been taking on too much responsibility to begin with if that sort of path was so easy to define separately. For example, if you previously had OrderConfirmation, CustomerOrder, and OrderCheckout controllers, and you're thinking about an Order flow to link all three together, what you should probably be thinking about is an Order controller instead.
I think defining flows is useful in a web application. In answer to your main points:
Everything is a Flow.
There is nothing intrinsically wrong with that, it's just a name to give something. A flow can be short or long - I agree it's a bit weird that there is a "main" flow that starts everything but it doesn't really cause any problems in practice.
Fragmentation
You have some valid points here, although I get the feeling that the greatest contributor to this is the design of the DSL. For example, Spring WebFlow v2 is a vast improvement over SWF v1 in terms of readability and understandability.
Config files
I strongly disagree with this point. I feel that xml is the best way to express this code. If you think about it - managing controllers, views, state changes and actions is really just "configuration" rather than "code". And xml (in my opinion) is the best way to express configuration. Just think about the word "Controller". All a controller does is direct and configure things - call services, return views and models etc. There is no need for any richness or expressiveness of Java to define what is basically just configuration of your web application.
Flows break encapsulation
GMail could expressed in a series of flows. Think about the number of steps it takes to compose and send an email. Flows really just define the wiring of how the application works - sure you could have a number of components that interact with each other, but the way that you configure them to work together is essentially the flow you have defined in your application. Making this flow explicit in a separate DSL seems like a good idea to me, as fundamentally it is separate.
The first question that should be answered is whether a flow framework is really the best tool for your specific web application. I'm a fan of Spring Web Flow, myself, but I'll only use it if my web app can easily be broken down into flows, and if navigation should be tightly controlled. If the navigation is very loose, where you can get to almost any page from any other page, then SWF isn't the right tool for the job.
As you mention, there are other drawbacks to flow frameworks. They usually aren't RESTful, and thus not bookmarkable. If that conflicts with what you want for your application, then SWF probably isn't for you.
That said, SWF, and some of the other flow frameworks, offer some features that few other web frameworks deliver. This includes complete solutions for double-submit issues and browser back button and history handling. SWF's implementation of these features lends some additional security. Since the flow execution IDs for each page change as the application is used, you get immunity to forced browsing and some protection against cross-site request forgery.
The concept of flows is quite nice, in my opinion, since flows tend to mirror use cases. Scoping data to a flow or a conversation removes the responsibility for its cleanup from the developer, which I think is a very good thing. It's like the difference between manual memory management and garbage collection. Not only does it make less work for me, but it eliminates the possibility of introducing bugs should I forget to cleanup attributes. One thing I hated about Struts was that I needed to duplicate my cleanup code in several actions to ensure correctness. It's much easier just to scope the data to the use case.
Flows also present a context for related actions and views. If I look at a struts-config or faces-config file, I can see all kinds of navigation rules or action mappings, but there is no immediate context for me to mentally group related items together. I have to manually trace through the configuration, and even then sometimes I get stuck. With Struts, I need to look at the specific web pages in order to figure out which actions can be invoked from a view.
With SWF, I can clearly see all the actions, views, and models related to a flow. With Eclipse plugins, I can see this as a state diagram. Even if you're not using eclipse, it's very easy to translate a flow definition to a state diagram. These diagrams are useful for myself, my project manager, and pretty much anyone who wants to understand the high-level of how a use case is performed. In short, chunking related things together allows for easier understanding, and a shallower learning curve. That's one reason why OOP is so popular. With web apps, the idea of chunking these elements together to form a use case just feels natural.
Everything is a flow
Everything really is a flow. Computer programs had always been a flow and will always be a flow containing of theese processes:
input -> process -> output
The MVC design pattern in fact corresponds with this..
controller -> model -> view
Fragmentation
You're right. But I think this "issue" might be reduced by a good suppport in IDEs.
Config files
There's no doubt xml is the best way to express configuration.
Flows break encapsulation
I would disagree with this. You can make black boxes using flows and then use these black boxes in another flows.
IMHO, web apps are best developed as independent modules rather than modules that are "bound by flow".
Since most web apps today are ajaxy apps, having independent modules on the page helps a lot.
Configurations can be handled by XML or JSON files.
Web 2.0 presents a serious challenge to the notion that "everything is a flow". And when the presentation tier is fully transposed to the client layer, we'll be back on the solid, and familiar (from GUIs of yore) ground of event based processing.
Flows arise because of the inherit mismatch between traditional application interaction and the way web applications actually work. Flows are merely a convenient way to describe what would be more traditionally modeled as a series of GUI dialogs (think wizard) in a way that is compatible with the way in which web pages are delivered and interacted with. Imagine if you will that you were writing a traditional program, but every time the user ran the program you could only display a single dialog box, and when the user clicked "Ok" (or "Cancel", or "Next", or "Previous") your program would terminate. In that situation, how would you go about modeling the expected behavior of the program (to further complicate matters, assume many users are running the program at different times)? I think you would find you would rather quickly arrive at something similar to flows.
I think perhaps what you're really asking is, "Why are most flow frameworks so easy to abuse?", which naturally leads to the followup question "What can be done to fix that?".