Related
How do you deal with a client who has different time estimates for the software product than yours?
I am going to describe a scenario that is not mine, but that captures broadly the same problem. I am working as a subcontractor to a large company that has a programming department. The software project we are working on is in an area that the department believe they have a handle on, but because their expertise and mine are very different we tend to get different results.
Example: At the start of the project I suggested one way of development which they rubbished as being unrealistically difficult and suggested integrating a different framework (one they are familiar with) with the programming language we are using (Python) to get more or less the same result.
Their estimate for this integration: less than a week (they haven't done the integration before).
My estimate for the integration: above two weeks.
Using my suggested way to get the result needed (including using matplotlib among other libraries used elsewhere within the project): 45 minutes. This is not an estimate, the bit was actually finished in 45 minutes.
Example: for the software to be integrated with their internal system, they needed to provide a web service for me to use. They provided a broken one, though it does work with their internal tool (doesn't work with .Net or Java mainstream packages among other options). They maintain that it is my fault that the integration has taken longer than the time estimated.
The problem is not that they don't know, the problem is that they have enough knowledge about programming to be dangerous (in my opinion). Is there some guidelines for how to deal with this type of situation? A way for expectation management? Or may be I shouldn't get involved in such projects from the start and in this case what are the telltale signs?
If a client isn't happy with a time estimate, don't do the work. If they think they can do it better or faster, tell them to go ahead.
The one thing I never allow is for my estimates to be modified. That's something that caught me out early on in my career but we learn our lessons.
If clients were so good at doing the work, they wouldn't be hiring me. I'd simply point out that they hired me for my expertise so why are they disregarding that expertise. Of course, if they were to allow the scope of the project to change (i.e., less work), that would be another matter, and one up for discussion.
If you didn't lock in exactly what they were meant to provide as part of the deal, then it's a "he says, she says" situation and, unfortunately, the customer controls the purse strings. However, often, the greatest power you can have is the ability to just walk away.
No-one says you have to do the job.
Of course, all that advice above is worth every cent you paid for it :-)
I don't know your specific circumstances.
Or may be I shouldn't get involved in such projects from the start and in this case what are the telltale signs?
My answer for sure. If you can avoid those projects, do it.
Some signs : people thinking they know how to do things when you can guess they can't. The "oh no let's not use this perfectly suitable tool because I don't know it" is a major indicator that the person is technically challenged.
first of all, it is no fun to be in such an environment.
So, if you like to have fun at your job, and you do not need to take this job for extenuating financial reasons, then simply do not take the job that is not fun.
Since that is hardly realistic in many cases, you will end up with the job and need to manage the situation as best you can. One way is to make sure there is a paper trail documenting your objections and concerns with the plan. Try not to be overtly negative, but try to be constructive and present valid alternatives. Here you will need to feel out the political landscape, determine if the 'boss' will be appreciative or threatened by your commentary, and act accordingly.
Many times there are other issues that management is dealing with that you are not aware of. Be cautious of this fact, and maybe ask the management team if this is the case, again without being condescending or negative.
Finally, if you have alternatives that take less time than the meetings it would take to discuss them, just try it in a sandbox, and show it off. This would go a long way to 'proving' your points. Caution here is that you could be accused of not being a team player, or of wasting resources, or not following direction. Make sure this is mitigated by doing these types of things on your own time, or after careful consideration of how long you are spending on these things as well as how vested your boss seems to be on the alternatives.
hth
I ran into the same problem with integration. Example: for the
software to be integrated with their internal system, they needed to
provide a web service for me to use...They maintain that it is my
fault that the integration has taken longer than the time estimated.
Wow very similar to what I was experiencing with a client. The best thing I can suggest is to keep good documentation. In the end that is what saved me. When it came to finger pointing I had all of the emails and facts in order and was prepared to defend my self.
One thing I would suggest is to separate out a target/goal and an estimation. I would not change my estimate unless it involved actually removing features or something is revealed that would make it easier. Tell them you will try to hit the target in anyway you can and you care about the business goal. However, your estimate will not change. If its getting no where and they are just dense then smile and nod and take it if its the only gig around.
Was just writing about this in my blog
How to estimate the WRONG way
Im curious what your preferences and thoughts are on the idea of doing as little testing as possible behind the scenes and rolling our as many new features as possible, as quickly as possible, and testing on the production site, or troubleshooting them to hell until they're bulletproof, and then releasing them to the public.
Perhaps a middle ground may be more appropriate. Your "brand" will suffer a great deal if either:
the software you release is a steaming pile of dung (as in your former cases); or
the software is not released in a timely fashion (as in your latter case).
In both those cases, you won't be very likely to stay in business for long.
The shop I operate in recognises the fact that software will have some bugs in it. All high severity bugs must be fixed before release and all low severity bugs must have a plan in place for fixing after release.
Software maintenance (basically bug fixing and answering customer questions) is an important part of our development process.
In addition, the "cost" of fixing a bug becomes more as the discovery of said bug moves away from the developer and towards the customer.
Fixing a bug I find during unit testing involves only me though it can affect others if my stuff is delayed.
Finding a bug during system test means other phases are definitely delayed since the code has to come back and be changed and unit tested again before once again being promoted to system test.
Finding a bug after your software is live is a whole other world of pain, involving communications with customers, multiple managerial reporting lines all wanting to leave an impression of their boot in your rear end and putting any bug fix through all phases, or risking adverse effects otherwise - a particularly nasty place in the ninth circle of hell is reserved for those developers who, in fixing their bug, introduce yet another one.
Rolling out code to a production server with 'as little testing as possible' to get it live quicker is setting yourself up for a life of pain. What you're suggesting really is to get your users to test your system for you, that would be a beta program, but even before you get there you should have performed a good level of testing and be confident that the app works, otherwise you're not going to keep many users for long.
From a developer perspective I would only be happy releasing code that I am confident is working as planned. From a user perspective I wouldn't want to be using an app that kept falling over, no matter how early in the development cycle it is.
If it's not ready, then don't release it.
It rather depends on the desires of the management than on the desires of the customers. Given a choice of 'you can have it working, or you can have it Friday', the average target-and-goal loving manager will prefer to have it Friday.
If you actually have a choice, please leave it until it works. You'll save yourself and everyone else a deal of time and trouble.
Time(do it right) < Time(do it again) + Time(correct database) + Time(explain and apologise)
(Fundamental law of software engineering.)
You should test and review the code during development, before the feature is even finished.
You should test the whole feature for functionality before moving to production.
You should release a small number of features often, so that you get feedback on the feature. Even if the feature works perfectly, it may still not be exactly what the user wants, or you find that something can be improved when the feature is used in practice.
It depends on the pain levels, and expectations, of your customers and how well your customer facing staff can manage their, ahem, 'feedback'.
If your customers are expecting to go quickly to high volume mass production, on a very tight schedule with fierce competition, with what you're delivering them (think consumer electronics like mobile phones) then they won't thank you at all for any surprise. They'll be very scared of having to recall hundreds of thousands of units for an upgrade.
Perhaps you're delivering to someone who's also doing research, a university department or similar, who may bend your delivery to fit a purpose that it's not intended for. They don't mind, may even expect, problems and are happy to find a way through. They may well be excited by the features and forgive you the bugs as long as they find you're listening to their feedback.
The most skillful customer facing staff I worked with were able to judge how long it would take the customer to notice the deficiencies in the deliveries we were providing, how long it would take us engineers to plug the gaps, and realise that by the time the customers noticed the problem we'd have a patch. The customer gets an early delivery so the contract is secure, is not too inconvenienced by the bugs, is happy with the support, all in all a happy world. It's a tricky call though; If you don't release anything until it's perfect you'll never have a customer as someone will always undercut you yet release something too early and disappoint then you're going to be replaced when the opportunity arrises. Get your judgement of the patch development time wrong and your customer will be unhappy.
In short it's something about open communication, something about bluff, deceit and deception, and a whole lot about judgement to know which to do and when.
I think it depends on your userbase somewhat. For example, I choose to use cutting edge and less stable features on my linux box. But I think in general, and especially in web development where generating pageviews is usually a high priority, you want to go with what works for most people, and that's stability.
If you are performing a beta or alpha test with a handful of people, sure. However, if this is code that is meant to be used by the general public or your business, then no. Buggy code reflects poorly on the programmer, and I know that when something crashes or behaves unexpectedly it tends to annoy people.
Therefore, I would much rather release polished, thought out code that may not have as many bells and whistles than code that gives people a poor experience.
One footnote, however, is that you must know when enough is enough. Every programmer can spends eons going over every line of code, saying "Well, if I move this to here, I can get a .001% speed boost" or anything on the same line of thinking. Believe me, I have this problem as well, and tend to obsess. The skill to say something is "good enough" is a hard one to learn, but it is absolutely necessary, in my opinion.
We have been trying now for a while to assist the management (of a customer) with the implementation of a a new system that is custom developed by ourselves, to their requirements. Their old system is text based (DOS) and their employees have been using it for years. The new system is Windows GUI and have many advanced features which will make their lives easier and their organisation more efficient. The problem is that the staff do not want to adapt to the new GUI environment and they are now resorting to be unfriendly and as unhelpful as possible, often placing serious obstacles in our way. The management is adamant that implementation must proceed. The system runs trouble free. We have been consistently friendly and helpful with all parties.
Any advise would be greatly appreciated! Have you encountered something like this before and did you manage to turn it round?
Note:This question is intended to help Programmers etc. with implementation difficulties by sharing experiences and factual solutions that worked. It is not intended to be subjective and indeed Programming techniques may help to solve the problem.
Use the tool
Somebody needs to really understand how the existing tool works. Not just well enough to walk through it; but well enough to do it for real. Why not take 2 weeks and go and do their job with them? That will both improve your understanding of the tool, and may foster a better working relationship with them. And while you're there, perhaps buy the drinks once or twice - it sounds corny, but anything that lowers the hostility, and lets you communicate.
User experience
Getting a good developer (or better: designer) who understanding user-experience is probably key. You can't just completely change their tooling and expect their productivity to remain the same.
Keyboard use:
Think of tools like Visual Studio, AutoCAD, etc - in most cases you don't need the mouse, and "die hard" types wouldn't notice if you took their mouse away. Try to respect this; provide shortcuts / chords (ideally the same as the existing system).
Terminology:
Keep it the same. Don't invent new terms for things.
Talk to them?
This may or may not be possible, but getting a few key users "on board" early can be pivotal; especially if you genuinely empower them to help with the user experience.
Find the faults
In the existing system. Take away their existing pain points and they may forgive you a lot.
Unfortunately it sounds like a case of needing to close the barn doors after the horses have bolted. You really need to get grass roots buy-in on the need for an improved system before beginning the project and maintain that relationship during the development.
By having champions of the system at the "coal face" level in the business would a) make sure you meet not only the management requirements but also the users goals which is all important in a successful system and b) the users get a system to which they have been a development party not just had a system thrust on them.
Getting people to moan about the short comings of an existing system is easy. Describing possible new systems before its create in way which allows the users to comment enables them to feel some control and gives you vital feedback. Be absolutely sure to identifier those killer gripes about the old system and make sure those are addressed in the new system.
Of course this all a bit late for you. The way forward is to create a review forum with the most vocal opponents and put them in a room with you and management. Get them to defend their reasons for not wanting the new system. If you can't show how your new system is better then perhaps it isn't. If you can see how the new system might be slightly improved (the movement may only need to be small) then do that, it may go a long way to get back the feeling of involvement you missed out on before.
I would sit together with the staff or a couple of the more loud mouthed opposers, go through what they find lacking with the system and suggest some of these changes to be incorporated in a future release(s). That way they will pay more attention to your the system and also feel more a part of the process instead of just being handed something like some peon. In addition it would also help avoid any misunderstandings about the system.
Get one / more of the user to be your champion by involving them in the development process. Make sure to choose the right ones. Hopefully one that you can reason with. When launching, do a launch event. Make it a big deal. Not necessarily applied to an application, but I've seen it worked in my previous work environments. If this is too late (you went ahead without any involvement from the actual users before), well... there is always things called staff turnover, lol. Out with the old and in with the new. Make the new users your buddy :).
You have to show some kind of benefit for making the change. A demo / mockup can be useful for this. Choose a manager to demo it to and wait. Let it become his idea. Then it might move forward. Being to pushy can cause a negative knee-jerk reaction which might block further consideration of the idea.
It is sad that software often gets replaced by a management decision without any user involvement and then people wonder why the system is rejected.
I've witnessed this first hand. A guy I once worked was told to develop a new version of an application "in secret". At the end of 6 months development it was shown to the users. It didn't meet their requirements and they were angry they hadn't been involved. Needless to say the software didn't make it into production and the developer left shortly after (I felt sorry for him as he had wasted 6 months effort and actually did a real good job considering the circumstances).
The chances are that the software is inferior to the previous application- perhaps data entry is actually slower (you will be biased as you wrote it- everyone likes to think their software is better).
Re-engage with the users, do some analysis and work out what is bad about the old system. If the new system can address the grips the users have with the old system you might be able to turn this around.
Edit- who was involved in engaged with your developers? Presumably the managers at the client, who probably never use the system? This is another big mistake people tend to make- managers driving requirements.
If the old system is perfect, then it never needed to be replaced in the first place!
Here's what I'm wondering. Every night that our 3 months old baby lets us sleep, I jump to my computer and start coding my hobby projects. I have about 20 different projects that I'm working on: different types of projects, from C++ games to web apps along with some contribution to open source projects. It's truly a passion and has been for a lot of years.
Yet, when I look back, I see that I haven't been able to fully complete one of my hobby projects. I've always done the prototypes and setup the most important features, but with time instead of finishing my project I end up switching to another project that seems "so much cooler" at the moment. Hence I usually end up with buggy and incomplete games that have no end nor story, 3D engines that have the fastest PolygonDraw routine ever, yet lack to implement anything else, etc... The list is long. I think I must have written unfinished Pong over a hundred times different!
I've been told that the remedy is to write specs for my hobby projects.
On one hand, I write a lot of specs at work. I know how crucial they are for defining a product's roadmap and staying within schedule. On the other hand, specs and hobby project just quite don't seem to go along! It seems to me that the learning curve to building a game is actually what makes it fun; not the game itself. Hence the fun of losing time restructuring an entire engine, the fun of creating the most useless features, and so on...
So here comes the question: Do you ever write specifications for your hobby projects? How are they different then the ones from work? How do you manage to complete your hobby projects?
I'd be glad to know while I work on my new project: a piano sonata generator :)
I don't think writing specs is the solution to your problem. Clearly, your "hobby projects" are things that you find fun. You write the fun parts but then avoid the not fun parts that would be necessary to complete something.
If you're just "programming for fun" then good, you're succeeding. I don't think writing specs is fun.
If you really want to "finish" something, the best way isn't to write a spec, it's to not jump to another project when the fun factor dips.
It is all about 'Self project management' ... even for fun.
I feel for you ... I used to have many repos that tended to all get stuck at around revision 200 or so.
Here is what used to happen, because I didn't do enough planning, after around 200 commits, things get messy and need a rewrite ... then interest disappears because it seems like too much hassle.
I learned to write my own specs for personal use
to
Give me focus to get the job done, and not go off into feature creep lane
Remind me what I am working towards
To have great ideas before I get coding
Keep thing more fun for a longer time
For me, writing my own specs is vital to getting anything done!
You wouldn't start a business without a plan would you?
For personal projects I have tons of moleskine books filled with rough specs and ideas. When they mature, they migrate from the note books into real documents and the coding begins.
BIG EDIT: On a drive for personal efficiency and, to get projects finished. I read "Getting Things Done" ... Despite all the hippy crap about 'psyche' and various levels of mind (which im sure is not based in any science) the tips are very good.
I don't get too complicated, but listing out all of the features and requirements that you want included in your application really does help. As with most hobby projects you often don't just sit down and code them straight through for 2 months and finish them. It's an hour here, two hours there, etc. Basically it's very common to forget what you were working on last and what the original purpose of this super great idea for an application was.
If you spend a few hours writing down specs and requirements it will be very valuable to you 6 months down the road when you get some free time or your ADD switches to that project and you try to remember what it is this was suppose to do.
I just found out recently that writing specs is really the thing I need to get my projects done.
I've been a bit like you, tons of projects, jumping from one to the other and never getting things finished. Until about 6 months ago, when I started to actually write specs and have a kind of roadmap for my projects.
All that I can say is that, it actually works, because you break your projects into smaller steps, just like a race with checkpoints, and when you start to mark the checkpoints as done, it feels good, addictive and your focus will be on the finish line.
This way, you get to keep only 1 or 2 projects at the same time, but actually finish them. And of course, you have the extra and pretty valuable bonus of keeping up with the project even if you don't touch it for about a month or more. The specs will always be there to remind you of the goals and purposes of your project.
This is just my personal experience, and I believe that you should give it a try. Hopefully it will workout for you too.
I've been able to do some hobby projects and finish some of them. I try to finish them all but some i just cant muster.
The reason i think is that the amount of details that are needed to finish a projects are so many that it goes from a passion project to a chore of a project.
What helped me finish most of mine is that they stayed a passion until the finishing touches were left. So i just plowed through them.
Will a spec help, to some degree yes. They get you further into the project but almost always there's a point where the passion fades and you look for the next shiny object.
It doesn't work for me! Infact whenever I'm writing up specs I'm generally making the projects even bigger, and less likely to be finished.
Sometimes the best way to do it is to just do it.
Ze Frank explains this much better than me:
http://www.zefrank.com/theshow/archives/2006/07/071106.html (video link with swearing)
EDIT: Just to add. If you are finding you want to leave your half-finished project for a new, grand idea... do it! Don't look back!
Completion is not a requirement for your own pet projects. Nobody will blame you for not finishing stuff that barely anyone else would even bother starting.
The reason you started was because of passion. That is very important. You should not force yourself to 'slog through' in your free time. You will drain your passion which is your most vital resource.
I usually write a first set of spec when I get started.
I'm also a big fan of paper thinking, so I'll draw screens, UML, diagrams, flow charts, design elements... It's just a matter of defining the scope of your project and be able to watch what you had in mind. It really helps me think.
These documents will be my specs for the whole project. I will add others as I go, but I'm not trying to maintain the old ones as much as I would have it it was a work project: I know where I'm going and I can keep track of the changes looking at my code.
Of course, some of my hobby projects are done collaboratively. In these cases, I write down more specs in order to have a better communication with my team and I try to keep documents such as DB Diagrams up to date.
I also have several hobby projects that I have not finished. I have about 10 and have written a specification for exactly one of them, the largest in scope (also a game).
I have not finished either the ones without specifications, nor the one with. I think this is because I never publish the work or show it to anyone so it remains full of bugs and never 'finished.
I suppose that this means that regardless of whether or not you have a spec, it will not affect the success of the project as much as other factors, like having the time, motivation, help, and having confidence.
The single best thing I've ever found to help move towards completion is to have someone else working on the project with you. Find a friend (or two) who is interested in the same thing and design/code it with them. Not only do you have someone to bounce ideas off of, but you've also got someone to motivate you, not to mention progress is twice as fast so you'll hopefully finish before you give up :)
Of course, it requires source control, but you were already using that for your projects, right? :)
Do you want to finish them?
I think it's reasonable to never finish a hobby project. You can just keep working on it as long as you live. Aciddose has been working on his virtual instrument xhip for years, stubbornly never getting to 1.0, making the instrument patches people program worthless from one release to the next. Yet he and the users of his softsynth seem to be having a grand time.
Maybe if you just aim for a "release" and not being "finished" you'll be more satisfied. Betas let you keep dreaming.
Yes and no. I write notes in a notebook as I'm thinking about it, and add to it as I implement it. It is a somewhat different process from work projects where someone else may have to see the spec.
I finish about half of what I start.
I've helped with development on a range of systems from safety critical avionics to throwaway personal projects like a Sudoku solver. Obviously with the avionics systems, specifications were critical to the safe operation of the system and to prevent killing somebody, but I've never bothered with my personal projects.
I think this is because specs are generally boring to read and write. Joel wrote an interesting article about this, and how to make them better if you do write them:
Painless Functional Specifications
Unfortunately I haven't had the guts to try making my specs more fun to read at work yet.
Maybe intead of writing specs you should try working on some projects for or with other people? That could provide some external motivation. I do some web devleopment for my cousin's drive in theater, and if they need a feature they won't stop asking me about it until I finish it.
The single biggest piece of advice I could give you would be to get something out there - make the spec for your first version small enough that you actually feel you can complete it, even though it won't have nearly all the features you want.
Once you get something out there, the pressure from users of your software will be enough to hopefully keep you going on it. It also ensures that the direction you take in development is the same direction your users want you to go.
If you don't actually get any users, then don't feel so bad about dropping the project - if nobody is interested, it probably isn't worth pursuing.
If pressure from your users isn't enough to keep you focused, then open source it. If there's enough interest in it, somebody else will pick it up where you left off, and you are free to move on to bigger and better things.
Unfortunately, after writing specs for the core of the DIFL engine (don't bother looking it up, as there's no trace of it outside my home systems), I still didn't finish it up.
Short answer: developing specifications for a hobby project is neither necessary nor sufficient to guarantee completion.
That being said...
I keep an engineering notebook for all of my personal projects. I use the notebook to capture all sorts of things about the projects on which I work. This includes project motivation, valuable resources leveraged during the project, things developed over the course of the project that might potentially be reused later, key insights gained, etc. etc. It also includes, more to your question, specifications for most of the projects. I employ an agile/lean approach to creating these specifications which, for me, is compelling from a cost/benefit perspective.
btw...I have many, many personal projects that did not culminate in a complete working system. Some of these I might get around to completing 'someday maybe'. I consciously chose to stop working on some of the others because they had served their purpose (e.g. introduced me to a new technology, helped me better understand a language feature, etc.) Continuing to crank away at projects like these would have led to diminishing returns so I chose to reallocate my time to projects I felt were higher leverage.
The real question is: what is your hobby? Is it finishing a project, or tinkering. If getting the last ten yards is a chore, you have to decide if it's worth it to you. Writing detailed specs will work; so will self-flagellation if you're into that sort of self-discipline. Nothing will make it easy if it's against your make-up, so you have to decide whether the end-goal is worth anything to you.
And, just to demonstrate that there is nothing programming-specific about this point, you might really like this guy. One of the main points in his work is that conceptual artists, such as Picasso and Da Vinci never really cared about the final execution--the idea was everything, and, having asserted it, they were strangely content with someone else finishing the actual work or leaving the sketch unfinished and unpublished.
I'm not sure that writing specs is the solution to your problems (or mine which seem similar) however in the case where I want to make something more than a throwaway experiment there are a few things that help me slightly without taking the fun out of it.
Specs really are quite tight and should be technical but for a hobby approach you could write up a little bit of something similar much more loose that outlines some of the things you would like to feature and shows how they fit together in a sort of design draft. Though not as detailed or restrictive as a proper spec it might help to keep the tinkering leading in the right direction.
Secondly you could break it down and depending on your time allowances maybe add a few goals in. If you focus on building one part of the project as a time breaking it into subprojects that can be linked together at the end, it gives a feeling of progress as you move from part to part rather than feeling like you have been working on the same thing for ages and can't be bothered any more. It works if you tick it off on a list, since usually it has to happen atleast mentally anyway.
In saying this if your goal is to play with certain concepts and not actually create a final product then you probably won't because you aren't working towards it. One way might be to take the above mentioned idea of breaking it up and then find a way of adding something personally interesting into each part that bores you, maybe trying to add a challenge into it or something.
I'm not particularly experienced still learning, but this is how I keep my tinkering together(sometimes unless I hit a total block cause by inexperience) and how I've approached many multimedia and web projects on a hobby basis in past years. Though the guy who said open-source it when you get bored and let someone else pick it up, that was a good idea if you want to see your code used but have satisfied your personal goals.
I have much the same problem. One thing I've noticed that HAS helped though, is lowering my ambitions. like WAY WAY low. Writing a spec is one way to reign in the ambitions, if you have some kind of limiting rule for the spec, like "The spec can only be one page", or "the spec can be no longer than 300 words long", or "Spec only something that I can get done in one day of coding". Getting the balance right can take some practice. If you go with the last limit, you can impose the rule of MANDATORY dismissal of the project if you can't finish it in one day.
The nice thing about this, is it limits you to achievable goals. This might sound really stupid or wrong at first. Or maybe it sounds reasonable, but you just can't help it, you wanna do amazing things, not ordinary things! Not small things that you can only get done in a few hours!
but keep this in mind:
“A complex system that works is
invariably found to have evolved from
a simple system that worked. The
inverse proposition also appears to be
true: A complex system designed from
scratch never works and cannot be made
to work. You have to start over,
beginning with a working simple
system.”
—John Gall
It is SO MUCH easier to make that ambitious project, if you already have a FINISHED and WORKING project to base it on. Then the "more complex thing" CAN be a project that fits in a day. This is the ideal and philosophy I'm working towards, because I think it has the best chance of succeeding. Looking at past successful projects, the vast majority of them evolved in this way, whether it was intentional or not.
What helps me a lot is to split a new feature into small tasks that could each be done in an evening hacksession. So if I have time, I simply pick one task from the list and just finish it. This is often enough to get "in the flow" and do "just one more".
I do this only for one feature at a time so I don't get distracted by all the other cool things I could add to my application.
I constantly write specs for my projects, in work, at university and outside in my free time. The biggest weakness of a programmer is his/her memory, so I find it good to keep myself busy during my thinking time by writing down my every thought into some sort of structured document. Before you know it you've written a full database schema or have a Requirements Specification.
At the moment I'm working on improving my SQL skills, and I've been spending a lot of this free time between writing queries writing down my experienced. After a couple of tweaks I had a decent document outlining what needed to be done.
I think the core problem is not the lack of specs, but rather that finishing something (anything) is hard.
It is hard work. It may seem as if your program is 90 % done. But doing those last 10 % (rooting out all bugs, getting the application to release quality, writing documentation, etc) requires as much work as the first 90 %. And if you want to be serious about marketing your program, answering support emails, fixing other people's bugs, that's more work still. And perhaps not the kind of work you are most interested in.
It is also hard mentally. An unfinished project has unlimited potential. It is an empty canvas where you can project your unbridled ambitions, lofty ideals and revolutionary thoughts. Once it is finished and made real you have to see it for what it is. Limited. Flawed. Never as pretty as the idea that spawned it.
That said, finishing something can also be very rewarding. You learn a lot, get a reality check on your ideas, the satisfaction of having completed something and you get to see what other people think of your work.
Some advice:
Make sure that you really want to finish the project. I.e., that the rewards are worth all the hard work. (If not, then accept that fact and remain a happy tinkerer.)
Find ways of motiviating yourself through the "boring" parts. Specs, maybe, if it keeps you focused. But find whatever works for you, whether it is ticking of todo-items, rewarding yourself with a cookie or dreaming of fame and fortune.
Release early, release often. The more you save for a "big release" the bigger is the chance that that release never happens.
First release, then rewrite. When you feel the urge to do a major rewrite, do a release first, then do the rewrite (if you are still up for it). Software is never perfect. If you strive for perfection without any pressure to release your half-baked (but existing) code, then you will never be done.
Most hobby projects of mine don't really get finished either. As long as I'm working on something and learning though I don't think thats a problem. Currently I'm not writing specs, but I am practicing/training TDD. I bring it up as I write tests that are the specs. Some days I'll sit down and just create a bunch of tests outlining what the software should do. Some days I make those tests pass. Its enjoyable in that I don't have to keep the code all in my head, and at any point I can sit down and make further progress by fixing the broken tests. Things just work, its kind of surreal.
Joel's article about the Evidence Based Scheduling works for me. Though I implemented it differently.
The idea is to break the project into small tasks and give estimates, then make a forecast when your project will finish based on the time the finished tasks took to finish them.
You may think your project will take years to finish, but actually from the estimate it's just two months or less. If you work more and finish tasks quickly, you will see the finish date coming earlier.
I think the most motivating thing to proceed forward is seeing the goal coming closer you run towards.
Plus: create something you will use later. Using stuff gives you incentive to improve it later.
Looking at the cool new principles of software development:
Agile
You Ain't Gonna Need It
Less As A Competitive Advantage
Behaviour-Driven Development
The Evils Of Premature Optimization
The New Way seems to be to dive in and write what you need to achieve the first iteration of scope objectives, and refactor as necessary to have elegant solutions. Your codebase grows incrementally, and never has a big planning/hierarchical design stage. That, to me, suggests that software design (worthy though it is) has been subsumed into refactoring, because that's where the elegant code comes from, not the incremental additions to functionality.
Am I wrong?
Well the trouble with refactoring is that you need to know good design before you jump head first in. BDD/TDD are supposed to make the design emerge but without other factors, such as Domain knowledge and technical competence you will end up in trouble.
I'd say that doing it the way you describe is a recipe for disaster, I would still recommend to do the overall design up front. Of course during the project you will need to change the design, it is never set in stone, flexibility is a must! (That's where the refactoring has to come in). I would also recommend to do the more detailed design for a module just before you start coding it.
But a solid general design is worth its weight in gold: It gives all developers a common base from which to start, a common idea or perhaps vision, of the goal.
Without that everybody will do as he/she thinks best, with the result that everybody does things in a different way. And suddenly you have to refactor a lot, just to align everybody to what has emerged as the apps architecture. The resulting code is ... not very elegant.
Wrong? Partially.
"Your codebase grows incrementally, and never has a big planning/hierarchical design stage."
Correct.
"That, to me, suggests that software design (worthy though it is) has been subsumed into refactoring..."
Not quite correct.
There's a huge gulf between Big Design Up Front (BDUF) and a more Agile design approach.
BDUF dictates that all design is completed before any coding. This is still popular (just read an RFP yesterday which absolutely required all design be reviewed by the customer before any coding could begin. Sigh.)
Agile suggests that perhaps all design isn't a helpful goal. You need to do enough design that TDD will work. You can't, for example, start TDD until you have a working infrastructure that allows someone to write tests and incrementally evolve a solution knowing that there won't be a weird production deployment problem to solve.
Design is still king. Agile Design is better than monolithic design.
A consequence of Agile Design is YAGNI, DRY and Less-is-More. These don't replace design, they're a consequence of how you prioritize and do design.
BDD and TDD are ways to structure your time so you have focus on what people need, what they do and what really matters. TDD, in particular, focuses on testable behaviors of the software. Not zero-value nuance, but actual behavior.
Premature optimization is interesting, but unrelated. Even Agile teams can run down a low-value rat-hole pursuing a nuance or optimization that doesn't add any value. Premature optimization is a habit of overthinking (== "hand wringing") a technology choice without facts about actual performance.
Agile is supposed to help you focus on the big picture: What actual people will actually do with the actual software and avoid technology rat-holes.
It doesn't replace engineering. It refocuses it.
Reminds me of the following quote:
The goal is clean code that works. [...] First we'll solve the "that works" part of the problem. Then we'll solve the "clean code" part. This is the opposite of architecture-driven development, where you solve "clean code" first, then scramble around trying to integrate into the design the things you learn as you solve the "that works" problem. (Kent Beck, "TDD by Example")
There's nothing in the agile manifesto, which says that you're not allowed to think before you act. Of course you can be agile and still design up front. Architecture is best designed, so before you start coding/refactoring, you should have some ideas as to how your application should be constructed.
The point is you don't have to complete each and every step before moving on. Do as much design as you need to get started.
When you have code, you can refactor as needed, but changing the fundamental architecture through refactoring becomes hard if you start from a simple dummy application every time.
The nature of design is changing, I'd say the new way is to think before coding, but just about what will be implemented in the next iteration. See "Is design dead".
Design for today, code for tomorrow.
Depends what you're designing. If it's a complex algorithm thats going to be the next video compression standard, you can iteratate and refactor 'till the cows come home and it isn't going to get any faster. The perfomance comes from design.
Similarly, if you are writing a very large application, that will grow through regular releases, you need to put in place an architecture that will support growth, and this will be by design. You can go down a long road by jumping head first into code just to discover a dead end.
Am I wrong?
IMHO, pretty much.
Edit: The reason I make many design decisions early in the process rather than mid flow is this can often be the cheapest time to do so. For example, if we start writing an application using platform dependent technology early on, it may be a very expensive decision to reverse. If we take time to consider the platforms we want to support before starting to code, it is much cheaper. We can't and won't get everything right first time, but this does not relieve us of the duty of exploring all important design choices up front. Every tried refactoring a MS MDI program to MVC? I have, and wouldn't recommend it ;)
New? No. I was reading about agile ten years ago. And agile is just the crystalization of ideas that have been around for longer than that. It's hardly new, it just hasn't diffused everywhere yet.
As for your view of design, I think there's still a place for an overarching idea and some up front design. It's the waterfall notion that you can't make a move before requirements and design are 100% complete that has been discredited everywhere but in the large firms that still cling to it.
Let's see if we can get a definition. I'm going to suggest that there may be a book we could reference. How about the one by Martin Fowler?
"Refactoring: Improving the Design of Existing Code"
Now let's take as an example the TDD mantra:
until done do
Red (we wrote a test and it failed because there was nothing that could pass it)
Green (we designed and wrote some code and the test now succeeds)
Refactor (we need to integrate the design we did with everything else)
end
I know the Agile books tend to just say "write enough code to pass the test", but there's design implicit in that statement. By necessity. Choosing a variable name is a design decision. Not a big one, but a decision nonetheless. naming a function or method is a slightly bigger one, and so on up the food chain.
There's nothing in Agile that can ensure you always make good design decisions. Nothing in waterfall or any other process either. Agile does assert that you can't figure it all out upfront and tries - with some success in my experience - to give you a set of tools to help you make better decisions throughout the whole exercise.