We have a situation at work where developers working on a legacy (core) system are being pressured into using GOTO statements when adding new features into existing code that is already infected with spaghetti code.
Now, I understand there may be arguments for using 'just one little GOTO' instead of spending the time on refactoring to a more maintainable solution. The issue is, this isolated 'just one little GOTO' isn't so isolated. At least once every week or so there is a new 'one little GOTO' to add. This codebase is already a horror to work with due to code dating back to or before 1984 being riddled with GOTOs that would make many Pastafarians believe it was inspired by the Flying Spaghetti Monster itself.
Unfortunately the language this is written in doesn't have any ready made refactoring tools, so it makes it harder to push the 'Refactor to increase productivity later' because short-term wins are the only wins paid attention to here...
Has anyone else experienced this issue whereby everybody agrees that we cannot be adding new GOTOs to jump 2000 lines to a random section, but continually have Anaylsts insist on doing it just this one time and having management approve it?
tldr;
How can one go about addressing the issue of developers being pressured (forced) to continually add GOTO statements (by add, I mean add to jump to random sections many lines away) because it 'gets that feature in quicker'?
I'm beginning to fear we may lose valuable developers to the raptors over this...
Clarification:
Goto here
alsoThere: No, I'm talking about the kind of goto that jumps 1000 lines out of one subroutine into another one mid way into a while loop. Goto somewhereClose
there: I wasn't even talking about the kind of gotos you can reasonably read over and determine what a program was doing. Goto alsoThere
somewhereClose: This is the sort of code that makes meatballs midpoint: If first time here Goto nextpoint detail:(each one almost completely different) Goto pointlessReturn
here: In this question, I was not talking about the occasionally okay use of a goto. Goto there
tacoBell: and it has just gone back to the drawing board. Goto Jail
elsewhere: When it takes Analysts weeks to decypher what a program is doing each time it is touched, something is deeply wrong with your codebase. In fact, I'm actually up to my hell:if not up-to-date goto 4 rendition of the spec goto detail pointlessReturn: goto tacoBell
Jail: Actually, just a small update with a small victory. I spent 4 hours refactoring a portion of this particular program a single label at a time, saving each iteration in svn as I went. Each step (about 20 of them) was smallish, logical and easy enough to goto bypass nextpoint: spontaneously jump out of your meal and onto you screen through some weird sort of spaghetti-meatball magnetism. Goto elseWhere
bypass: reasonably verify that it should not introduce any logic changes. Using this new more readable version, I've sat down with the analyst and completed almost all of this change now. Goto end
4: first *if first time here goto hell, no second if first time here goto hell, no third if first time here goto hell fourth now up-to-date goto hell
end:
How many bugs have been introduced because of incorrectly written GOTOs? How much money did they cost the company? Turn the issue into something concrete, rather than "this feels bad". Once you can get it recognized as a problem by the people in charge, turn it into a policy like, "no new GOTOs for anything except simplifying the exit logic for a function", or "no new GOTOs for any functions that don't have 100% unit test coverage". Over time, tighten the policies.
GOTOs don't make good code spaghetti, there are a multitude of other factors involved. This linux mailing list discussion can help put a few things into perspective (comments from Linus Torvalds about the bigger picture of using gotos).
Trying to institute a "no goto policy" just for the sake of not having gotos will not achive anything in the long run, and will not make your code more maintainable. The changes will need to be more subtle and focus on increasing the overall quality of the code, think along the lines of using best practices for the platform/language, unit test coverage, static analysis etc.
The reality of development is that despite all the flowery words about doing it the right way, most clients are more interested in doing it the fast way. The concept of a code base rapidly moving towards the point of imploding and the resulting fallout on their business is something that they cannot comprehend because that would mean having to think beyond today.
What you have is just one example. How you stand on this will dictate how you do development in the future. I think you have 4 options:
Accept the request and accept that you will always be doing this.
Accept the request, and immediately start looking for a new job.
Refuse to do and and be prepared to fight to fix the mess.
Resign.
Which option you choose is going to depend on how much you value your job and your self esteem.
Maybe you can use the boyscout-pattern: Leave the place a little more clean than you found it. So for every featurerequest: don't introduce new gotos, but remove one.
This won't spend too much time for improvements, would give more time, to find newly introduced bugs. Maybe it wouldn't cost much additional time, if you remove a goto from the part, which although had to spend time understanding, and bringing the new feature in.
Argue, that a refactoring of 2 hours will save 20 times 15 minutes in the future, and allow faster and deeper improvements.
Back to principles:
Is it readable?
Does it work?
Is it maintainable?
This is the classic "management" vs. "techies" conflict.
In spite of being on the "techie" team, over the years I have settled firmly the "management" side of this debate.
There are a number of reasons for this:
It's quite possible to have well written easy to read properly structured programs with gotos! Ask any assembler programmer; conditional branch and a primitive do loop are all they have to work with. Just because the "style" is not current and doesn't mean its not well written.
If it is a mess then its going to be a real pain to extract the busines rules you will need if you are going for a complete re-write, or, if you are doing a technical refactoring of the program you will never be sure the behaviour of the refactored program is "correct" i.e. it does exactly what the old program does.
Return on investment -- sticking to the original programming style and making minimal changes will take minimum effort and quickly statisfy the customers request. Spending a lot of time and effort refactoring will be more expensive and take longer.
Risk -- rewrites and refactoring are hard to get right, extensive testing of the refactored code is required and things that look like "bugs" may have been "features" as far as the customer is concerned. A particular problem with "improving" legacy code is that business users may have well established work arounds that depend on a bug being there, or, exploit the existense of a bugs to change the business procedures without informing the IT department.
So all in all management is faced with a decision -- "add one little goto" test the change and get it into production in double quick time with little risk -- or -- go in for a major programming effort and have the business scream at them when a new set of bugs crops up.
And if you do decide to refactor in five years time some snotty nosed college graduate will be moaning that your refactored program is not buzzword compliant any more and demanding he be allowed to spend weeks rewriting it.
If it ain't broke dont fix it!
PS: Even Joel thinks this is a bad idea: things you should never do
Update!-
OK if you want to refactor and improve the code you need to go about it properly.
The basic problem is you are saying to the client is "I want to spend n weeks working on this program and, if everything goes well, it will do exactly what it does now." -- this is a hard sell to say the least.
You need to gather long term data on the number of crashes and outages, the time spent analysing and programming seemingly small changes, the number of change requests that were not done because they were too hard, business opertunities lost because the system could not change fast enough. Also gather some acedemic data on the costs of maintaing well structured programs vs. letting it sink.
Unless you have a watertight case to present to the beancounters you will not get the budget. You really have to sell this to the business management, not, your immediate bosses.
I recently had to work on some code that wasn't legacy per se, but the former developers' habits certainly were and thus GOTO's were everywhere. I don't like GOTO's; they make a hideous mess of things and make debugging a nightmare. Worse yet, replacing them with normal code is not always straightforward.
IF you can't unwind your GOTO's I certainly recommend that you no longer use them.
Unfortunately the language this is written in doesn't have any ready made refactoring tools
Don't you have editors with macro capabilities? Don't you have shell scripts? I've done tons of refactoring over the years, very little of it with refactoring browsers.
The underlying problem here seems to be that you have 'analysts' who describe, presumably necessary, functional changes in terms of adding a goto to some code. And then you have 'programmers' whose job appears to be limited to typing in that change and then complaining about it.
To make anything different, that dysfunctional allocation of responsibilities between those people needs to change. There are a lot of different ways to split up the work: The classic, conventional one (that is quite likely the official, but ignored, policy in your work) is to have analysts write a implementation-independent specification document and programmers implement it as maintainably as they can.
The problem with that theoretical approach is it is actually unworkably unrealistic in many common situations. In particular, doing it 'properly' requires employees seen as junior to win an argument with colleagues who are more senior, have better social connections in the office, and more business-savvy. If the argument 'goto is not implementation-independent, so as an analyst you simply can't say that word' doesn't fly at your workspace, then presumably this is the case.
Far better in many circumstances are alternatives like:
analysts write client-facing code and developers write infrastructure.
Analysts write executable specifications which are used as reference implementations for unit tests.
Developers create a domain specific language in which analysts program.
You drop he distinction between one and the other, having only a hybrid.
If a change to the program requires "just one little goto" I would say that the code was very maintainable.
This is a common problem when dealing with legacy code. Stick to the style the program was originally written in or "modernize" the code. For me the answer is usually to stick with the original style unless you have a really big change that would justify a complete re-write.
Also be aware that several "modern" language features like java's "throw" statement, or SQLs "ON ERROR" are really "GO TO"s in disguise.
Related
On on hand:
1. You never get time to do it.
2. "Context switching" is mentally expensive (difficult to leave what you're doing in the middle of it).
3. It usually isn't an easy task.
4. There's always the fear you'll break something that's now working.
On the other:
1. Using that code is error-prone.
2. Over time you might realize that if you had refactored the code the first time you saw it, that would have saved you time on the long run.
So my question is - Practically - When do you decide it's time to refactor your code?
Thanks.
A couple of observations:
On on hand:
1. You never got time to do it.
If you treat re-factoring as something separate from coding (instead of an intrinsic part of coding decently), and if you can't manage time, then yeah, you'll never have time for it.
"Context switching" is mentally expensive (difficult to leave what you're doing in the middle of it).
See previous point above. Refactoring is an active component of good coding practices. If you separate the two as if they were two different tasks, then 1) your coding practices need improvement/maturing, and 2) you will engage in severe context switching if your code is in a severe need of refactoring (again, code quality.)
It's usually isn't an easy task.
Only if the code you produce is not amenable to refactoring. That is, code that is hard to refactor exhibits one or more of the following (list is not universally inclusive):
High cyclomatic complexity,
No single responsibility per class (or procedure),
High coupling and/or poor low cohesion (aka poor LCOM metrics),
poor structure
Not following the SOLID principles.
No adherence to the Law of Demeter when appropriate.
Excessive adherence to the Law of Demeter when inappropriate.
Programming against implementations instead of interfaces.
There's always the fear you'll break something that's now working.
Testing? Verification? Analysis? Any of these before being checked into source control (and certainly before being delivered to the users)?
On the other:
1. Using that code is error-prone.
Only if it has never tested/verified and/or if there is no clear understanding of the conditions and usage patterns under which the potentially error-prone code operates acceptably.
Over time you might realize that if you would have refactored the code the
first time you saw it - That would have save you time on the long run.
That realization should not occur over time. Good engineering and work ethics calls for that realization to occur when the artifact (being hardware or software) is in the making.
So my question is - Practically - When do you decide it's time to refactor your code?
Practically, when I'm coding; I detect an area that needs improvement (or something that needs correction after a change on requirements or expectations); and I get an opportunity to improve it without sacrificing a deadline. If I cannot re-factor at that moment, I simply document the perceived defect and create a workable, realistic plan to revisit the artifact for refactoring.
In real life, there will be moments that we'll code some ugly kludge just to get things running, or because we are drained and tired or whatever. It's reality. Our job is to make sure that those incidents do not pile up and remain unattended. And the key to this is to refactor as you code, keep the code simple and with a good, simple and elegant structure. And by "elegant" I don't mean "smart-ass" or esoteric, but that displays what is typically considered readable, simple, composable attributes (and mathematical attributes when they apply practically.)
Good code lends itself to refactoring; it displays good metrics; its structure resembles both computer science function composition and mathematical function composition; it has a clear responsibility; it makes its invariants, pre and post-conditions evident; and so on and so on.
Hope it helps.
One of the most common mistakes i see is people associating the word "Refactor" with "Big Change".
Refactoring code does not always have to be big. Even small changes such as changing a bool to a proper enum, or renaming a method to be closer to the actual function vs. the intent is refactoring your code. With the exception of the end of a milestone, I try to make at least a very small refactoring every single time I check in. And you'd be amazed at how quickly this makes a visible difference in the code.
Bigger changes do take bigger planning though. I try and schedule about 1/2 a day every two weeks during a normal development cycle to tackle a bigger refactoring change. This is enough time to make a substantial improvement to the code base. If the refactoring fails 1/2 a day is not that much of a loss. And it's rarely a total loss because even the failed refactoring will teach you something about your code.
Whenever it smells, I refactor. I may not make it perfect now, but I can at least make a small step towards a better state. And those small changes do add up over time...
If I am in the middle of something when I notice the smell, and fixing it isn't trivial (or I am just before release), I may make a (mental or paper) note to return to it once I am finished with the primary task.
Practice makes one better :-) But if I don't see a solution to a problem, I put it aside and let it brew for a while, discuss it with coworkers, or even post it on SO ;-)
If I don't have unit tests and the fix isn't trivial, I start with the tests. If the tests aren't trivial either, I apply point 2.
I start to refactor as soon as I find I am repeating my self. DRY principles ftw.
Also, if methods/functions get too long, to the point where they look unwieldy, or their purpose is being obscured by the length of the function, I break it into private subfunctions that explain what is really going on.
Lastly, if everything's up and running, and the code is dog-slow, I start to look at refactoring for the sake of performance.
When implementing a new feature I often notice that the task would be much simpler if the code I'm working on was structured in a different way. In this case I usually step back, try to do the refactoring first, and only after this is done I continue implementing the new feature.
I also have a habit to track all potential improvements that come to my mind either in notes or the bug tracker. The ideas bake there for some time, some of them don't feel so compelling anymore, and the reasonable ones are implemented during a day which I dedicate to smaller tasks.
Refactor code when it needs to be refactored. Some symptoms I look for:
duplicate code in similar objects.
duplicate code in within methods of one object.
anytime the requirements have changed twice or more.
anytime somebody says "we will clean that up later".
any time I read through code and shake my head thinking "what goofball did this" (even when the goofball in question is me)
In general, less design and/or less clear requirements means more oppurtunities for refactoring.
This might sound like a joke, but really, I only refactor when things "get messy". When a simple task starts taking more time and usual, when I have to twist my mind around to remember what function is doing what and such. Also, if the code starts running slow and it's not because I'm running in a development enviroment (a lot of variable outputs and such) if I can't optimise it, I refactor the code. As you said, it's worthed on the long run.
Still, I allways make sure I have enough time to think things through before I start so I don't get in this sittuation.
Cheers!
I usually refactor when one of the following is true:
I have nothing better to do and waiting for the next project to come to my inbox
The additions/changes I'm making to the code cannot work unless, or would be better if, I refactor
I am aesthetically displeased with the way the code is laid out
Martin Fowler in his book if the same name suggests you do it the third time you're in a block of code to make another change. First time in the block, you happen to notice you should refactor, but don't have time. Second time back...same thing. Third time back-now refactor.
Also, I read the developers of a current release of smalltalk (squeak.org, I think) say they go through a couple weeks of intense coding...then they step back and look at what can be refactored.
Personally I have to resist the impulse to refactor as I code or I get 'paralyzed'.
How do you deal with a client who has different time estimates for the software product than yours?
I am going to describe a scenario that is not mine, but that captures broadly the same problem. I am working as a subcontractor to a large company that has a programming department. The software project we are working on is in an area that the department believe they have a handle on, but because their expertise and mine are very different we tend to get different results.
Example: At the start of the project I suggested one way of development which they rubbished as being unrealistically difficult and suggested integrating a different framework (one they are familiar with) with the programming language we are using (Python) to get more or less the same result.
Their estimate for this integration: less than a week (they haven't done the integration before).
My estimate for the integration: above two weeks.
Using my suggested way to get the result needed (including using matplotlib among other libraries used elsewhere within the project): 45 minutes. This is not an estimate, the bit was actually finished in 45 minutes.
Example: for the software to be integrated with their internal system, they needed to provide a web service for me to use. They provided a broken one, though it does work with their internal tool (doesn't work with .Net or Java mainstream packages among other options). They maintain that it is my fault that the integration has taken longer than the time estimated.
The problem is not that they don't know, the problem is that they have enough knowledge about programming to be dangerous (in my opinion). Is there some guidelines for how to deal with this type of situation? A way for expectation management? Or may be I shouldn't get involved in such projects from the start and in this case what are the telltale signs?
If a client isn't happy with a time estimate, don't do the work. If they think they can do it better or faster, tell them to go ahead.
The one thing I never allow is for my estimates to be modified. That's something that caught me out early on in my career but we learn our lessons.
If clients were so good at doing the work, they wouldn't be hiring me. I'd simply point out that they hired me for my expertise so why are they disregarding that expertise. Of course, if they were to allow the scope of the project to change (i.e., less work), that would be another matter, and one up for discussion.
If you didn't lock in exactly what they were meant to provide as part of the deal, then it's a "he says, she says" situation and, unfortunately, the customer controls the purse strings. However, often, the greatest power you can have is the ability to just walk away.
No-one says you have to do the job.
Of course, all that advice above is worth every cent you paid for it :-)
I don't know your specific circumstances.
Or may be I shouldn't get involved in such projects from the start and in this case what are the telltale signs?
My answer for sure. If you can avoid those projects, do it.
Some signs : people thinking they know how to do things when you can guess they can't. The "oh no let's not use this perfectly suitable tool because I don't know it" is a major indicator that the person is technically challenged.
first of all, it is no fun to be in such an environment.
So, if you like to have fun at your job, and you do not need to take this job for extenuating financial reasons, then simply do not take the job that is not fun.
Since that is hardly realistic in many cases, you will end up with the job and need to manage the situation as best you can. One way is to make sure there is a paper trail documenting your objections and concerns with the plan. Try not to be overtly negative, but try to be constructive and present valid alternatives. Here you will need to feel out the political landscape, determine if the 'boss' will be appreciative or threatened by your commentary, and act accordingly.
Many times there are other issues that management is dealing with that you are not aware of. Be cautious of this fact, and maybe ask the management team if this is the case, again without being condescending or negative.
Finally, if you have alternatives that take less time than the meetings it would take to discuss them, just try it in a sandbox, and show it off. This would go a long way to 'proving' your points. Caution here is that you could be accused of not being a team player, or of wasting resources, or not following direction. Make sure this is mitigated by doing these types of things on your own time, or after careful consideration of how long you are spending on these things as well as how vested your boss seems to be on the alternatives.
hth
I ran into the same problem with integration. Example: for the
software to be integrated with their internal system, they needed to
provide a web service for me to use...They maintain that it is my
fault that the integration has taken longer than the time estimated.
Wow very similar to what I was experiencing with a client. The best thing I can suggest is to keep good documentation. In the end that is what saved me. When it came to finger pointing I had all of the emails and facts in order and was prepared to defend my self.
One thing I would suggest is to separate out a target/goal and an estimation. I would not change my estimate unless it involved actually removing features or something is revealed that would make it easier. Tell them you will try to hit the target in anyway you can and you care about the business goal. However, your estimate will not change. If its getting no where and they are just dense then smile and nod and take it if its the only gig around.
Was just writing about this in my blog
How to estimate the WRONG way
Sometimes, when I present a part of the software development process to certain people, say the supervisor or the manager that they don't have experience say
Automated unit tests and integration tests vs. their manual functional testing.
Using code generators, and scripts for repetitive tasks.
I sometimes met with resistance. Some of the reasons are the following:
They say that that's the way we do things here. Our system works and there is no need to add in our process.
They are busy being busy. They say is their job is to get us projects and our job is to deliver it to them to their satisfaction. They are satisfied when if it is a manual system, repetitive but on time.
They are very conservative about code generators. I gave them an estimate that it takes a significant time overhead for the first project to use this and time to train my teammates since this approach is relatively new to them. The overhead for the first project to them overshadows the benefit in the long run, but I explained the convenience it is to us developers, but they are always stuck to do things the old way.
What would be your strategy for this?
Wait for a problem to show up and then make your move.
You have to be a salesman, at the end of the day. You have to tell people why your proposals will make their lives easier.
If you can back up your claims with some sort of time spent/time saved data, you're onto a winner. Another thing is to get yourself a reputation gradually, by agreeing changes be implemented in phases. Implement a simple change on a small piece of the project and prove that it made a difference to them. Then roll it out a bit more, and move onto the next thing like unit testing or code generation. Given time it'll work itself out.
I don't believe you can't force people to read books, they'll shelve 'em and think you're being obnoxious. Best thing is to get small results, and use those as stepping stones to be allowed to aim for higher goals as people realise that maybe there are better ways of doing things after all.
If you're really passionate about it, you can always invest a little of your own time, and prepare a short demo (30 mins tops) that shows them how quickly you can create a tiny app without code gen, then the same app with a couple of bits code-genned. The proof of the pudding is in the eating.
I think that only way to convince someone about something is to reveal benefits what it provides.
It's easier to ask for forgiveness than it is to get permission.
There's no objective return-on-investment style measurements for "improving" a software development process. Software development is inherently hard -- it's knowledge capture -- there must be unknowns. If everything was known, you'd already have the software in hand.
Consequently, you can't ever convince a manager of anything up front.
You can only demonstrate that you are able to done something better, cheaper or faster. When they ask what the secret to your productivity is, you can show them your tools, method or approach.
Until they ask, you don't really have enough evidence to change anyone's mind. When they finally ask, then you don't need to change their mind, you need to show them your solution.
Since they don't want to derail their "do everything by hand" schedule to invest in your tools, you have to build your tools in steps, one project at a time.
"You can get a lot farther with a smile and a gun than you can with just a smile."
- Al Capone
Just kidding, but its the first thing that cross my mind :)
The gun is a metaphor (duh), like for a bug that someone spent days figuring out that with a good process he good spend in a more fun ways.
Here's what I'm wondering. Every night that our 3 months old baby lets us sleep, I jump to my computer and start coding my hobby projects. I have about 20 different projects that I'm working on: different types of projects, from C++ games to web apps along with some contribution to open source projects. It's truly a passion and has been for a lot of years.
Yet, when I look back, I see that I haven't been able to fully complete one of my hobby projects. I've always done the prototypes and setup the most important features, but with time instead of finishing my project I end up switching to another project that seems "so much cooler" at the moment. Hence I usually end up with buggy and incomplete games that have no end nor story, 3D engines that have the fastest PolygonDraw routine ever, yet lack to implement anything else, etc... The list is long. I think I must have written unfinished Pong over a hundred times different!
I've been told that the remedy is to write specs for my hobby projects.
On one hand, I write a lot of specs at work. I know how crucial they are for defining a product's roadmap and staying within schedule. On the other hand, specs and hobby project just quite don't seem to go along! It seems to me that the learning curve to building a game is actually what makes it fun; not the game itself. Hence the fun of losing time restructuring an entire engine, the fun of creating the most useless features, and so on...
So here comes the question: Do you ever write specifications for your hobby projects? How are they different then the ones from work? How do you manage to complete your hobby projects?
I'd be glad to know while I work on my new project: a piano sonata generator :)
I don't think writing specs is the solution to your problem. Clearly, your "hobby projects" are things that you find fun. You write the fun parts but then avoid the not fun parts that would be necessary to complete something.
If you're just "programming for fun" then good, you're succeeding. I don't think writing specs is fun.
If you really want to "finish" something, the best way isn't to write a spec, it's to not jump to another project when the fun factor dips.
It is all about 'Self project management' ... even for fun.
I feel for you ... I used to have many repos that tended to all get stuck at around revision 200 or so.
Here is what used to happen, because I didn't do enough planning, after around 200 commits, things get messy and need a rewrite ... then interest disappears because it seems like too much hassle.
I learned to write my own specs for personal use
to
Give me focus to get the job done, and not go off into feature creep lane
Remind me what I am working towards
To have great ideas before I get coding
Keep thing more fun for a longer time
For me, writing my own specs is vital to getting anything done!
You wouldn't start a business without a plan would you?
For personal projects I have tons of moleskine books filled with rough specs and ideas. When they mature, they migrate from the note books into real documents and the coding begins.
BIG EDIT: On a drive for personal efficiency and, to get projects finished. I read "Getting Things Done" ... Despite all the hippy crap about 'psyche' and various levels of mind (which im sure is not based in any science) the tips are very good.
I don't get too complicated, but listing out all of the features and requirements that you want included in your application really does help. As with most hobby projects you often don't just sit down and code them straight through for 2 months and finish them. It's an hour here, two hours there, etc. Basically it's very common to forget what you were working on last and what the original purpose of this super great idea for an application was.
If you spend a few hours writing down specs and requirements it will be very valuable to you 6 months down the road when you get some free time or your ADD switches to that project and you try to remember what it is this was suppose to do.
I just found out recently that writing specs is really the thing I need to get my projects done.
I've been a bit like you, tons of projects, jumping from one to the other and never getting things finished. Until about 6 months ago, when I started to actually write specs and have a kind of roadmap for my projects.
All that I can say is that, it actually works, because you break your projects into smaller steps, just like a race with checkpoints, and when you start to mark the checkpoints as done, it feels good, addictive and your focus will be on the finish line.
This way, you get to keep only 1 or 2 projects at the same time, but actually finish them. And of course, you have the extra and pretty valuable bonus of keeping up with the project even if you don't touch it for about a month or more. The specs will always be there to remind you of the goals and purposes of your project.
This is just my personal experience, and I believe that you should give it a try. Hopefully it will workout for you too.
I've been able to do some hobby projects and finish some of them. I try to finish them all but some i just cant muster.
The reason i think is that the amount of details that are needed to finish a projects are so many that it goes from a passion project to a chore of a project.
What helped me finish most of mine is that they stayed a passion until the finishing touches were left. So i just plowed through them.
Will a spec help, to some degree yes. They get you further into the project but almost always there's a point where the passion fades and you look for the next shiny object.
It doesn't work for me! Infact whenever I'm writing up specs I'm generally making the projects even bigger, and less likely to be finished.
Sometimes the best way to do it is to just do it.
Ze Frank explains this much better than me:
http://www.zefrank.com/theshow/archives/2006/07/071106.html (video link with swearing)
EDIT: Just to add. If you are finding you want to leave your half-finished project for a new, grand idea... do it! Don't look back!
Completion is not a requirement for your own pet projects. Nobody will blame you for not finishing stuff that barely anyone else would even bother starting.
The reason you started was because of passion. That is very important. You should not force yourself to 'slog through' in your free time. You will drain your passion which is your most vital resource.
I usually write a first set of spec when I get started.
I'm also a big fan of paper thinking, so I'll draw screens, UML, diagrams, flow charts, design elements... It's just a matter of defining the scope of your project and be able to watch what you had in mind. It really helps me think.
These documents will be my specs for the whole project. I will add others as I go, but I'm not trying to maintain the old ones as much as I would have it it was a work project: I know where I'm going and I can keep track of the changes looking at my code.
Of course, some of my hobby projects are done collaboratively. In these cases, I write down more specs in order to have a better communication with my team and I try to keep documents such as DB Diagrams up to date.
I also have several hobby projects that I have not finished. I have about 10 and have written a specification for exactly one of them, the largest in scope (also a game).
I have not finished either the ones without specifications, nor the one with. I think this is because I never publish the work or show it to anyone so it remains full of bugs and never 'finished.
I suppose that this means that regardless of whether or not you have a spec, it will not affect the success of the project as much as other factors, like having the time, motivation, help, and having confidence.
The single best thing I've ever found to help move towards completion is to have someone else working on the project with you. Find a friend (or two) who is interested in the same thing and design/code it with them. Not only do you have someone to bounce ideas off of, but you've also got someone to motivate you, not to mention progress is twice as fast so you'll hopefully finish before you give up :)
Of course, it requires source control, but you were already using that for your projects, right? :)
Do you want to finish them?
I think it's reasonable to never finish a hobby project. You can just keep working on it as long as you live. Aciddose has been working on his virtual instrument xhip for years, stubbornly never getting to 1.0, making the instrument patches people program worthless from one release to the next. Yet he and the users of his softsynth seem to be having a grand time.
Maybe if you just aim for a "release" and not being "finished" you'll be more satisfied. Betas let you keep dreaming.
Yes and no. I write notes in a notebook as I'm thinking about it, and add to it as I implement it. It is a somewhat different process from work projects where someone else may have to see the spec.
I finish about half of what I start.
I've helped with development on a range of systems from safety critical avionics to throwaway personal projects like a Sudoku solver. Obviously with the avionics systems, specifications were critical to the safe operation of the system and to prevent killing somebody, but I've never bothered with my personal projects.
I think this is because specs are generally boring to read and write. Joel wrote an interesting article about this, and how to make them better if you do write them:
Painless Functional Specifications
Unfortunately I haven't had the guts to try making my specs more fun to read at work yet.
Maybe intead of writing specs you should try working on some projects for or with other people? That could provide some external motivation. I do some web devleopment for my cousin's drive in theater, and if they need a feature they won't stop asking me about it until I finish it.
The single biggest piece of advice I could give you would be to get something out there - make the spec for your first version small enough that you actually feel you can complete it, even though it won't have nearly all the features you want.
Once you get something out there, the pressure from users of your software will be enough to hopefully keep you going on it. It also ensures that the direction you take in development is the same direction your users want you to go.
If you don't actually get any users, then don't feel so bad about dropping the project - if nobody is interested, it probably isn't worth pursuing.
If pressure from your users isn't enough to keep you focused, then open source it. If there's enough interest in it, somebody else will pick it up where you left off, and you are free to move on to bigger and better things.
Unfortunately, after writing specs for the core of the DIFL engine (don't bother looking it up, as there's no trace of it outside my home systems), I still didn't finish it up.
Short answer: developing specifications for a hobby project is neither necessary nor sufficient to guarantee completion.
That being said...
I keep an engineering notebook for all of my personal projects. I use the notebook to capture all sorts of things about the projects on which I work. This includes project motivation, valuable resources leveraged during the project, things developed over the course of the project that might potentially be reused later, key insights gained, etc. etc. It also includes, more to your question, specifications for most of the projects. I employ an agile/lean approach to creating these specifications which, for me, is compelling from a cost/benefit perspective.
btw...I have many, many personal projects that did not culminate in a complete working system. Some of these I might get around to completing 'someday maybe'. I consciously chose to stop working on some of the others because they had served their purpose (e.g. introduced me to a new technology, helped me better understand a language feature, etc.) Continuing to crank away at projects like these would have led to diminishing returns so I chose to reallocate my time to projects I felt were higher leverage.
The real question is: what is your hobby? Is it finishing a project, or tinkering. If getting the last ten yards is a chore, you have to decide if it's worth it to you. Writing detailed specs will work; so will self-flagellation if you're into that sort of self-discipline. Nothing will make it easy if it's against your make-up, so you have to decide whether the end-goal is worth anything to you.
And, just to demonstrate that there is nothing programming-specific about this point, you might really like this guy. One of the main points in his work is that conceptual artists, such as Picasso and Da Vinci never really cared about the final execution--the idea was everything, and, having asserted it, they were strangely content with someone else finishing the actual work or leaving the sketch unfinished and unpublished.
I'm not sure that writing specs is the solution to your problems (or mine which seem similar) however in the case where I want to make something more than a throwaway experiment there are a few things that help me slightly without taking the fun out of it.
Specs really are quite tight and should be technical but for a hobby approach you could write up a little bit of something similar much more loose that outlines some of the things you would like to feature and shows how they fit together in a sort of design draft. Though not as detailed or restrictive as a proper spec it might help to keep the tinkering leading in the right direction.
Secondly you could break it down and depending on your time allowances maybe add a few goals in. If you focus on building one part of the project as a time breaking it into subprojects that can be linked together at the end, it gives a feeling of progress as you move from part to part rather than feeling like you have been working on the same thing for ages and can't be bothered any more. It works if you tick it off on a list, since usually it has to happen atleast mentally anyway.
In saying this if your goal is to play with certain concepts and not actually create a final product then you probably won't because you aren't working towards it. One way might be to take the above mentioned idea of breaking it up and then find a way of adding something personally interesting into each part that bores you, maybe trying to add a challenge into it or something.
I'm not particularly experienced still learning, but this is how I keep my tinkering together(sometimes unless I hit a total block cause by inexperience) and how I've approached many multimedia and web projects on a hobby basis in past years. Though the guy who said open-source it when you get bored and let someone else pick it up, that was a good idea if you want to see your code used but have satisfied your personal goals.
I have much the same problem. One thing I've noticed that HAS helped though, is lowering my ambitions. like WAY WAY low. Writing a spec is one way to reign in the ambitions, if you have some kind of limiting rule for the spec, like "The spec can only be one page", or "the spec can be no longer than 300 words long", or "Spec only something that I can get done in one day of coding". Getting the balance right can take some practice. If you go with the last limit, you can impose the rule of MANDATORY dismissal of the project if you can't finish it in one day.
The nice thing about this, is it limits you to achievable goals. This might sound really stupid or wrong at first. Or maybe it sounds reasonable, but you just can't help it, you wanna do amazing things, not ordinary things! Not small things that you can only get done in a few hours!
but keep this in mind:
“A complex system that works is
invariably found to have evolved from
a simple system that worked. The
inverse proposition also appears to be
true: A complex system designed from
scratch never works and cannot be made
to work. You have to start over,
beginning with a working simple
system.”
—John Gall
It is SO MUCH easier to make that ambitious project, if you already have a FINISHED and WORKING project to base it on. Then the "more complex thing" CAN be a project that fits in a day. This is the ideal and philosophy I'm working towards, because I think it has the best chance of succeeding. Looking at past successful projects, the vast majority of them evolved in this way, whether it was intentional or not.
What helps me a lot is to split a new feature into small tasks that could each be done in an evening hacksession. So if I have time, I simply pick one task from the list and just finish it. This is often enough to get "in the flow" and do "just one more".
I do this only for one feature at a time so I don't get distracted by all the other cool things I could add to my application.
I constantly write specs for my projects, in work, at university and outside in my free time. The biggest weakness of a programmer is his/her memory, so I find it good to keep myself busy during my thinking time by writing down my every thought into some sort of structured document. Before you know it you've written a full database schema or have a Requirements Specification.
At the moment I'm working on improving my SQL skills, and I've been spending a lot of this free time between writing queries writing down my experienced. After a couple of tweaks I had a decent document outlining what needed to be done.
I think the core problem is not the lack of specs, but rather that finishing something (anything) is hard.
It is hard work. It may seem as if your program is 90 % done. But doing those last 10 % (rooting out all bugs, getting the application to release quality, writing documentation, etc) requires as much work as the first 90 %. And if you want to be serious about marketing your program, answering support emails, fixing other people's bugs, that's more work still. And perhaps not the kind of work you are most interested in.
It is also hard mentally. An unfinished project has unlimited potential. It is an empty canvas where you can project your unbridled ambitions, lofty ideals and revolutionary thoughts. Once it is finished and made real you have to see it for what it is. Limited. Flawed. Never as pretty as the idea that spawned it.
That said, finishing something can also be very rewarding. You learn a lot, get a reality check on your ideas, the satisfaction of having completed something and you get to see what other people think of your work.
Some advice:
Make sure that you really want to finish the project. I.e., that the rewards are worth all the hard work. (If not, then accept that fact and remain a happy tinkerer.)
Find ways of motiviating yourself through the "boring" parts. Specs, maybe, if it keeps you focused. But find whatever works for you, whether it is ticking of todo-items, rewarding yourself with a cookie or dreaming of fame and fortune.
Release early, release often. The more you save for a "big release" the bigger is the chance that that release never happens.
First release, then rewrite. When you feel the urge to do a major rewrite, do a release first, then do the rewrite (if you are still up for it). Software is never perfect. If you strive for perfection without any pressure to release your half-baked (but existing) code, then you will never be done.
Most hobby projects of mine don't really get finished either. As long as I'm working on something and learning though I don't think thats a problem. Currently I'm not writing specs, but I am practicing/training TDD. I bring it up as I write tests that are the specs. Some days I'll sit down and just create a bunch of tests outlining what the software should do. Some days I make those tests pass. Its enjoyable in that I don't have to keep the code all in my head, and at any point I can sit down and make further progress by fixing the broken tests. Things just work, its kind of surreal.
Joel's article about the Evidence Based Scheduling works for me. Though I implemented it differently.
The idea is to break the project into small tasks and give estimates, then make a forecast when your project will finish based on the time the finished tasks took to finish them.
You may think your project will take years to finish, but actually from the estimate it's just two months or less. If you work more and finish tasks quickly, you will see the finish date coming earlier.
I think the most motivating thing to proceed forward is seeing the goal coming closer you run towards.
Plus: create something you will use later. Using stuff gives you incentive to improve it later.
Setup
Have you ever had the experience of going into a piece of code to make a seemingly simple change and then realizing that you've just stepped into a wasteland that deserves some serious attention? This usually gets followed up with an official FREAK OUT moment, where the overwhelming feeling of rewriting everything in sight starts to creep up.
It's important to note that this bad code does not necessarily come from others as it may indeed be something we've written or contributed to in the past.
Problem
It's obvious that there is some serious code rot, horrible architecture, etc. that needs to be dealt with. The real problem, as it relates to this question, is that it's not the right time to rewrite the code. There could be many reasons for this:
Currently in the middle of a release cycle, therefore any changes should be minimal.
It's 2:00 AM in the morning, and the brain is starting to shut down.
It could have seemingly adverse affects on the schedule.
The rabbit hole could go much deeper than our eyes are able to see at this time.
etc...
Question
So how should we balance the duty of continuously improving the code, while also being a responsible developer? How do we refrain from contributing to the broken window theory, while also being aware of actions and the potential recklessness they may cause?
Update
Great answers! For the most part, there seems to be two schools of thought:
Don't resist the urge as it's a good one to have.
Don't give in to the temptation as it will burn you to the ground.
It would be interesting to know if more people feel any balance exists.
I'm a big fan of making lists!
As soon as the urge takes you to re-write something - spend 10 minutes making a list of the things that need re-writing. Follow all of the alleys that take you further into the code that needs attention and list those, too.
Hopefully within a relatively short space of time you'll have one of two things:
A really long list that completely puts you off wanting to rewrite anything ever again.
A list which actually isn't that long, so why not indulge yourself and go for a re-write?!
I've found that spending two months fixing bugs caused by my "harmless" re-write was enough to cure me of the urge to do these sorts of things without a clear mandate/project plan to do so.
The most memorable project of this kind for me occured some 4 years ago when I was called in to a remote office to "help" with a project that was due in 1 week for major presentation to the client and was not working yet at all. The project had been primarily off-shored to India, and IMO, a project management failure resulted in a ton of spaghetti code that was too fragmented to ever work properly in its current form.
After a full day's review, I presented my opinion to the management that the project simply needed wholesale refactoring and reorganization or it would never work properly. The result of this discussion was 6 days of 20 hours work / 4 hours sleep, 2 of which I actually spent sleeping on the couch in the company lobby due to the wasted time when driving back to the hotel.
The major improvements to the code included:
Application of naming standards
Moved into source control
Development of a build process
Documentation of the individual components
Most of the original code was left in place, but simply moved and reorganized / refactored to make it sustainable in the long term. Was it hell week? Sure. Did it make the project more successful? Yep.
I can't live with spaghetti code, and I'll often donate my own personal time to address it.
How to restrain one’s self from the overwhelming urge to rewrite everything?
Become old*
As this has happened to me, I've gradually lost the urge to rewrite everything. Why?
Once you've done it a few times, you realise that you often end up worse off than you started.
Even if you're god's gift to programming and your brilliant rewrite introduces no new bugs, you'll simply fail to notice or implement about 30% of the small edge-case features/bugs which things rely on. This will cost you months in fixing
Nothing wears down your irrational exuberance like time. Often this is a sad loss, but in this case, it's a win.
*(more experience may also be a suitable substitute if you lack the free time to become old)
The impulse to rewrite is righteous, provided that:
you "freeze" the existing code (with a label)
you start the rewrite attempt in a separate branch
you prepare first some unit-test in order to ascertain the current behavior and make sure you reproduce the existing features...
That said, you have to balance the rewriting process with the measure stability of the legacy code.
"If it is not broken, do not fix it" ;)
It's not quite an answer, but reading the beautiful article The Narcissism of Small Code Differences might help.
You're absolutely right that there's a right time and a wrong time to rewrite (and destabilize) the code.
Reminds me of an engineer I knew who had a habit of diving in and doing major rewriting whenever he felt like it. This drove the QA staff for his product up the wall -- if he report a bug about something trivial, the bug would get fixed, but the engineer's major rewrite of stuff he noticed while fixing the one bug would introduce fifty other new bugs, which then the QA staff needed to track down.
So one way to cure yourself of the temptation to do rewrites is to walk a mile in the shoes of the QA engineer who has to test, diagnose, and report all those new bugs. Maybe even lend yourself to the QA department for a few months to get a personal feel for the type of work they do (if you've never done that).
Another suggestion would be to make a rule for yourself to write up notes to describe any change. Perhaps post these notes when you check in code changes to source code control. You may be motivated to make the changes as minor as possible this way.
Just don't - The rabbit hole always goes too deep. Make any small local changes that leave the place cleaner than you found it, but leave big refactoring for when you're alert and have plenty of time to burn.
A nice starter on tackling big refactorings in small stages at sourcemaking.com :
you have to make like Hansel and
Gretel and nibble around the edges, a
little today, a little more tomorrow
reread "Refactoring".
Take a piece of paper and itemize the "Bad Smells" list.
(for each smell in BadSmells() {
print smell.name;
}
Add comments to the code including item(s) from the list.
while( odorPersists() ) {
Work through the list, laser-focused on one smell at a time.
}
Personally, this is where bug tracking software such as JIRA or Bugzilla come in to place with me.
I have a hard time NOT fixing all the broken windows when I see them, but if the situation is bad enough (and time permits) I will open a ticket and either assign it to me or assign it to the group responsible, that way I don't go off on a tangent.
-- I do only what needs to be done right them, but yet the issue is documented and will be fixed in time.
-- This is not to say that small broken windows should be handled this way; small issues should always be fixed on contact IMHO.
As far as I am concerned, if you have nothing better to do than to rewrite the code then you should make everyone else aware and just do it. Obviously, if it's going to take days to get any sort of result and the changes would mean excessive downtime then it's not a good idea, but eventually that code will become a problem. Make others aware that the code is awful and try to get some sort of movement to rewrite it.
The problem with the above statement is that as a programmer/developer you will ALWAYS have other things to keep yourself busy. Just leave it on the low-priority list of things-to-do, so when you're struggling with some work you can always keep your rhythm going with the rewrite.
Joel has an article about this:
There's a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong.
I've never worked anywhere particularly agile, so what I do is:
1) Figure out whether there's a reasonable fix that doesn't involve major rewrite. If not, then clear some time (perhaps by explaining to others how difficult the fix is) and do the rewrite.
2) There is a reasonable fix without major rewrite. Apply the fix, run the tests, check it in, mark the bug as fixed.
3) Now, raise a new bug/issue (enhancement request), outlining the proposed rewrite and how it would improve the code (simpler? more maintainable? reduces coupling? affects performance or resource use?). Assign it to myself, CC anyone interested in that bit of code.
4) Give people a chance to comment, then prioritise that new bug within my existing tasks. This usually means don't do it now, because most of the time if I have one "proper" bug to fix, then I have at least two. Do it once the critical list is cleared, or next time I need to do something that isn't boring. Or maybe after a couple of days it won't seem worth doing any more, and it won't get done, and I've saved the time I would have spent doing it.
The important thing, I think, is to avoid shaving a yak every time you make a small bugfix.
The trade-off is that if the code you want to rewrite is really bad, then it's easy to under-prioritise the rewrite, and you end up spending so much time maintaining it that you don't have time to replace it with something that would require less maintenance. That needs to be borne in mind. But no matter what priority the rewrite should be, or how your organisation assigns those priorities, fixing bad design in the same area of code is not the same thing as correcting the out-by-one error that caused the crash you originally went in to deal with. This has to be considered at step 1: if the design means there are probably a whole bunch of other out-by-one errors lurking in the code, then just fixing this one is probably not "reasonable". If it really was a typo, but you happened to spot a refactor opportunity because you had the file open, then correcting it and touching nothing else is "reasonable".
Obviously the more agile your shop, the more significant a refactor you can do without it being so disruptive / time-consuming / political as to require a separate task.
If it's some code you've inherited, start making the code your own. Write unit-tests, then refactor.
Write more unit tests just to find out that the code is running perfectly fine.
If you still have the urge to rewrite it, you will have some tests to find out that your rewriten code is now failing ;)
Bad code is not something that can be avoided totally, but it can be isolated by proper abstraction. If it is indeed wasteland, which means, there is no landfills around, focusing design process is much more effective than trying to make people write perfect code.
IDon't think you SHOULD stop yourself from this. Mostly if you FEEL for the big rewrite it's mostly correct to do. I insanely disagree with Joel Spolsky on this thing...
Though it's one of few places where I disagree with him ... ;)
Stop rewriting when it is good enough. For this you need to know what good program is for you not only for your employer. After all you do programming for life not for good impression. Develop sound criteria which really would tell you that you are good and result is descent and it is time to go to next task. You should like your programs not less then your boss likes them.
Refactor only if your boss/company actually encourages it, otherwise you'll end up doing frequent extra time to bring up code to perfection... Until someone less dedicated touches it again.
Starting assumption: you already "commit early, commit often".
Then there's never a bad time to start a refactoring, because you can easily back out at any point. I find that at the beginning, it always feels like it's going to be easy, but after making a few changes and seeing the effects, I start to realise quite how big a job it's going to be. That is the time to ask whether there is really time to make the changes, or whether living with it until next time you come to this code is the better course.
The willingness to stop, throw away a half-refactored branch, and do it the nasty but quick way where appropriate, is key.
That's for refactoring, where the changes are incremental, and running (or nearly-running) software keeps you well grounded. Rewriting is different, because the time until you figure out it's going to take longer than you thought is so much greater. I may be taking the question too literally, but there's almost never a good time to throw it away and start again.