<< Testy Development | Home | The Technical Manager >>

TDD Considered Harmful?

posted @ Friday, February 22, 2008 7:30 PM

Firefighter Man, I’m going to catch it for that title. Okay, let me get this out up front: this post is a hypothesis that I consider probable. That’s why there’s a question mark in the inflammatory title. Okay, asbestos underwear in place, let’s continue.

The Hypothesis

Here’s the logic flow for my hypothesis.

  1. Unit testing (and its bosom buddy automated testing) provides substantial benefits when used regularly. These benefits are well-understood.
  2. Ensuring or regularizing unit testing is the primary value proposition of Test-driven Development (TDD).
  3. A majority of vocal unit testing advocates are also TDD proponents.
  4. Developers searching for unit testing information are more likely to find references that describe unit testing as subservient to or at least included in TDD than they are to find references that are simply about unit testing.
  5. Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn’t have had a problem with simpler unit testing methods.

Simple enough, really. If 1, 2, 3, and 4 then 5.

I’m not going to bother trying to verify statements 1 and 2. You might be able to quibble with 2, but it’d take some real mental gymnastics to do so.

Testing Propositions 3 & 4

Propositions 3 and 4 are key to this hypothesis being correct so let’s explore them. Here’s a simple test: what are the top 10 links on Google for unit testing and are the addresses they link to substantially advocates for test-first development. This test assumes that the top results from a Google search will be representative of what unit testing advocates are saying. Since Google earns money by the metric ton to make sure this is the case, I’m going to let the results stand for both 3 and 4.

To help with the analysis, I’ll include two rankings for each link. Both rankings are my subjective opinion; included so that you can check my figures for yourself if you wish by seeing how much you agree with my evaluation (or if you agree at all, I suppose). Both are a scale from 0 to 5.

The first scale, Helpful, will indicate whether a developer looking to find initial unit testing information is likely to read the whole thing. A five is something I think most developers will stop to read and digest. A 2 or 3 is either too general or too specific and thus might or might not be interesting to a developer looking for initial unit testing information. Lower scores indicate links that make me question Google’s ranking system.

The second scale, Test-first Focus, will indicate how focused that link is on writing unit tests before coding. A five is something that puts unit testing squarely in the realm of test-first development and includes substantial explanations on how to do so or a base assumption that testing first will be practiced by those reading. A 3 means that it contains information on testing first, but in a more neutral tone—i.e. unit tests will be written when the developer determines it is important with testing first given as a (mere) option. Lower scores indicate that testing first is mentioned in passing if at all. And yes, I am going to lump TDD into XP and other test-first methodologies because they will tend to lead to step 5 as much as formal TDD will.

Okay, here’s the top 10 Google hits for unit testing as of this moment:

Link Description Helpful Test-first Focus
Unit testing - Wikipedia, the free encyclopedia This is the wikipedia entry for unit testing. As is common for wikipedia, the article is concise and well-focused. TDD is mentioned but only briefly and as a "related topic". 3 1
Effective Unit Testing A magazine article supposedly about unit testing. Third sentence: "What if I told you we are going to start with unit testing?" 2 5
Extreme Rules A page on unit testing at extremeprogramming.org. "you should try to create your tests first, before the code." 4 4
Six Rules of Unit Testing A developer’s list of six rules for unit testing. Good post, actually. "1. Write the test first" 5 5
Chapter 13. Unit Testing A chapter from Dive Into Python. The link itself is only the introduction to the chapter, but the very next section leads with this: "Now that you’ve completely defined the behavior you expect from your conversion functions, you’re going to do something a little unexpected: you’re going to write a test suite that puts these functions through their paces and makes sure that they behave the way you want them to. You read that right: you’re going to write code that tests code that you haven’t written yet." 2 5
JUnit, Testing Resources for Extreme Programming JUnit is an automated unit test framework for java. I’ve never used it, but I understand it is the gold standard for Java unit testing frameworks and the conceptual ur-progenitor for unit testing frameworks in other environments. 3 0
Unit Test A page from WardsWiki. I’ve no idea what that is, but its primary focus seems to be software development. It’s almost more of a forum than a wiki, though. That said, the introduction for the topic includes this: "Key here is to CodeUnitTestFirst. " 1 4
Testing, fun? Really? An old blog post (March 1st, 2001) about unit testing in Extreme Programming. "Unit tests are so important that you should write your tests before you write the code." 5 5
Unit Testing in scriptaculous wiki A wiki for the script.aculo.us Java Script unit test framework. I didn’t know they had those. Cool. 2 0
Unit Testing in BlueJ A pdf manual for unit testing in BlueJ (I’ve no idea what BlueJ is and don’t feel like looking it up). Chapter 10: Writing tests first. This is actually a little misleading, though because the chapter is about how to test first if you want to. 1 3

Well, fancy that. Over half the entries are a 4 or 5 for their test-first focus. Of those that aren’t directly test-focused, one is wikipedia and the others are unit testing frameworks (I think—that last one is weird).

I’m going to call propositions 3 and 4 proven.

So Then

Here’s the question: given that 3 and 4 turn out to be true, do you think that "Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn’t have had a problem with simpler unit testing methods"? Personally, I think this is likely the case.

Since the barrier to implementation is higher for TDD than it is for POUT, you have to acknowledge that more developers will be willing to spend the resources on POUT than will do so for TDD. The question is how many are we talking here? How can we find out? I find TDD a much more substantial resource hit than POUT. Thus, at least some people would be willing to spend resources on POUT that wouldn’t spend them on TDD. You can double-check this by trying to imagine someone being willing to spend resources on TDD who would hesitate to spend them on POUT. Do you think there are any? I don’t.

If this is a trend in the general population, then what happens when someone who would opt for POUT if they knew about it is only able to find information for TDD? It’d be only natural if they skipped unit testing altogether.

TDD folks gripe about projects with no unit testing. Non-TDD folks resist and resent being badgered about implementing a methodology they aren’t convinced will justify its cost. I wonder if we aren’t locked in an unnecessary struggle between poles with each side providing ammunition to the other.

I wonder what would happen if the TDD folk started with encouraging people to unit test regularly. After all, TDD folks would have to admit that having some unit tests is better than no unit tests. How about starting your testing discussions with POUT? Or how about defaulting to POUT when you run into resistance for TDD and push for more later if you can? How badly do you want regular unit testing? Badly enough to let go of presumptive TDD discourse? I’ve noticed that TDD advocates never discuss unit testing outside of a TDD framework. Since many of the most popular developers fit this category, I wonder if we aren’t unwittingly encouraging a TDD or nothing effect in the wild.

I wonder what would happen if every TDD advocate went on the record that POUT is way better than no unit testing (with examples and/or encouragement and without going on to propound the vast superiority of TDD). Will you TDD advocates reading this accept the challenge from me to do so?

And here’s an interesting question you can address: how much of the advantages of TDD do you think can be achieved with just good design and timely unit testing? 90%? 50%? 20%?

Technorati Tags: ,,

Comments

  1. Dave^2

    Posted on: 2/22/2008 8:01 PM

     # re: TDD Considered Harmful?

    I disagree with 2. Ensuring unit testing is a side effect of TDD. TDD is primarily about design (as mentioned in the comments of your previous TDD posts).

    No mental gymnastics required :)

  2. Jacob

    Posted on: 2/23/2008 3:19 AM

     # re: TDD Considered Harmful?

    @Dave: I wish you had tried some gymnastics before trying to dismiss my point with something so flimsy. Here's the thing: TDD is a methodology. The only relationship that a design has to a methodology is as a limitation. Design literally cannot be a benefit of a methodology. Why is that, you might ask? Simple, because in the absence of the methodology (in this case, TDD), a developer is free to select the very best design based on a project's specific attributes and needs. If a methodology forces a particular design (or limits the use of particular designs), then there is a chance that the optimal design for a project will be excluded by use of that methodology.

    I know this because a big part of my frustration with TDD is being forced into strict SoC. Say what you want about the virtues of SoC, the fact is that no design is perfect for every situation (just as every design has a space where it rules supreme--however small that space may be). What I am saying is that even if I thought strict SoC was the bees knees and absolutely required in every project I tackle, TDD has absolutely no bearing on that fact, nor does it have any bearing on my ability to implement SoC to my heart's content.

  3. Dave^2

    Posted on: 2/23/2008 4:49 AM

     # re: TDD Considered Harmful?

    I'd try some mental gymnastics on ya, but I'm a bit underpowered in the intellect department <dopeyGrin />. Instead I'll resort to flimsy arguments.

    "The only relationship that a design has to a methodology is as a limitation. Design literally cannot be a benefit of a methodology... In the absence of the methodology (in this case, TDD), a developer is free to select the very best design"

    I disagree. TDD does not limit your design options. It gives you rapid feedback on your design decisions, starting before you even write the production code. It might be tricky to write a test for code that doesn't have SoC -- that is feedback from the process telling you your code is tightly coupled. You can ignore that feedback. You advocate POUT, so you'll be writing a test for the unit regardless of how tricky it is, right? Same design, same test written, but if you are using TDD the coupling is immediately obvious, giving you a opportunity to change the design before you have committed to it.

    That argument and semantics aside, it doesn't change the fact that the *intention* of TDD is to help developers produce a good design. That is explicitly stated in all the literature on TDD I have ever read (books by Beck, Martin, Feathers, blogs by Jeremy Miller, Phil Haack et al), and is supported by my experience with TDD. The whole push behind BDD is to help stop the misconception that the point of TDD is encouraging unit testing.

    Improved design is the "primary value proposition" of TDD. To pretend otherwise, as you do in point 2, is misleading.

    Now I know you believe that TDD does not improve design, but that's a different argument entirely (and one we've discussed before) :)

  4. Dave^2

    Posted on: 2/23/2008 6:17 AM

     # re: TDD Considered Harmful?

    Despite our disagreement over point 2, I should add that I completely agree with you that everyone should be encouraged to do unit testing, regardless of whether they use TDD. All the TDD material I can recall reading has supported that explicitly.

    Another point I'm not entirely convinced by is your assumption that POUT is significantly easier than TDD. For most of the people and teams I've worked with, writing tests that work with their design is the hard part.

    If their design is keeping them from unit testing, then is it possible TDD might give them a lower barrier to entry for unit testing than going straight to POUT?

    And in answer to the last question on your post: "how much of the advantages of TDD do you think can be achieved with just good design and timely unit testing?". If by "advantages" you mean "positive results", I would say 100%. TDD is a tool to help you reach a goal. It is not the only way to get there.

  5. Mark Levison

    Posted on: 2/24/2008 8:39 PM

     # re: TDD Considered Harmful?

    My reply was far too long for a comment. So I blogged it instead.

    See: http://www.notesfromatooluser.com/2008/02/test-driven-dev.html

  6. William Pietri

    Posted on: 2/25/2008 12:35 AM

     # Look at the history.

    People have been talking about unit testing for decades, but almost nobody did it. Now they are. I think it's no coincidence that most of the people serious about unit testing are of the TDD school.

    In my view, test-first development is much easier than test-after, and for me it's much more effective. I would be very hesitant to suggest to somebody that they start with test-after; I don't think it will be a stepping-stone to good testing.

    Instead, I think it will be what it has been for many years: a way to convince people that testing is too hard and that they should just skip it if they want to get anything done.

  7. Miles Thompson

    Posted on: 2/25/2008 1:11 AM

     # re: TDD Considered Harmful?

    thought my example might shed some light.. our development shop approach to these things went like this..
    1. "TDD/Unit Tests/Wha?"
    2. Read a bunch of stuff including TDD hype and JUnit things, attend meetings/conferences where the hype gets spread further.
    3. Figure it out by reading code. Figure out how the Unit test libraries work. Play with green light GUIs for a bit, realise that they are kinda pointless.
    4. Install (in our case) NUnit and TestDriven.NET. Start writing tests.
    5. Occasionally say to each other "you know you/we really I guess you 'ought' to write the tests first".. generally, though, we just write the tests when it suits - which is generally (thought not always) just after writing the code when you first want a nice simple/fast-to-start way to start 'testing' out the code you just wrote.

    Its a fact that I remember saying 'hey you know you 'ought' to write the tests first'. I do try to do that sometimes (for clarity) but generally just do it when it suits.

    Practically wins over talk in our case.. and I suspect in most peoples cases. Programmers look to code for examples of what they should do - and all the code examples are POUT basically. 'Methodologies' have a very weak hold in the real world of programming, 'out there'.

  8. David Arno

    Posted on: 2/25/2008 2:23 AM

     # re: TDD Considered Harmful?

    You have it spot on William: the history shows that OUT has been around for decades and no one did it. Along came TDD and now some people unit test.

    The reason is simple: TDD forces you to do unit tests first and so they get done. Traditional unit testing was scheduled after the coding was done. Coding always runs late and so unit testing always got shelved.

    This is a case of the "bleedin' obvious" and it is a puzzle that Jacob refuses to see it.

  9. Alex McMahon

    Posted on: 2/25/2008 3:13 AM

     # re: TDD Considered Harmful?

    In my opinion, one of the problems is that people don't recognise the difference between what Jacob calls POUT and the traditional perception of unit testing; As far as I am aware Jacob hasn't suggested that unit tests be written after all the development work is complete, rather they are written iteratively as you write bits of code.

    I would definately say that POUT is better than no unit testing and also better than 'traditional' unit testing (test-at-the-end). I like the idea of TDD, and I think unit testing needed an exciting idea that people could promote such as TDD. I struggle to see the big selling point for moving from POUT to TDD from a unit testing perspective. I can see the argument that TDD is not really about unit testing, and it's more of a bridge to BDD.

    Jacob, keep up the good work in putting forward a different view. Whether you are right or wrong, healthy debate, rather than blind following has got to be beneficial all round.

  10. Troy DeMonbreun

    Posted on: 2/25/2008 9:33 AM

     # re: TDD Considered Harmful?

    Here, Here!

    Yes, my humble request to all TDD evangelists, at least explain that POUT is a valuable subset of TDD and is SIGNIFICANTLY more effective than NO TESTING at all.

    No matter how simple you (as a TDD evangelist) believe TDD is to learn and implement, it is more of a barrier to entry than POUT.

  11. Jeff Santini

    Posted on: 2/25/2008 11:08 AM

     # re: TDD Considered Harmful?

    I think the gymnastics are the sticking point. The idea that TDD is not primarily about design seems less than self-evident. Agreed based on the chosen acronym, and as other's have said the BDD crowd are intent on rectifying this situation.

    What seems to follow from the concept of TDD being about Unit Testing is that a linear progression exists from no testing to Unit Testing to TDD.

    If you acknowledge that, for some TDD practitioners, the practice is more about design than unit testing then the linear progression breaks down and we can move past the idea that TDD is a step beyond just Unit Testing.

    Of course, many people do TDD and there are probably many reasons to do it. But it is not hard to find references in the literature about TDD providing early and clear pointers towards a clean design.
    http://www.developer.com/design/article.php/3622546
    http://blog.objectmentor.com/articles/2008/01/10/generated-tests-and-tdd
    http://www.codeodor.com/index.cfm/2008/2/11/Stumbling-Blocks-and-Rewards-in-Test-Driven-Development/1980

    Not claiming the above links are correct, just pointing out they are easy to find. These came from a Google search on "TDD Design Benefits"

    One other reason I would suggest for doing TDD is clarity of purpose. If I have stated what I want to achieve via a test, before I begin achieving it, I will have a more difficult time veering off track, or progressing beyond the needs of the client, or for that matter, stopping short of the needs of the client. The test expresses my understanding of the problem. For me, without the test written, I can fool myself into thinking I know what I should do without having to provide any evidence. It is analogous to the idea that if you think you understand a concept, go explain it to someone who doesn't. If you can't explain it clearly to them, you probably don't understand it very well yourself. If you can't write a test to state your expectations of the code you probably aren't ready to solve the problem yet. And if you do understand the problem well enough to write an executable test for it, then why not write it?

    I have seen the position above countered with the argument "I don't need TDD to force good design on me". That same argument can be made about seatbelts and car safety, a healthy diet and longevity, or modern programming languages and GUI driven applications. There are ways to achieve the second item in the combo without the using the first, but statistics favor users of the first item when trying to achieve the second.

  12. Jeff Santini

    Posted on: 2/25/2008 2:09 PM

     # re: TDD Considered Harmful?

    Sorry, in my post above I start by stating "The idea that TDD is not primarily about design seems less than self-evident". I meant to say "The idea that TDD is not primarily about TESTING seems less than self-evident"

  13. Troy DeMonbreun

    Posted on: 2/25/2008 2:35 PM

     # re: TDD Considered Harmful?

    Jacob,

    As first I read the statement "Ensuring or regularizing unit testing is the primary value proposition of Test-driven Development (TDD)", I thought to myself "Boy, Jacob, I hope you ARE wearing your asbestos undies."

    However, short of Dave^2's comments, there seems to be little resistance to the idea so far.

    I am shocked, as I figured the last thing that most TDD evangelists would accept was that TDD was a sort of POUT+ or "POUT sandbox".

    Maybe the threat of "gymnastics" was too intimidating. Or maybe the sound of this shotgun blast hasn't reached the coasts, yet. ;-)

  14. Jacob

    Posted on: 2/25/2008 3:22 PM

     # re: TDD Considered Harmful?

    @jeff:

    I have seen the position above countered with the argument "I don't need TDD to force good design on me". That same argument can be made about seatbelts and car safety, a healthy diet and longevity, or modern programming languages and GUI driven applications. There are ways to achieve the second item in the combo without the using the first, but statistics favor users of the first item when trying to achieve the second.

    I've made that argument myself and stand by it. All of your examples are fatally flawed as a comparison to TDD and good design. The TDD for design argument is about protecting me from myself from an outcome my actions will absolutely and solely determine (or that a team determines collectively). My own analogy would be wearing safety goggles while cross-stitching to that I won't poke myself in the eye. It could be useful if I were prone to poking myself in the eye while holding a needle, but as I'm not I don't see the point.

  15. Jeff Santini

    Posted on: 2/25/2008 5:03 PM

     # re: TDD Considered Harmful?

    Perhaps it is a difference in world views at issue. This will sound a little corny. But the me that starts a project is often not the same person that ended it. Hopefully during the course of a project I will have learned something more about the business domain in which I am working, maybe learned a new tool, library, or language. With that learning comes different decisions. With learning you have introduced an unknown into the development process. So something besides myself(the agent that taught me) has at least partially determined the outcome of my project. If you introduce a team into the equation, then I think absolute and sole control of outcomes becomes much more tenuous.

    So in my view, though I may be in complete control of me, I have no idea what tomorrow will bring in my software creation world. For that reason I chose the analogies I chose, and stand by the form I chose. Though I do not begrudge you the position that you don't need the help TDD offers in building towards a stronger design.

    OK. So what if I drop the "You are never in an absolute position of sole control" argument. Even if sole and absolute control were possible there are helpful and unhelpful tools. Let me attempt an example more strictly analogous to yours. For me, in this context, TDD would be more like MS Word for an author. I can write my novel by hand, but it will be much easier if Word is around to help.

    OK. so that was all just a bit of wordplay yeah? Getting back to the point of your post, I would reiterate from my original comment, that TDD people do not say "At least do Unit Testing" because for most TDD practitioners TDD is about something besides Unit Testing. The evidence is easy to find, and I think it would be hard to take a serious look at TDD and not see what I am talking about. I have been doing TDD for 5 years now, and one of the first lessons I learned is that writing tests first is a means to an end, and the end is not test coverage.

    Here are some early examples from the mists of 2000 to bolster the examples given in my previous comment.

    From the xp site. Here Design is mentioned as an advantage, but not as THE advantage(It was early days after all).
    http://www.extremeprogramming.org/rules/testfirst.html

    Here is Micheal Feathers leading us through an example of TDD and discussing the design benefits. He talks about Growing an application. The gist is that TDD helps you grow a design.
    http://www.xprogramming.com/xpmag/test_first_intro.htm

    These links were found by searching "Test First" on Google. Test First was what it was called BEFORE people realized that design was such a large part of TDD.

    Hope this is helpful.

  16. Paul Ingles

    Posted on: 2/26/2008 3:00 AM

     # re: TDD Considered Harmful?

    @jacob: The poking yourself in the eye analogy is interesting. In my experience (and I've worked with a fair few number of teams across all types of businesses) you'd likely never know, until it was too late. Less like poking yourself in the eye and more like losing your ticket for a train, boarding, and then realising when the conductor comes you've lost it.

    Of course, you may be the exception to the rule, but I would imagine that you probably do make mistakes- you are after all human.

    For example, I'm working with a team at one of the largest UK investment banks on their trading and pricing application. There's little evidence of any test first development (the system is *way* too coupled for one). But, there are flourishes of unit testing (or POUT if you're so inclined towards cute acronyms). Unfortunately, since they were done after-the-fact, they *only* prove the system works. They have not provided any other benefit. Of course, understanding what it means when I say it *works* is itself a problem- since the tests are too large and difficult to understand, and are more a statement the code does what the author wrote it to do (as opposed to describing what the code *ought* to do), but I'm willing to accept it's possible to write good unit tests _without_ writing them first (since that's where I started 6 years ago). Just that it's orders of magnitude more difficult and as a result, less likely.

    As it stands our productivity is excruciatingly and painfully slow - I've worked on teams where we've delivered whole releases in the time it takes a single (small) feature to be developed on this system. Productivity is low because the design isn't good. And by good design I don't mean *my* interpretation of how *I'd* like things to look and feel, but the ability to maintain and change things- it's about having a team go quicker with more confidence.

    I would suggest that by ignoring the immediate feedback of writing clients for your code you've poked yourself (and others) in the eye a few times, you just don't know it :)

  17. William Pietri

    Posted on: 2/26/2008 9:52 AM

     # Poking yourself in the eye indeed

    It could be, Jacob, that your design skills cannot be improved upon even an iota by writing tests first. That's what I'm taking from your comment, anyhow.

    But consider that a lot of people who think TDD improved their design skills thought they were pretty good designers before. Some of them, like Martin Fowler and Bob Martin, were published OO experts. Kent Beck himself was the OOPSLA program chair a decade before publishing about test-first programming.

    So you might be right; you might just be a naturally better designer than every single person who thinks--like I do--that TDD helped step up their game. You might have nothing to learn. But given your strong skepticism toward others' bold claims, you can see how others might be reluctant to just take your word for it, yes?

  18. Jacob

    Posted on: 2/26/2008 4:23 PM

     # re: TDD Considered Harmful?

    @Paul & Jeff: Good comments. I think our disagreement actually ends up highlighting some things I've brought up before: our assumption that we're all doing the same kind of work is a bad one. I probably ought to start off each post with a disclaimer linking to those articles just so that my personal development context is explicit.

    I work for a small company that needs internal customization of our business processes. Few of my projects last longer than a week, but I also end up revisiting projects over and over. In this environment an ability to refactor ruthlessly is king. Unit testing is a good skill to develop as well, but mainly for things that are tricky or reused in a lot of places.

    Not that I haven't been in (slightly) larger development shops. For example, I was a development manager in a larger in-house shop (~20 developers) that is roughly analogous to Paul's bank experience. While there, we got unit testing (among other things) going and found good results with it. Building a unit test around a bug before fixing it was an excellent technique for getting unit testing built up around prior projects, for example. At any rate, I've experienced a broad sampling of programming environments and been in charge in enough of them to have confidence in my opinions.

    So while I certainly admit that I'm no more immune to mistakes than the next guy, I have also experienced the beauty of tight iterative cycles and their corrective effects on development projects. Tight cycles and solid feedback trump pretty much every other methodology I've ever tried. The thing is, where you test in each cycle is immaterial to the results I've experienced. In a tight cycle, testing first doesn't really carry much design benefit because any coupling weaknesses you have are revealed just as quickly.

  19. Jacob

    Posted on: 2/26/2008 4:47 PM

     # re: TDD Considered Harmful?

    @William: You have some important logical fallacies here.

    First off, blindly mimicing others just because that's what the cool kids are doing was never actually a ticket to success (not even in High School). They are not you. They are not me, either. My software development needs are substantially different from anything Martin Fowler, Bob Martin, or Kent Beck are likely to find themselves doing. While their experiences are good to know and I certainly attempt to learn all that I can from people with the respect of their peers, any extrapolation of their experiences has to be done with care and intelligence.

    Second, just because TDD was instrumental in the discovery and exploration of certain design patterns for a group of developers doesn't mean that TDD is the only (or even the best) way to learn those design patterns. If you could only learn a design pattern by adopting a specific methodology, I'd question the utility of that design pattern. It'd be like only being able to learn math in a schoolroom with blackboards in it.

    Third, ad hominem attacks aren't as effective as you seem to think that they are. Those petty little jabs end up hurting your argument more than any counter-argument I could come up with. You might get encouragement from people who already agree with you, but those wavering even a little bit will find it off-putting.

  20. andrew

    Posted on: 3/6/2008 7:37 PM

     # re: TDD Considered Harmful?

    I went the "the sound of one man testing" site, and was rather amused that there were grammatical errors in the text... words that were spelled right, but used incorrectly.... which to me highlights the problem with trusting testing tools (spelling/grammar checkers) too much - it can give you a false sense of confidence.... then again, so can not testing... I wrote a routine, the level of confidence was high - until I wrote a test script (after the development) and found a couple of cases where there was an issue... then I wasn't so confident and wrote some more tests... I guess the question is, would I have known to write those tests before development? would I have written enough test? could I have given myself just as much confidence by writing one or two tests that it passed? What if you get lazy writing the tests and don't cover the full scope? Personally I figure it's always better to get someone else to do the testing - someone with an incentive to break it. You just can't rely on people to do their own testing... If you can't trust someone to code something correctly, then how can you trust them to test correctly (completely) either? Anyway, yes some testing is better than no testing... at least the words were spelled correctly!

  21. Jacob

    Posted on: 3/7/2008 5:11 PM

     # re: TDD Considered Harmful?

    @andrew: Yes, some blog writers misspell or misuse words slightly. We have enough non-native English speakers in the industry, though, that I try to look past that to the ideas they're trying to express. I'm not sure about Derik Whittaker's background, but his ideas tend to be clear enough, even if his English usage is sometimes technically off a bit. I disagree with his one post, but his points are clear and interesting.

    As for getting someone else to test, yes, that's definitely valuable. I've made the case for in-house QA before and expanded it with how to convince a business to pull the trigger. So. QA == good.

    That said, I think unit tests are best written by the developer who wrote (or will write) the code. It'd be interesting to expand on this, but it comes down to two main reasons (off the top of my head): a) the cost of understanding the problem domain in enough detail to write the code would have to be repeated to write the tests and b) there's a synergy bonus to both testing and coding if they're done closely enough together (as the developer jumps from one to the other for some final refactoring).

  22. Jeff L.

    Posted on: 5/5/2008 8:27 PM

     # re: TDD Considered Harmful?

    I've done TDD for about 9 years now on many different sorts of applications. I have done "test-after" (or "POUT") many times since, as well. It's simply not as effective, nor as enjoyable.

    POUT is a lower barrier to entry, but most of the benefits of TDD do not exist, primarily the drive toward lower coupling and high cohesion. I'd say maybe 20% of the benefits accrue, so little as to suggest that efforts are best spent elsewhere.

    Sure, a good developer will be able to produce a pretty good design using POUT. However, the majority of developers aren't that good, and thus they suffer a similar quandary--the barrier to entry for *really* understanding how to produce a good design is probably similar to that for TDD.

    One other thing that's rarely mentioned is what happens to the tests over time. In a TDD environment, they are central to what the team does, and they stay "live." In contrast, where TDD is sporadically practiced, or POUT is the norm, many of the tests fall into disrepair over time, and often get turned off. Even if they ever had value, the ROI for those tests goes to 0, suggesting that they were most likely a waste of time (sure, they told us that the code as shipped was probably working, but the reason such tests go into disrepair is that the system is changing, which is when the defects start occurring).

    If POUT makes you and your team feel better, then by all means go for it. Same for the TDD proponents.

    Perhaps neither choice is wrong, but I've seen results from very good TDD teams (i.e. places where the developers "got it") and from very good non-TDD teams. The non-TDD code was what most people would consider good, but the TDD code on the *good* TDD teams (which are admittedly rare) dramatically outshone the other systems. My choice is made. Make your own.

Your comment:



 (will not be displayed)


  Please add 8 and 4 and type the answer here: