A couple of months ago, I mentioned a project I had coming up that might benefit from using Dependency Injection. The use-case on this is simple enough. We receive all of our EDI text files to a specific directory. Those files need to be processed into standard, internal temporary tables. Since each of our vendors uses the EDIFACT X12 fields differently, we need to customize parsing the files according to vendor.

This is actually one of the larger projects I undertake at our small in-house development shop so it needs more up-front architecture than most and it seemed like a good time to test my understanding of Dependency Injection and maybe develop some familiarity with a DI framework. For one, I know in advance that I’m going to want an interface for my EDI processors and I know for certain sure that I’m going to add file processors in the future.

Those of you familiar with DI might be able to see the ripples around the reefs I eventually cracked up on from here.

The Basic Solution

The solution isn’t that complex in theory.

• I’ll want a service that monitors a given directory for new files with a given file mask. Both the directory and file mask are in a config file that reloads when altered (all handled by built-in .Net processes).
• I’ll want a simple interface that the service can share with implementing classes.
• I’ll want classes to process the files that are found.

See, simple.

Failure

As much as this looked like a DI shoo-in at the time, it turns out to be a bad fit. I figured this out after digging through the Castle Windsor documentation trying to find a way to iterate through all the registered objects that implement a specific interface. I was frustrated at not being able to find one. It took me a while to realize that I was trying to bend the framework to do something it hadn’t been intended to do—invoke methods on arbitrary registered objects. A DI framework is supposed to handle injecting a class that implements a given interface into the class that depends on that interface.

Indeed, DI frameworks in general are meant to abstract and centralize the complexities of implementation object selection. As such, they are built around the assumption that the run-time already knows what class it wants to use for a given interface (either in a configuration file or determined programmatically at run-time).

It took me a while to figure out that my use of the pattern itself is at fault in this mismatch. I’m not injecting a dependency at all. My service isn’t dependent on classes that implement IEDIProcess, it just determines if there are any available that apply to a newly created file and invoke it correctly if there are. In other words, I’ve already architected around the conceptual dependency here and Dependency Injection is an additional complexity with no purpose.

If DI should be used anywhere in this solution, it’s in the processes themselves to separate out the data updates. And, as I’ve mentioned before, there’s no payoff for me in worrying about my data dependencies.

Busted

So it turns out that shoe-horning a design pattern just to play with it didn’t work out too well. Who’d have thought that could happen?

The broader take-away, at least for me, is that you look for a solution/pattern/tool based on the attributes of your problem. In other words the problem comes first and the solution flows from it. Having a pattern looking for a problem to solve is like having a hammer looking for something to fix—both methods are hit or miss.

I’ve been putting off a follow-up on Dependency Injection for a couple of months now. The amount of heat I anticipate receiving is so disproportional to the probable light gained that it makes me hesitate. This weekend, I picked up on a stream of referrals from a post at InfoQ that mentions my Dependency Injection post (though not the follow-ups). It does a reasonable job of spelling out the conversation that happened, though I was feeling picked on until it brought in Eli Lopian’s contribution to the discussion. In whole, it’s a good summary. The real pile-on happens in the comments and it illustrates so much of what I dislike in software development sloganeering that it has jolted me out of my reluctance to respond (I’d have responded there, but you have to register to comment and I hate that).

There are two things that contribute to the silver-bullet, band-wagon boosterism that sends up red flags for me.

Good for What Ails You

This is characterized best by a comment by Steven Devijver when he says

It’s an interesting discussion. Just as interesting as asking oneself: shall I wear underwear or shall I try something really nifty today?

It’s the same kind of thing pointed out by Jay Kimble about a month ago as he expressed his concern with Dependency Injection.

I have been asked "Why don’t you like this pattern or that pattern? I mean [Some famous blogger] says s/he always uses them.  I think they are smart or at least they seem to be.  You must be a 'Mort' or a pretender..."  At least that’s the way it has always been stated to me.

The claim made by these individuals is that The Pattern (it can be any pattern, but this is increasingly frequent when referring to Dependency Injection) is universally applicable and should be used in all cases, preferably by default. I’m sorry, but this line of argument only shows the inexperience or narrow focus of those making the claim.

Claims of universal applicability for any pattern or development principle are always wrong.

Study any pattern or development principle in enough depth and you’ll find the edge cases and counter-indicators applicable to it. This is true of even basic principles like data normalization and object oriented design.

Variations on the universal applicability include the pseudo cost-benefit comments like that from Gabriel Lozano-Moran.

If you anticipate [no] potential future problems using a DI framework like Spring.NET and it takes you a couple of minutes to implement this hardly violates the YAGNI principle.

I call this "pseudo cost-benefit" because it seriously under-values initial implementation, training, and the burden of long-term skill set dependency created by implementing so invasive a pattern—a point I made at the end of my post about small company line-of-business software development a bit back. Yeah, you can probably add Spring.Net to a given project in just a couple of minutes, but that treats Spring.Net familiarity as a sunk cost (or worse, null cost) and commits that project indelibly to Spring.Net familiarity. You may feel that familiarity with Spring.Net is a given for any competent developer, but that proves my point that this is merely a variation of treating DI as a given for all projects in all situations.

Save Me From Myself

This argument is made in a comment by Ole Friis.

When I use DI, I see it as an advantage that I am forced into splitting up my application in well-defined services with a nice interface.

The term "forced" here indicates that he doesn’t trust himself to split his applications into "well-defined services with a nice interface" on his own. In this, Ole Friis is echoing the response I got from the intimidating Oren Eini when I suggested that TypeMock makes arguments that couple DI with unit testing go away.

The main weakness of Type Mock is its power, it allow me to take shortcuts that I don’t want to take, I want to get a system with low coupling and high cohesion.

The implication is that he wouldn’t be developer enough to create a system with low coupling and high cohesion if he used powerful tools like TypeMock. Oren is inconsistent here because he’s otherwise all for developer empowerment as illustrated by his support for Jeffrey Palermo’s excellent post regarding treating developers as professionals.

The fact is that nobody is really in favor of limiting themselves. Taken seriously, these kinds of statements indicate a profound lack of trust in their own competence. I’ve never yet met a software developer above Jr. level with that particular insecurity. I’ve met a veritable host who lack trust in the competence of people they work with, however, which is what I suspect is really behind these statements. Either way, it’s hard to take that argument seriously.

I’m the Anti-DI

Or so I’ve been painted. The fact is that I’m not, though my use for it is pretty limited. Since I don’t need it as a foundation for mocking objects in unit testing, it’s only really useful when I have a service with more than one implementation and I’ve found that even then DI isn’t necessarily the answer. I’ll deal with at least two reasons behind this in a future post(s).

I’ve been musing about software architecture lately and trying to come up with a framework to help choose when to go with more as opposed to less—something that’ll help me feel less arbitrary in my choices. I mean, software design is something of a dark art, but how much of that is inherent and how much is simply being too lazy to formulate good internal guidelines?

My latest ruminations have revolved specifically around Inversion of Control in general and Dependency Injection in specific. Here’s the thing: for the development I do right now at a small reading glasses company, I’m reluctant to implement any of the stronger separation of concerns architectural tools like IoC and a DI framework. The DI zealots would draw from that the conclusion that I am a sub-standard designer and a compromised developer—at least, judging from explicit claims that every project should start out with formal DI (please note both the "every" and the "should").

Pain is an Efficient Teacher

Perhaps too efficient. Most developers have had the experience of working on a project where decisions made in the expedience of the moment came back to hurt them. In a recent DotNetRocks episode, Scott Cate gives an example of hard-coding to a specific search engine and coming to find that they need to generalize in order to allow alternatives. He wishes that they had used a DI framework from the start (in this case, Castle Windsor) to make that an easy change. Scott tells about how they were lucky this need was discovered in time for a major release upgrade so they could take the time to go the full DI route.

There’s two problems with this example when translating to my own situation. First, Scott’s is a commercial product that they sell to large clients and thus the scope is far beyond anything I work on here. Second, I wonder how much pain was really involved in making the change.

You see, pain magnifies itself in memory. You consider all the might-have-beens and how much the problem could penetrate even as you work to ameliorate it. And you consider things you can do in the future to avoid this pain—both the immediate pain and the imagined pain of all those might-have-beens.

The YAGNI Nazi

This tendency to artificially inflate potential scope is the driving force behind YAGNI. YAGNI is the reminder that you can go too far in countering the mere possibility of future pain and that you can plan too far in advance of actual need. I’m a huge fan of YAGNI simply because it has allowed me to realize significant time savings and produce useful software for clients quickly and efficiently.

In many software design discussions, however, I feel like a lone YAGNI voice in an over-designed wilderness. I’m accused of negligence or stupidity (though typically through implication as opposed to outright accusation) because I look for the justification of larger design patterns before blindly implementing them. People make the unearned assumption that understanding a given principle must mean that I would adopt it. This comes through loud and clear in the discussions with DI proponents. It seems entirely foreign to them that I could both understand it and decide to give it a skip on some of my projects.

Sometimes this judgement comes in the form of a logic chain. This happens when designers use the strengths of one principle to piggy-back in a second. Unit testing is most often the piggy-back victim here. That’s because the benefits of unit testing are well-understood and apparent in projects of even minimal scope. Scott Cate admits in that DotNetRocks episode, for example, that a large part of his current push for the Model View Presenter pattern as well as DI is to facilitate unit testing (when he also took pains to point out that implementing Castle Windsor is not without cost he went on my list of DI heroes).

The problem with a logic chain, however, is that you’re right back where you started if a link breaks. What if I can achieve strong unit testing without IoC and DI?

So here is my YAGNI wisdom (for what it is worth): you don’t protect yourself from future change by throwing every possible technique or pattern at every project. You protect yourself from future change by refactoring ruthlessly to keep your code as change-ready as possible.

The Bottom Line on Dependency Injection

So here’s where I think I’ll come down as a rule on implementing DI: I’ll use it as soon as I actually need to support multiple services in a given object. Since I don’t need DI in order to mock dependencies in unit testing (yay TypeMock), this means that I’ll look to implement DI when I have to support multiple services in a given object in production. I have two examples of related projects here that will serve as useful test cases.

The first is a need to integrate customer orders into Dynamics GP. We have many customers who send us orders in a variety of ways and with a variety of processes for approving and tweaking them before we let them hit our system. It’d be nice to have a single module integrated into Dynamics GP that can accept orders in the variety of formats we end up with and go to town. I’ve already got this project up and running for a single customer using a generalized interface and I think DI can help extend that further faster than I can without it.

The second is the initial processing when we receive EDI documents from our clients. Since EDI X12 isn’t really a standard so much as a soup of possible ways to do things, each customer tends to put their own unique spin on what fields are significant and how they should be interpreted. I’ve hated our current EDI software for months (and not just because it uses FoxPro at its base) and I’d dearly love to pull that processing into an internal framework I can use and extend on my own.

Both projects will take me a while, which is appropriate because I’ve been saying all along that DI seems like something suited to larger-scoped projects. I’ll post my experiences as I go, assuming I actually get the time to tackle either one.

Both of these examples are also an inversion of typical DI examples because in both I technically have a service that needs to act on multiple objects. That said, I’ve been an architect long enough to understand that while services and objects might differ conceptually (though defining either is an exercise in nit-pickery), it’s trivially easy to turn each into the other (if you bother differentiating them at all—see nit-pickery above). Sometimes it is quite useful to invert them as I plan to do here.

I typically work with small companies who need to customize software to fit their business practices. A lot of companies have critical competitive advantages embedded in the way that they do things and need to ensure that their software doesn’t get in their way. That typically means that I deal with specific vertical markets (either at the vendor or client level) and dance with 500lb. gorillas to make things work the way companies expect them to. It’s business programming in the trenches and can be nasty, brutish, and, well, not short so much as constrained.

The Typical Battlefield

This is the underbelly of software development. Because the companies are small, being hired is a matter of impressing a very small group of non-technical people—often a single person such as the CEO or President. I’ll sometimes have a full development team, but occasionally I get to do it all myself. That degree of variation means that my methods and opinions have to a) scale well and b) not suck (because I’ll often be implementing them myself).

Because being paid depends on non-technical people evaluating technical work, a lot of really bad things can happen down here. This is where the president’s fresh-out-of-college nephew gets to prove he didn’t waste four years of tuition. Or where that new English degree and some aptitude can turn into a technical career. It’s also a place where developers whose self-confidence exceeds their actual ability can avoid public accountability. For a while.

What that means in practical terms is that I work with small companies who have extraordinary needs and the means and motivation to meet them. It also means that I see a lot of train wrecks (some admittedly of my own making). I’ve found that most of these train wrecks have their origins in one of two fundamental problems.

Over Confidence

You know those eager puppies that get so excited with each person who stops to pet them that they wet themselves? I sometimes see developers who are like this each time they discover a new process, technology, or design pattern.

I don’t fault the underlying impulse, really. Excitement with new technology is typical in IT people for the same reason that air is a common addiction. People who hate change tend to be filtered out of IT pretty quickly. To extend the initial puppy analogy, most developers learn to get by with simple tail-wagging when they find a previously unknown technology that solves a common problem.

What leads to train wrecks are those developers who take off in the midst of their excitement and begin implementing new technologies or techniques without bothering to understand them thoroughly. Like the developer I mentioned in a previous post that implemented n-tier design in each project separately, they may have a vague idea of how the thing is done but they end up with something that is worse than if they’d never heard of the Great New Thing™ in the first place.

If you suspect that you are this kind of developer, stop. Take a breath. Dig deeper. Don’t proceed until you can identify at least two drawbacks and can explain why the Great New Thing™ was invented in the first place. If this is a developer on your team, I’ve heard you can get good results with rolled-up newspapers and consistent, immediate feedback.

A Hammer Finding Nails

I have a daughter afflicted with artism. To her, every surface is an opportunity to improve with color and/or texture. You’ll find developers and architects like this as well—treating each project as an opportunity to improve with their favorite pattern, practice, or technology. This is the opposite of the previous problem because these individuals know their stuff, know it well, and have likely used it successfully before. Indeed, the more knowledgeable the designer, the more prone they are to this particular vice.

The problem here isn’t a lack of detailed understanding of the Great New Thing™, but rather lack of focus on the real needs of the project. Here’s the simple truth: in the world of small business development, a short project delivered quickly is usually better than a perfectly-crafted but more complicated project delivered even a little bit later. In short, It’s a YAGNI world.

I have to be careful here because I’m not saying that any given design principle or practice is a bad idea. What I’m saying is that it is really easy to over-architect these projects. The company has so many needs and can improve in so many ways that it is tempting to go into full framework-building mode and spend months putting something comprehensive together when all that was needed is a basic customization or a specific process.

In some places, I’ve had to pull the YAGNI flag out so often that I’ve considered having a stamp made to save time. Maybe I should make a suggestion to Think Geek.

Keep it Lean

Whether coming from a background of large development or yearning to do so, it is really easy to include things just because they are good design or a "best practice" without bothering to review their actual impact on the current project. The problem is that each pattern, technique, and tool not only commits the project itself forever, it also sets up a barrier for future work by creating a developer familiarity dependency that can be a significant hurdle for later modifications (particularly if a developer only thinks they are familiar with the pattern, technique, or tool).

Each technique, pattern, technology, and design principle should be evaluated in light of immediate need and weighed against future scenarios that include the possibility that you wont be the one doing the work. Indeed, I recommend a slight bias in favor of not implementing a Great New Thing™ because of the tendency of really good developers and architects towards heavy designs that can end up hurting small companies in hundreds of little ways that add up to death by a thousand cuts.

The best "lean" skill a programmer can develop is to program with an eye on refactoring. Moving to something better when it becomes clear that it is needed isn’t a lot harder than if you had implemented it in the beginning (and has the added benefit that you know it is needed) if you have kept refactoring in mind throughout. Leave "quick and dirty" programming for the amateurs. Solid refactor-ready code is the best legacy you can create and one that doesn’t age as people get over themselves or discover some new Great New Thing™.

In many companies developer career progression is deceptively straight-forward; Jr. Programmer, Programmer, Sr. Programmer, Team Lead, Architect, Sr. Architect, Bob (Bob being the semi-mythical entity referred to in obscure comments, worshipped by now-extinct aboriginal tribes, and rumored to haunt the sub-sub-basement).

The differentiation between these positions starts off with how much you know. A Sr. Programmer is a Jr. Programmer who knows his tools inside and out and can complete assigned tasks quickly and without a lot of supervision. Around Team Lead time, however, progression stops being about what you know and starts revolving around your ability to choose wisely between competing trade-offs.

This is an easy transition to miss if you aren’t paying attention. Indeed, I’ve known "Architects" that I wouldn’t ask to design a "Hello World" application. Because being a good architect is about developing good judgement, there are two things that are essential to becoming and remaining competent.

Know Your Stuff

Yeah, I know I just said that being an architect isn’t about what you know. That’s only slightly true. What changes between Sr. Programmer and Architect is the depth of your knowledge—you have to penetrate to the reasons and complexities behind patterns, practices, and technologies. You go from knowing what an n-tier architecture is (and how to create and maintain one) to understanding why that architecture was invented, what problems it solves, and what costs are involved in creating and maintaining one.

Take, for example, the "architecture" I found in coming to a new position. A number of projects are used for this intranet application, each in its own application space. Here’s a short illustration of the structure of two of the solutions.

As you might guess, BOL stands for "Business Object Layer", DAL for "Data Access Layer", and OBJ for "Objects". You might not be surprised, given the closely related project names, to learn that each project contains classes that exist in the other. In short, despite having the structure and using the terminology of n-tier architecture, this isn’t actually anything of the kind.

The developer responsible for this design is not an architect, no matter how much experience he may have, because he has failed to understand when and why you would use an n-tier architecture. Understanding the reasons behind n-tier architecture would make it impossible to create this structure.

Understanding architecture takes hard work. It requires more than reading blogs, attending conferences or purchasing books (though each of those are certainly useful in developing understanding). True understanding requires thinking things through and asking tough questions. It requires breaking down concepts until you can identify and understand their component parts. Often, it takes asking the questions that risk confronting a room full of people glaring at you in varying degrees of shock. Most of all, it takes admitting that you might not know something and working to rectify that lack—or, harder still, being willing to re-examine assumptions (or even conclusions) when presented with new information.

The Development Twins: Cost and Benefit

It’s easy to forget that every technology, architecture, and practice has two sides; a cost and a benefit. To borrow from Heinlein, "There ain’t no such thing as a free lunch." Because making choices is hard, it’s easy for people to remember only one side of a given technology, architecture, or practice—to see only the cost or only the benefit.

This is particularly true in new processes or designs. We go through so many silver bullets at least partially because people teaching new processes tend to concentrate on benefits. This is understandable even if the presenter isn’t financially tied to your adoption of the Great New Thing™ because the benefits are the reason the Great New Thing™ exists. This means that you’ll be taught the benefits, but you’ll typically have to discover the costs on your own.

It’s also important to bear in mind that developers have a natural bias towards solving problems, not finding them which leads to a tendency to grasp benefits quickly. Because architects are grown from developers, it’s important to examine and balance this potential optimistic bias. And it is a balance. Ask an average Linux geek about Microsoft if you want to see a bias towards cost.

The Scope Multiplier

A project’s scope has a multiplicative effect on both cost and benefit. Understanding this became explicit for me in a reply I made to a recent comment by Udi Dahan.

I’d be willing to bet that the benefits of DI, SoC and other good design practices increase exponentially with the number of developers while their costs are a linear progression. Huh. That bears some thought...

For any given process, you need to understand how both benefit and cost are affected by the scope of your project. Practices that scale benefits exponentially and costs linearly are excellent when the scope is large. Indeed, the larger the scope, the better the net gain from applying the given practice.

A corollary of that evaluation is that practices designed for large scales are less likely to provide a net benefit in a small team with a small project.

It’s possible that project scope can be calculated separately for project complexity and team size, but I’d have to give that some more thought. Since high complexity projects tend to require larger team sizes, you’d seldom have one without the other (making this a moot conjecture). That said, my initial instinct is that something like n-tier is more important vs. complexity whereas the benefits of SoC scale more based on team size (though it helps mitigate complexity as well). It is possible that this difference may play an important role in how and when you’d apply each.

Time

That pesky fourth dimension can also have a dramatic effect on cost and benefit. A quick down-and-dirty project with limited scope may seem like it wouldn’t benefit from any real architecture at all. That evaluation can change drastically if the project is likely to change a lot over time. Most developers have been ambushed by application maintenance at some point because they hadn’t planned adequately for change.

Predicting the future is a dubious pursuit at best, however, so be careful here. There’s a reason YAGNI became a well-recognized acronym. Don’t forget that a project can be refactored into a useful pattern after the fact if circumstances change to warrant it—particularly if you have prepared for easy refactoring (unit testing, anyone?).

The biggest problem with evaluating the effect of time on your project is that it is literally impossible to measure the impact of the choice not taken. As a result, I’m always very careful when I hear something like "If only we had..." or "Good thing we...". Personal, anecdotal experience is error prone at best—something to keep in mind during project postmortems.

The Courage to Choose

The most important thing an architect does is decide. The architect chooses the technologies, the patterns, and the practices to apply to a given software development problem knowing that success and failure can hang on the effects of those choices. That’s a lot of responsibility. The impulse to push your choice off to a vendor, or some guru’s public statements, or industry consensus is a seductive one that can work for a time. Reality being the harsh b**tch she is, however, it isn’t a safe impulse to indulge. Personally, I prefer to hold my fate in my own hands.

The folks who visit the actual blog (as opposed to using an RSS feed) have probably noticed that I re-skinned the site. Now that I'm more comfortable with SubText's skin structure, I figured this would be the next step. Personally, I think that it looks tons better now, but you can let me know in the comments if I'm just being delusional. I still need to tackle that calandar, though.

I can't take much credit for it if you like it. I'm perfectly aware of my (lack of) design sense. I can generally do okay evaluating existing designs, but I suck very, very much in creating my own. I found a design that I liked on Open Source Web Design and picked it apart enough to apply it to my SubText skin. Sweet that folks'll share like that.