The Rabid Paladin

For those of you who are tired of City of Heroes posts and who haven't been tracking my technical blog, a bunch of posts there lately have been more thoughtful than technical. You might want to check them out.

• Is Microsoft Evil?
• Undercutting Your Own Argument
• On Being a Microsoft Shill

Don't those titles alone make you want to check them out?

One of the cardinal rules of creating a blog you want people to read is to keep it well-focused. That generally means keeping to a single topic, at least mostly.

Personally, I'm too ADD to only post on a single topic, so I've been struggling. I've wanted to post mainly on technical and professional topics and want that to be my main audience. On the other hand, having 10 siblings and being married to a woman with 8 means that blog posting is a really easy way to keep them all informed about what's going on with our family and things I want to share.

Till now, I've concentrated mostly on the technical stuff here (with a very modest amount of success). One or two of my articles have even survived past the first day on Joel Spolsky's custom sub-Reddit—a fact of which I am inordinately proud.

Keeping this focus has been difficult. Members of my family, timid souls every one, have even been known to complain, however gently, that they like my blog and all, but what's with all the complicated tech stuff they don't understand?

So I've decided to make a change. I'm going to split to two different blogs. Since the name Rabid Paladin is a better fit for my personal and political junk, I'm going to leave that stuff at this address. All of my technical posts will move to a new site: Scruffy-Looking Cat Herder. Yeah, the name is odd, but then, you probably shouldn't have expected any different.

I've already moved all the old technical posts over, so those of you who are here for my developer insights, do yourselves a Redirect 301 over yonder. You're welcome to stick around here, of course, but all the juicy (complicated or not) technical posts will be over there.

For the rest of you, stick around. I'll post more personal stuff more often here, so this should be a good move for all interested.

Developers hate it when someone changes the requirements in the middle of a development project. What few have realized yet is that they've gone and changed the requirements to be a developer right in the middle of our careers.

An Unnecessary, But Illustrative, Story

The summer of 1994 found me in Mesa, Arizona, new-minted diploma in hand, getting ready for my first real job. A couple days before I was to report for the new position, I received news that the job that I had pulled my wife and baby daughter into the gods-forsaken desert for had, well, been eliminated. Not a happy piece of news to receive. Since my parents lived there in Mesa we ended up taking refuge in the familial homestead. I love my parents, don't get me wrong, but this was not a celebrated event—by any of us.

This was apparently all the impetus that my dad needed to leave the largish firm he worked for as the main estate attorney. He asked me to help him automate the documents he used to create trusts and wills. For money. This was the start of my transition from an English B.A. to a computer programmer. Who knew you could make money with computers?!?

The Point

My dad found that having much of the drudge work in creating those documents automated allowed him to concentrate on larger aspects of estate law. He was able to grapple with the greater context and put things together in a way he hadn't been able to before. This change in focus helped both his practice and his clients because he was able to provide better service at a cheaper cost and still make more money himself.

So here's what this has to do with software development: advances in tools, design patterns, and technologies have created the same shift for software developers that my dad realized as an attorney. While the occasional Luddite might quail at the thought of leaving vi for an IDE with built-in debugger support, most developers are happy to move away from the drudge work that has preoccupied us for so long (as with any boost in efficiency the expected new leisure has yet to show up).

In fact, what we find now is that developers are expected to take on more and more responsibility. This is a good thing overall because increased responsibility == increased value (and hence, generally, salary). This can be a bad thing, however, if software developers neglect some of the new skills that we need in order to handle these responsibilities well.

This is what lies at the heart of recent calls for developers to learn "soft" skills and develop better communication. That's because, unless you work in some back-water in the benighted hinterlands, the days of the cave-troll coder are over. A developer can no longer count on being isolated from users, managers, or <shudder> marketing weasels.

What Everybody Seems to be Missing

All of this is well and good and recognized by those who are paying attention. But it's not enough.

I like to compare business software development with practicing business law because they are both logical disciplines that have to support the human decision-making processes that are inherent in running a business. There's one large difference between the two that I think is important—lawyers are taught the fundamentals of business as a part of their law degree. They don't have an MBA or anything, but they do attend classes explaining typical business structures and basic accounting principles. That's because doing so helps them both with their interface to the business and in their ability to deliver work that properly supports the companies they work for or with.

Well, software developers are increasingly in a similar relationship with the companies they work for. I'm fortunate in my English degree because I'm reasonably well set up in the whole soft/communication skills area. What I'm finding lately, though, is that the courses I took in Business, Economics and Accounting (I never did figure out what I wanted to major in) are becoming crucial to my ability to do my job as a software developer. Understanding double-book accounting and GAAP help me understand the needs of the finance guys. Understanding supply-chain and basic inventory control helps me understand the operations folks.

I'm not talking here about becoming an expert in business. What I'm saying is that developers need to take the time to make sure that they understand the reasons behind the weird and wacky things that other business units are doing (and requesting that we automate).

And it wouldn't hurt to take a formal class or two.

Too often, the software developer going over the current supply-chain configuration with an operations manager will jump to conclusions, or worse, offer "helpful" suggestions about how things can be done more logically. More often than not, we come across as isolated from the realities that the current configuration was created to overcome. Or worse, we come across as judgmental jerks.

It's a drag having to learn so much that was originally considered outside of our discipline. Unless you're a hardware or other highly-specialized developer, however, it's going to turn out to be the kind of thing that sets the competent apart from the competence-challenged.

The folks at The Golden Compass (advertising the upcoming movie) have an ingenious twist on the personality quiz meme. Obviously, I couldn't resist. The twist is simple: after you answer, let's give your friends 12 days to modify it. Well, here's your chance. My answers have me matched up with Hypatia, a badger. I've no idea what I'll end up with...

Steve Harman had a post back at the beginning of the month about stuff you'd tell a young developer. It's a reaction to a similar piece by Jeremy Allison. It's an interesting topic, so I thought I'd waste a few pixels on it myself.

If it's not what you love, don't do it

I wouldn't generalize this to other fields, but for software development, I think this is a good thing to keep in mind. A lot has been made in the past by career counselors and other gurus about "finding your bliss" or similar nonsense. I think that's mostly a crock. You can be a good doctor, mechanic, or professor without liking what you do. Sure, those who love what they do will tend to excel at it and rise to the top of their field, but you have to ask yourself how much better off people at the top of a field are vs. those who are merely good enough.

That said, I think software development is somewhat unique as a career. Software development advances incredibly fast and keeping up with new technologies and ways of doing things takes a non-trivial amount of effort. You have to paddle hard just to stay even and that means experimenting and learning on your own. You could probably do fine if you have Arnold Schwarzenegger level drive, but short of that, loving what you do is about the only way to get you where you need to be.

Reputation is important

People who have been around the block know that the difference between developers can be extreme. Top developers are 5 to 20 times more productive than their peers. Given some of the complete incompetents I've known, I suspect that there's really no upper limit on that number.

Unfortunately, both Jeremy and Steve use this point to jump into the virtues of Open Source software. Open source contributions can enhance your reputation, but that has nothing to do with people being able to look at your code—it's more about working with other developers who can then form their own opinions of you as a developer. That's helpful, but limited.

A good reputation is someone talking about how well you did a job for them. It's your name coming up when someone asks "do you know any good developers?" A good reputation is more important than mere code sample availability and the wider your reputation can penetrate the better off you are.

How important is reputation? Let me put it this way: I haven't gotten a job from anyone who didn't know somebody I'd worked with since 1997. Your reputation and your skills are your only true security in this field.

Never stop asking why

The vast majority of people in any field are content to learn what they have to do to get the job done. Top-tier people are the ones who are asking pesky, even impertinent, questions all the time. Jeremy puts this under the heading of "Learn the architecture of the machine" but that's too limiting. Learn as much as you possibly can about why and how things come together to get the job done.

There's two parts to this, really. The first is that you don't stop just because you got something working. Find out why iterating backwards through an array fixed your bug. Look for the best event to use for control loading rather than just the first one you found that happens to work. Learning how to parse technical documents, navigate help files, and put a concise Google query together are key skills you'll need to master.

The second part of this is harder: if you don't know something people are talking about, ask. This becomes harder to do over time because you start believing your own hype. It's easy to believe that inner voice that's saying you can bluff your way through this conversation and look up anything important later. The thing is, the really bright people, the ones you most want to impress, aren't going to be fooled. I've also found that top technologists enjoy getting intelligent questions almost as much as they love answering them. Note the qualifier there and spend the mental effort to make your questions intelligent, though.

Your employer is more important than the community

This goes directly counter to both Jeremy and Steve and I'm going to tank any chance I have with digg or reddit, but it needs to be said.

Jeremy looks like he's coming from that staunch anti-corporation Slashdot tradition that talks in moral terms about the beauty of open source software and the depravity of Microsoft. Reality check: it is not a common goal of open source developers to create "the greatest, most beautiful work of art that all of you can create together." That's certainly not why I contribute to open source projects.

Open source developers are no more virtuous than corporate developers. While you may program for fun and the satisfaction of a job well done, the choice of project you work on is inherently a selfish one. Always. Whether you program for a paycheck, community recognition, or because you need working blog software that you can tweak when you want to your choice is based on internal needs, not virtues.

If they're both selfish, shouldn't I be saying that they're equally important? I don't think so and it's not just because a company will help support your family. The reason your company is more important is that your company has to compete in the market place. It has no choice. Because a company has to charge money for its products or services, the company is forced to provide things that other people want badly enough to part with their hard-earned cash. In practical terms, it means that a company has a reality check that can't be ignored because sooner or later they're going to have to go out there and prove themselves in the marketplace. While the tech bubble bursting was tremendously painful when it happened, imagine the crap pile we would have if it didn't burst. Ever.

In addition, companies force you, the developer, to do work that you don't want to do. Developers hate that, which is perfectly understandable. Being forced to work with unreasonable managers, clueless users, and hopelessly disconnected marketing weenies is a drag. If you're good, though, it's also a rich source of opportunities to tackle things that you might never have thought needed tackling. And it forces you to justify your dogma of choice to a group of supremely skeptical, self-confident people who have the welfare of their families on the line. Yeah, that's not fun, but it means that you'll have to learn to understand and articulate things you just know are true. With the added bonus that you'll find that they sometimes aren't so true after all.

Finally, be wary of advice from old timers

Not me, of course. My advice is nigh infallible. Just ask me, I'll tell you. If you go awry following my advice, it's probably because of errors in implementation.

Okay, maybe not. I've blogged about recent errors I've made, so I have to assume there will be others. Be careful about who you listen to. Be aware that everybody has their own hobby horses and pet peeves and it's a human trait to cram those prejudices into everything (particularly advice). Also be aware that internet communities trend towards consensus as much as any other human community does. Consensus is dangerous and will tend to mask assumptions and pitfalls. Develop a well-honed BS filter and use it liberally.

While you're at it, the sooner you learn to pass your own ideas and theories through a well-developed skepticism the better off you'll be.

Have you ever found yourself in a situation where you know what's the best thing to do, but are unable to convince anyone else that you are right? Developers know that even simple problems have more than one solution. Developers who have worked on a team of more than one have probably been in a situation where they just knew that the team was heading in the wrong direction and that they had a solution that was more elegant, easier to program, and better to maintain.

Higher profile developers often find themselves trying to explain their solutions to non-technical people as well; sometimes before development has begun, but sometimes after it is truly too late to do anything different.

So what do you do when you have to butt heads with buttheads? Here is my secret to winning in these situations

Geek Whisperer

As a developer and/or manager of developers, it is astonishingly easy to find yourself trying to explain concepts to people who, however smart, just aren't equipped to understand what the heck you are talking about. This is an uncomfortable position for all involved.

It is tempting in these situations to pull out a call to authority and tell people that you are the subject matter expert and they simply have to trust your expertise. This works fine for the little stuff, but as the scope increases, so too does the reluctance to put up with this answer.

And with good reason. Technologists can find themselves literally with the fate of the company in their hands. Business leaders find that hard to take and are going to be extremely wary of any solution that involves the fate of their company.

How to Win With Business Folk

There is really only one way to handle this situation: concentrate on what is best for the company.

Everything you say and do in this situation should start and end with what is best for the company. The magic of this orientation is that you can defeat every argument and diffuse even personal attacks by leading back to what's best for the company. No matter how irrational the objection, how confusing the question, or how deliberate the ignorance, you'll lower tension and create space for a solution if you can turn back to what is best for the company. Even asking the question, "What course of action would be best for the company?" can be sufficient to redirect an out-of-control or confrontational situation.

Talk Amongst Yourselves

Developers tend to believe that they live in a rational universe—or at least that their work is based on the application of structured reason to provide solutions to defined problems. This belief is shared, more or less, among all developers—even those who believe themselves to be the only developer who is actually rational.

This makes it both easier and harder to solve disagreements between developers. The fundamental belief in reason tends to keep disagreements from being personal, so you'll find most developers are willing to discuss alternatives and the ramifications of design decisions.

Every now and then, however, it is possible to find yourself at an impasse with another developer (or group of developers). In an impasse, developers can become more intractable than business folk if only because you lose the ability to appeal to consensus. Rational is rational just as right is right. The best rational choice remains the best rational choice no matter how many delusional fools find themselves unable to comprehend it.

How to Win With Technical Folk

There is really only one way to deal with this situation: concentrate on what is best for the company.

When two (or more) sides have presented what they feel is the most rational solution to a problem and remain at an impasse, the source of contention almost inevitably lies in unstated assumptions. In this situation, asking the question, "Why is this solution better for the company?" helps to ferret out those assumptions. Some disagreements will need to apply this question repeatedly as it is not uncommon for assumptions to be stacked on other assumptions. Ferreting out the source of your assumptions can be tiring, but in the end well worth the effort.

Open Sores

But what if you aren't developing as a part of a company? The principle remains the same as does the effect. Every open source project has a problem that it is trying to solve and people working on that project will tend to be there because they believe they can help solve that problem. Leading disagreements back to the reason of a project's existence can help straighten out priorities and direct solutions back to the original problem domain (and away from the individuals involved in the discussion).

Winning Everywhere!

Is there application for this question outside of geekdom? Darn skippy there is. In fact, asking "what is best for the company?" is a technique I learned at a leadership conference when I was more manager than developer. It turns out that any time people group up, they'll tend to have a reason for doing so. In fact, "tend to" == "always" in this case. The reason can be explicit or implicit, but people simply don't group up for no reason. Learning to find that reason and drive disagreements back to it helps keep you focused and gives you the opportunity to pick your battles.

A Warning

Asking what's best for your company/group/whatever will often show you the flaws in your own arguments, particularly as you first start applying the question. If you want to win for personal egotistical reasons then this technique isn't going to do you much good.

In fact, ego can destroy the benefits of this technique more generally. Depending on how much power the person with the ego has in the organization, their presence can significantly alter the ability of this technique to work. An example: I once worked for a company where the owners truly did not care what was best for the company. As such, arguing from the standpoint of what was best for the company had no effect whatsoever.

Fortunately, reality has a way of asserting itself in these situations. Companies that can't act for their own best interest tend to disappear, members who don't care about what's best for the group tend to leave the group, and employees who don't care about the best interest of their company tend to find themselves parted from the company.

Still, there can be significant discomfort in the meantime and heaven knows that "tends to" != "will" in this case. You'll have to decide how you approach a situation where ego is going to be a problem. It's almost always best to separate yourself from the ego getting in the way—particularly if it is your own.

Good friend and Indie game developer Jay Barnson has just taken game development in a new direction: Developing in Public. He sounds a little nervous about it, which makes sense. Unlike those who have previously attempted this feat, though, I think Jay stands a good chance of pulling it off, and with good style.

There are two things that are likely to make this interesting. First, Jay's about ready to release his second indie game, Apocalypse Cow—So we're likely to see this through all the way to a completed game. Second, he's an honest and engaging writer.

It's that second that makes me eager to eavesdrop on the development of his new game. I know he'll present it warts and all and that it'll be entertaining along the way.

If you have the eye of a good copy editor, you might have noticed some volatility in the ole Tag Cloud on the right. I made some changes at the request of people on the SubText developers list and reworked some stuff—most of which is completely invisible. The biggest visible change is that I decided that assuming an even dispersion around the mean might work in natural statistics, blog post tags tend to be more of a declining curve however.

What that means is that most algorithms for displaying tag clouds use a formula that allocates about half their categories to styles that are never visible. Most tag clouds I saw had slots for up to seven different visual styles that only really displayed three or four.

Well, that's just a waste of a good idea, I say! So I mucked with the weighting formula so that I'd see more visual variation in my clouds. We'll have to see if my theory holds after scrutiny and on blogs with larger traffic.

I just completed some SubText hacking and the results should be visible now. In addition to upgrading to version 1.9.5 (which should officially release here shortly), I implemented a new feature: Tag Clouds. I'd had enough of my feature envy from all the cool kids who had them, so I went and rolled my own. If you're at my actual site (as opposed to a feed reader), the Tag Cloud is off on the right.

This was a non-trivial feature to add. In an email conversation on the SubText developers' list a couple months ago, Phil Haack (semi-benevolent project dictator) indicated that he planned on tags being first-order objects—meaning that they'd have their own table and post cross-reference. They'd also be allowed to penetrate into other public interfaces (as they'd pretty much have to do). The main point from the majority of developers in that discussion was that they wanted Tags to be done well and fully and support all the different tag providers out there.

Like RSS and other Internet wonders, tags are fundamentally simple in concept. A tag is defined as simply any hyper link with a "rel" attribute of "tag". Something like this:

The biggest gotcha in this setup is that officially, the "name" of the tag is determined by the link and not by the text of the link. So if your link were like this:

 The tag is officially "competence" even though it displays "incompetence".

Which means that the core of the feature is scanning posts on insert and update to catch any tags they might contain and adding/updating the links needed for the tags in the post. All I can say is thank heavens for RegEx. It's a pain in the pinky toe to understand, learn, or debug, but you just can't beat it for parsing text. For the curious, and to open myself to ridicule, I'll give you the expressions I used (both case insensitive).

To find a link: <a(?<element>.*?href=[\"'](?<url>.*?)[\"'].*?)>.*?</a>

To figure if the link is a tag: rel=[\"']tag[\"']

Oh, be careful there if you use or copy those expressions; this was done in C# so quote characters are escaped using the \ character.

Anyway, it's done now, and I'd appreciate hearing any trouble you might have. I already caught a choke when the tag includes a "." in it. Fortunately that one just took a RegEx change in web.config so the tag display handler would pick it up. I'll create another post later detailing how to add a tag cloud to a random skin once this upgrade gets added to the SubText project officially.

Like many businesses, my current employer uses Microsoft Dynamics GP for our main  accounting/inventory software. From a recent trip to Connections (the big convention for Dynamics), I learned that there were some cool data mining utilities available from Microsoft that we might be able to use. I love free stuff, and the opportunity to give our executives something to chew on for a bit (keeping them out of my hair) was one I couldn't pass up.

The Analysis Services install I managed to dig up (after much searching on Microsoft's Customer Source) went pretty easily. There are a couple of SSIS jobs included that populate an summary database with relevant data pulled from the objects you want to throw into your data warehouse. Not too shabby, even if it's an extra aggregating step. In fact, I kind of appreciate the added encapsulation and I know I'll have to restrain myself from using this database for other reports.

Anyway, the only thing left at that point was to "process" the data warehouse. That's where the data warehouse reads the "factual" information and chunks it up into nice little cubes that should have precise business-person meaning. It can take a while, so I was prepared to leave the process running a bit.

Five hours later...

We're just not so big a company that this thing should have taken five+ hours. Digging into the process that was taking so long revealed that at least one of the dimensions it was trying to process included a table join that seemed a bit fishy. It included a master-detail relationship that should have ended up with something around 500,000 rows. Checking the "Estimated Execution Plan" on the generated select statement showed that it was estimated to return closer to 2 billion rows. That's a messed-up join is what that is.

Fortunately, Visual Studio 2005 gives you some cool tools that work with Analysis Services. One of them is a wizard that you can point at a data warehouse and have it reconstruct it as a project. Checking the Data Source View "GPDataWarehouse.dsv" revealed what the problem was. Roughly half the relations in that view had split the key so that there was one for the actual relation minus the CompanyId and another with the CompanyId all by its lonesome. The effect of this is that the data warehouse would process once for the real relationship and again for the spare. Since CompanyId is the same value for all my records, what I ended up with was a data set getting ready to relate 147,698 records with every one of 11,937 records. I'll spare you digging out a calculator; that's 1,763,071,026 rows total.

Now frankly, I could have simply removed the CompanyId entirely from all relationships—CompanyId is only really relevant if your Dynamics install is trying to manage multiple companies. We aren't doing that. Still, I can be a little, uh, rigid with stuff like that so I went ahead and changed the relevant relationships to include CompanyId and simply deleted the spares. Once I deployed the project back to the data warehouse, processing it completed in a mere 15 minutes.

I suspect that the cause of this disjointed relationship is that the original warehouse was configured for SQL Server 2000 and we're using SQL Server 2005. At any rate, things are a lot happier now. I have an SSIS package that kicks off the summary database update and then processes the cube and the whole thing runs at night in 20 minutes or so.

For those who might be experiencing similar issues, I've uploaded my working Data Source View. You should be able to replace yours using SQL Server Management Studio. I selected all the options on install so it includes the Financials, Inventory, Payables, Purchases, Receivables, and Sales cubes. I'd do a backup before attempting it, but you were probably going to do that anyway...