I recently read “The fear of change at submission time” on Tom Hammersley’s Graphics and Games Development Blog. I want to post my response here:

I wholeheartedly agree with the conclusion of the article: more automated testing could only help improve the quality of the software (not just game) industry. In my limited experience, it’s almost shameful how little automated testing is done.

Why some fear is warranted

Next, I need to preface my statements that I’ve never worked in the game market myself and the article is about game development. That said, my experience in software development has been (primarily) with large companies with very old (and subsequently complex) code bases, which with that experience in hand, made me cringe a bit as the article called reluctance to introduce change at submission time as “groupthink” and “stockholm syndrome.” There’s evidence for a lot more legitimacy to fear of late changes in segments of the market I’ve worked in.

My rationale here is primarily due to two factors.

First, at large companies there are (very broadly generalizing) two types of software engineers: those that write code “that works” and those that write code “that appears to work in the cases they’ve tested.” In an ideal world, obviously everyone should be writing the former and any other such code should be rejected, but in practice, there are many mediocre (and below) quality developers when working within a large developer pool. The latter kind of code does end up getting written – and it makes it into the code base. Thus, an aging code base has been built up by both these types of developers and the overall quality of the code reflects that.

The first factor results in the second factor: even the expert developers can’t be expected to know what kind of unfair, incorrect assumptions one piece of old code might be making about another piece of code where a “perfectly reasonable” code fix might be going in. That nasty piece of code that the high quality developer doesn’t know about that makes a foolish assumption is also very likely the kind of code that has not been architected to be easily unit-testable; therefore, unsurprisingly, there are not unit tests to verify the change against. (I won’t even joke about there being documentation.) In other words, this really is very much the equivalent of the kind of code that is only testable when there are “passengers aboard the jet” (to reference the article’s analogy). [E.g. I have in fact encountered application defects in non-UI code that existed only when a command was typed in the console but work just fine when that exact same command is input from a script.] It’s ridiculous, it’s annoying, it’s impractical, and it could drive a good developer insane, but in my experience it does actually exist as horribly derpressing as that is.

Long story short: older, huge code bases developed by a mixed level of developers often results in a lowest-common demoninator stability to the code. It’s only as strong as it’s weakest point. This can be a real problem when a core part of the architecture even gets a minor fix since it’s virtually impossible to know what kind of irrational code might be layered on top of that.

How do you deal with this being the reality?

To alleviate (not fix) the problem, ask the following questions:

1) Is the code change from a high quality developer who understands the critical difference between code that is provably correct (as far as it can be) and code that merely seems to be correct?

2) How much experience does this developer have? Can they be expected to know some of the entirely unreasonable legacy limitations in the code, i.e. things that shouldn’t be expected but unfortunately are in fact there? (E.g. An expert developer might find a 2x speed improvement by merging processing requests, but not know that some ridiculous piece of code higher up – or worse, a third-party plug-in – actually relies on a request count to operate correctly.)

3) Where in the software stack is the change? In other words, how much other code flows through the change – which becomes a multiplier on the risk due to (2). What’s the weakest link in the network of potentially interacting code segments affected by the change?

4) Project schedules needs to allocate time to fix non-user, legacy architectural issues such that cause (2). If it’s so old and complex a problem that no one knows how to properly fix it, it needs to be rewritten from scratch. It can’t just be ignored.

Is the code the problem?

The real problem has little to do with code itself, as far as I can tell. For example, in my experience, (4) rarely ever happens. In the example in (2), if a “correct” fix is made that breaks a profitable third-party plug-in that causes major, unexpected problems for another company, I can assure you the people screaming won’t care one way or another about software development methodologies and the quality of their code – they’ll be caring about the financial impact and be talking to their lawyers to find some way to state it as a breach of contract. Again, that’s not how it should be, but seems to be how it is.

So is management the problem? Or is it the software development process itself? Is a language needed that enforces stronger code modularity and encapsulation? A source control tool that forces unit tests to be submitted and pass for every change? Code reviews that are orders of magnitude more strict? Is there any silver bullet? (If there were and I knew it, I’d probably be a lot richer…)

Regardless, it seems safe to conclude that the fear of change at submission time is a factor of the quality of the overall code base – not necessarily anything to do only with the specific developer or the specific change. Therefore, to remove that fear, the quality of the code base needs to be improved. How do you do that inexpensively with some degree of permanence, at least against regressions of previous problems?

As Tom Hammersley’s article said, improved automated testing sure would be an obvious and excellent start.