Assessing Software Development

How to Measure the Quality of Software Development Work

...Without Being a Software Developer

(Or, "3 Things a Developer Wishes His Manager Understood")

There's a great difficulty in managing software. This is because many organizations call on non-developers to make decisions impacting software development. But non-developers do not have the qualifications to understand the underlying software.

So, if you do not understand software, how can you make a good decision related to it? The easy answer is--you can't. But, if you don't like easy answers...

You need to break through the masked complexity and gain a shared understanding.

Masked Complexity

The challenge is in the masked complexity. The great thing about software is that it masks complexity. The terrible thing about software is that it masks complexity.

Software is made of stuff that is not readily understood by most people. Yet, to be useful, it is always applied to things that are understood by people. So, it masks that complexity: hiding what is not understood behind an illusion of something that is understandable.

The great thing about software is that it masks complexity. The terrible thing about software is that it masks complexity.

It is often difficult to tell the real thing from an illusion, leading to the risk of making bad decisions. If you don't understand something, that is a little problem. It is much worse if you don't understand it, but you think you do. This is a situation where successful management requires personal integrity: for example, honesty and humility.

Shared Understanding

A shared understanding may consist of accurate informants, metrics, and qualifications. Yet be aware that things are not as they seem to be. Let's start with what not to do.

1. Informants: To Whom You Should Listen: 
• Don't ask "a developer"...ask a developer
2. Metrics: What You Need to Measure:
• Don't measure "results"...measure results
3. Qualifications: Who You Need to Become:
•  Don't become "a developer"...become a developer

I. Informants: To Whom You Should Listen

If you are not a developer, and you need to make a decision regarding software, the first instinct that seems like a good one, is to ask a developer--an informant. This is both a great and a terrible idea. It is very important to listen. However, to avoid making bad decisions, you must also be very careful to whom you listen.

You may think you can know a developer when you see one. However, how could a non-developer discern whether someone else is a real developer? You should maintain the humility to admit that you do not know, what you do not know. Otherwise, you might pick the wrong informant because you never personally understood the underlying subject (of software).

Asking Developers About Developers

You may ask other developers, using democracy to seek an impression of which developers are good. But, developers conspire to sabotage the impression of each other, or to prop each other up (which may, surprisingly, be worse for everyone in the long run). In my experience, these counterproductive behaviors occur more commonly than one might like to think.

[D]evelopers conspire to sabotage the impression of each other, or to prop each other up...

Judging an Informant by Their Work

You might try to judge a person by their work. Thy might seem to do good work and produce good software, but remain aware that you might also be mistaken about the quality of the work (see sections below).

Listening to the right people can help; but, listening, by itself, is not enough. You still lack a trustworthy method to discern who are the right informants. Good developers make good informants about development. But, methods for non-developers to judge whether others are or are not good developers, are illusions. Those methods would provide a false sense of security while you are still left making an arbitrary decision.

Beware of picking an informant based on ulterior motives.

II. Metrics: What You Need to Measure

The key to using metrics to make good decisions, is to measure the right things.

If you can't simply pick an informant and listen, perhaps there's a more direct method. It may seem plain enough, that metrics would help you make good decisions.

You might evaluate something that is difficult for you to understand (such as software), simply by measuring that thing's results. However, the process of measuring is not necessarily useful in and of itself. Beware of adopting the external appearance of a principle without being truly faithful to its core idea.

Once again, the key to using metrics to make good decisions, is to measure the right things. Otherwise, you might initiate all kinds of activity surrounding metrics, while gaining nothing more than a greater false sense of security while continuing to make even worse decisions.

Examples

Note to the Non-technical: Don't let technical terminology in this section throw you off. If you don't know what some of these things mean (such as "lines of code," coding tasks being marked as complete, or automated tests running and passing), it's OK to skip those lines and go on reading. You won't miss anything important without these examples.

Some informants may claim to you that software with less lines of code is better, while they have simply compressed the same code onto fewer lines, (possibly also making it less readable, and, therefore, worse). Some coding tasks may be marked as completed quickly, but the code may be full of bugs. Code may pass tests now, but break in terrible ways later.

Unfortunately, automated tests are often written to verify that software is working a certain way, when that is the wrong way in the first place. Or a test may be written to appear to run successfully, only because it does not truly test the part of the software it is supposed to test.

Measurement is Not Enough

So, how can you tell what to measure, without understanding the underlying subject? Once again, we are left with a paradox like the chicken or the egg. You could try asking your informants; but how do you know if your informants are good without first having good metrics? You might think the sheer volume of inputs into your decision could help, and it might. But you could have many bad informants and bad metrics at the same time. And, in my experience, dare I say, that is exactly the situation at many big enterprises today.

III. Qualifications: Who You Need to Become

So, as a non-developer making decisions about development, what can you do? You can't simply ask a developer, because you might be mistaken about who's a real one. You can't simply start measuring a bunch of things, because you might be led to trust misleading indicators. Even using a combination of informants and metrics, they might both be bad. It seems that the only solution remaining is: decision makers must become developers, and developers must become decision makers.

There truly is not a silver bullet which allows someone to make good decisions directly related to something they do not understand. If you realize your lack of understanding and honestly admit it to yourself and others, that is a great start. That acknowledgement helps prevent you from having a false sense of security and making many wasteful expenditures on illusions.

But ultimately, you must gain an understanding of the underlying subject in order to be assured of the ability to make good decisions related to it. I hope that you are not offended by how this statement appears belittling. If we made this statement about any other subject, it would be simplistic. The fact is, that software is a thing which is particularly deceptive.

Why Smart People Are Dumb About Software

Simpler subjects simply have different people with varying levels of competence in those subjects. Unlike other subjects, software does not simply have that situation. Instead, the masked complexity of software causes there to also be many deeply sophisticated systems of illusion about who is competent in it, and how competent they are. You may be forgiven for falling prey to this system, as you would be in the vast majority in doing so.

Beware of Becoming "One of Them"

So this is great: now you know you can fix all these problems simply by gaining an understanding of software. But, last but not least, there is an even risk greater in that. If you learn software development, make sure you truly gain that understanding.

Whatever you do, please don't become one of those authority figures who thinks they understand software, but still doesn't. You would make the world a better place by remaining a non-developer and knowing it, rather than becoming an incompetent developer who thinks you are a competent one.

Beware of the Education System

Don't simply try to become a developer through the education system. The educational system falls prey to the deceptive nature of software just as much as business does. There are plenty of professors in the field who are completely incompetent on the subject. There are many courses which pretend to convey education in software development but actually teach bad and wrong things about it. There are exams in technology classes which you would pass if you are not a real developer, and fail if you are. It might be said that to come out of a software development education learning something, you must have either already known software development going into it and have a good memory, or have taught it to yourself in your own time despite the official courses.

Side Note: With the software development education system in such a state, it is no wonder there are so many issues with software development.

Don't get me wrong. There's a lot to be learned in the education system. But, once again, it's not a silver bullet. If you are not a real software developer, don't think the education system can make you one. Take things with a grain of salt and be discerning about what is not really as it appears. 

A Better Way

We hope you have found this article helpful, in warning you to avoid many common, deceptive, and painful pitfalls. In conclusion, we feel the true solution to the problems with software must go deeper. It must be one which deals with the root problem. We envision a system in which the software developers are the decision makers and vice versa. To facilitate this process, we have proposed a product idea. Please check it out and share your suggestions.