in Testing

The World Through Developer-Covered Glasses: Eight Bad Testing Ideas #5

This is post 5 of 8 in the Eight Bad Testing Ideas series.

Ah, developers.  There’s a natural sort of yin-yang relationship between us and them: they see the world one way, and we see it another.  If their world accurately reflected reality, then we would not have jobs; since it doesn’t, it means that we need to be careful about taking their words at face value.  Because the developers really do believe their application works.  They kind of need to in order not to go completely crazy.

We, however, know better. We kind of have to in order to do our jobs.

Imagine a new feature.  You’ve been tasked with testing it.  The feature story says something like “solve customer concern #26” so, naturally, when the ticket lands in your to-do pile, you curse silently, and go to the developer to ask them what, exactly, they built. The conversation goes something like this:

US: What does this feature do?

THEM: It adds two numbers.

US: Okay.  So what happens if you give it letters?

THEM: That should never happen.

US: Alrighty, then… what happens if you give it negative numbers?

THEM: That shouldn’t make a difference. They’re just numbers.

US: I see.  Is there any risk of breaking another application?

THEM: Nope.  None whatsoever.

Every tester has had a conversation like this before.  Usually, more than one.  And in how many cases has the developer ever been right? Anyone?  Bueller? We keep trying to justify believing them, however, because we passionately want them to be right.  Boy, would our lives be easier if users really did use the application the way it was intended… if all of our features lived in isolation from one another… if all the world’s computers inherently understood that even though there is a dash at the beginning, a negative number is still a number! But of course, we know better. There are going to be bugs in the application, and the real challenge, of course, is finding them.  Unfortunately, our job is only made harder by the sweet, reassuring words of our coworkers; we have to learn not just when to listen to them, but what to listen for.

Misconception #1: Developers know the system, and their code, the best It’s very natural to assume that the person who wrote the application knows it better than anyone else.  And in the beginning, when there was just The Startup, that was probably true.  But now our systems are mammoth monstrosities that could take over New York City.  In order to add a feature effectively, the developers get split into groups that each focus on one small part of the application.  Do that long enough, and you start to forget the rest of application is even there. Even more than that, however, is that developers see their code the way parents see their children.  No matter how much of a brat the rest of the world thinks the kid is, the parents can never see them as anything less than perfect.  And woe betide the person who tries to tell them otherwise…. What this means is that it is our job to know how the pieces of the application fit together, because the developers likely don’t.  It also means that they are far more likely to overlook any minor “quirks” of their feature than we, or a customer, or the project manager, would be.

Take our example feature.  If you were to believe the developer after your conversation, then you know the following: 1. It is impossible that it will be given anything except two numbers 2. Negative numbers are guaranteed to work fine, despite containing non-numeric characters 3. There is absolutely no chance that this brand new, never-before seen feature will not fit seamlessly with the existing system

The fact is, developers don’t know how an end user will try to use their code, any more than the computer itself. They are also usually struggling to meet a deadline, which makes it very tempting to write off edge cases as “too improbable to worry about.” This means that even the most quality-minded developer will miss things, and that makes us the last line of defense.

Misconception #2: It is a waste of time to test for cases that will never occur This seems like a no-brainer, but I promise you I can find ways to disagree.

It’s true that we are generally on the same tight deadline as our hapless developer friends, and it is equally true that it is impossible to cover every edge case anyway — but how do we know which cases will occur, and which will not? Sure, it seems reasonable to assume that nobody will try to sum two letters.  Or, at least, that nobody will right now.  Three years from now, with whatever our application looks like by then — who knows.  And who knows what our own developers might try to do with the feature?  Perhaps they need to combine two groups of text and figure the same logic can be used. At least if there is a test, then they can find the problem before it makes its way to our to-do piles.

The bottom line is, if you can think of it, an end user can think of that — after all, we testers are also end users of other applications (and I don’t know about you, but I find bugs in them all the time).  Any public feature can potentially be seen by everyone, no matter how small the intended audience. Not to mention, things change; three years from now, someone might decide that this small, specialized feature should take on more responsibilities. It turns out that a large part of testing is predicting the future; at least, we need to be able to predict what might cause problems later on.

What To Do I encourage all testers — and developers — to implement the Rule of Shoulds (TM): any time the word “should” is used, there needs to be a test. If you analyze every conversation under this rule, then a lot of test cases should pop out at you — cases you might not have thought of before. For example, take our original conversation about this addition feature:

US: What does this feature do?

THEM: It adds two numbers.

WHAT THIS MEANS: It’s supposed to add two numbers. I haven’t actually tested it myself, though, because that’s your job, but I used a plus sign in my code, so how could that not work?

US: Okay.  So what happens if you give it letters?

THEM: That should never happen.

WHAT THIS MEANS: C’mon, nobody is that stupid. I mean, our users requested this feature, so obviously they know how to use it correctly, and besides, why would anyone other than a tester ever do that?

US: Alrighty, then… what happens if you give it negative numbers?

THEM: That shouldn’t make a difference. They’re just numbers.

WHAT THIS MEANS: I didn’t think of that. However, if I just smile and nod they’ll probably go along with it and I can go back to building my Killer Robot app, which I would much rather be doing.

US: I see.  Is there any risk of breaking another application?

THEM: Nope.  None whatsoever.

WHAT THIS MEANS: I don’t see how it could, but I’m not really certain. Still, if I tell them yes, it is going to require hours of regression testing and this thing needs to go out tomorrow.

Basically, should means “I don’t know for sure.” If you don’t know for sure, you should figure it out. The best way to both figure it out and document that you did so is to write a test.  Besides, not worrying about it is kind of a giant dare to the Universe, and those tend not to work out too well for mere mortals.

Even more importantly, this rule can be applied to protect against future bugs.  There are always assumptions that go into the design of the feature; in this case, one of them might be “we should never be given anything except positive integers.” If you apply the Rule of Shoulds, you know you need a test case for this, even if the project manager himself swears to you that the users will only be giving us valid input. If anything, you need the test case so that when somebody in the future decides that negative numbers ought to be allowed, you will catch the problem during regression testing.  And then you will be heralded as a hero, because you foresaw the future! You’ll get a raise, and a promotion, and a parade in your honor….

Okay, probably not. But you totally should.

That means you have to try it, right?

Write a Comment

Comment