We are quite familiar with the eternal debate and endless list of opinions about what constitutes quality. We stated right up front, we were not trying to define quality for anyone but ourselves. But, until further evolution, (which is quite possible) this is our definition.

We spent a fair amount of the morning discussing our own opinions and knowledge and scouring the internet. Ultimately, we came away with 4 statements that we felt framed the starting basis for our discussion. They were –

1. Quality is value to some person.
2. Quality is a relationship between a user and the product.
3. Quality is fitness for use.
4. Quality is conformance to requirements.

From these four statements, we began to discuss the specific business goals of Raymond James, our corporate culture, our current processes, and the stated goals and vision of the business. We talked about how we very specifically wanted any formal statement we made not to be just words, but to be an actionable statement that drove the behavior of our SQA organization. We worked diligently to make that a reality. Ultimately, this is what we came up with:

What is software quality?
Confidence that the product provides value to the business by meeting the needs of its users in a cost effective and efficient manner.

How is software quality achieved?
Confidence – Defined measures and metrics that are used to determine success of meeting the user needs

that the software solutions provides value to the business — Each business area must interpret and articulate the value that the product must/will provide based on the understood user needs

by meeting the needs of its users — The users’ needs must be defined in some way. We do this by gathering requirements, doing customer surveys, interviewing, etc.

in a cost effective — Making the best choices based on the value to the business, working within the project management triangle ? time, scope, cost (faster, better, cheaper)

and efficient manner — Using appropriate process to create products with value to the business and long term sustainability (maintainability, scalability, security)

I am very proud of what we have done. I feel it lends real credence to the rest of what we claim we want to do. It is my manager’s stated goal for Raymond James to be the premiere software testing organization in central Florida in the near future. We want highly skilled testers to WANT to come and work with us. And doing something this simple, yet this important, is a big step in the right direction, in my opinion.

I will try to get permission to post the entire vision statement in a follow up blog, when it is all finished. In the meantime, I welcome any comments on our definition.

I would also like to hear anyone else’s experience in defining quality, and how that definition has stood the test of time.

David

become aware of something both amusing and enlightening. In our parking lot, there are lights atop tall poles; nothing unusual about that. But as the timing of my comings and goings varied around an

average, I noticed that the parking lot does not fill up from closest to the building outwards. There are key positions that are left vacant, even when they are close to the building. These spaces

are under, or very near, the light posts. Now, if you live near a body of salt water, you know where this is going. You see, salt water birds are, generally speaking, large…and messy…and uncouth… so it

makes sense that the spot right under the lightpost would be NOT desired…unless for some reason you WANT your car covered in bird poop.

But as the days rolled on, and I continued to observe this, I realized something a bit surprising. The spot UNDER the lightpost was not the worst. The worst was the spot next to that…and always towards the direction of the bay. Apparently, the birds go to the water to get food…then come sit on the lightpost to digest it…then, as they begin to fly away, some muscular reflex discharges

the excess. But by then they are a few flaps into flight, headed back for more food, so more often than not, the target zone is the space adjacent to the spot under the lightpost…but not always. So both

spots are to be avoided.

So the lessons I learn here are that problems don’t always show up near their source. They also don’t always show up where we expect them. And systems are way more interdependent than we normally believe.

I love learning lessons about software testing from nature; in the first place, I love synchronicity in the universe, and I think we mimic nature more than we admit. We create organic systems by design,

and even inject mistakes in them that have organic properties. If you are a tester tasked with finding those mistakes, this is a valuable insight. Binary thinking is NOT the best approach to software testing.

So observations from the world around me that help me understand how these complex systems we design may go awry are a favorite source of inspiration.

So this is all well and good…until the other day, when I walked out into the parking lot, and as I walked by a parking space, predictably vacant, something caught my eye. It was a dead fish, lying in the

parking spot. That was unexpected. I immediately thought, “What the hell is that doing here?” A mile from the water, yet there it is…a fish out of water. Ok, so obviously, some bird dragged it here and

then dropped it. But then what? Flew a mile back to the bay to get another one, rather than just jumping off the lightpost and getting this one back? Why? This is the heart of exploratory testing…the

poop in the parking lot is reasonable and may have even been predictable…but this dead fish needs more data. When the results of your last test drive the design of your next test, you are doing

exploratory testing. And it is the most natural behavior on the face of the planet.

And while you may say, “Why is it so important?”, ask yourself this…would you rather come out and find a few splats of bird poo on you windshield…or a dead fish stuck in your windshield wiper?

See…now there’s a lesson about the value of risk based testing!

As you come and go, remember that software is designed by people for people, and while it is ultimately just lots of 1’s and 0’s chasing themselves around, the behavior it will display will at times be strikingly organic. Learn lessons from the world around you when expectations get shaken up a bit; question why, and try to learn what the insight says on a larger level, and how that may apply to these artificial universes we create. You will be a better software tester for it.

And don’t park under the lightpost…

David

As this was going on, her team decided to use her in a testers role, and tasked her with creating and executing some test cases.

A few days later, some of us were having a working lunch, and Liz was commenting on the fact that she was executing some tests that she had created, and that it was kinda boring. Picking up on this, I began to go down a fairly standard path of context driven type questions. Why was she executing the tests? Did she understand that in the creation of the tests, the testing had already occurred? So what are you really testing?

“That’s a good question.” she responded. (Note to self: that phrase is a strong indicator that something great is about to happen) “I am really checking to see if I would write it up the same now as I did originally; so, I guess I am testing my ability to be a good tester.”

Wow! That is without question the best defense I have ever heard for doing a test you have already done, when you have no reasonable expectation of anything different happening. And it is the ONLY time in my career anyone answered a question about testing with an answer about thinking, specifically about their own ability to think critically about their own work.

This happened the week before I left to start my new position, so I don’t get to work formally with Liz now, and that saddens me. But I am not far, we all stay in touch, and in fact a bunch of us just had lunch together; and I hope Liz continues to grow and evolve as a tester, and I hope I get to occasionally help.

The software testing community needs more people like Liz…sharp, inquisitive, aggressive, critical but not mean spirited. I love finding people like that, and I love having the chance to work with them. And now, thanx to Liz, I have a new way of seeing value in something that I used to consider fairly wasteful.

David

Last week was mostly corporate orientation, welcome aboard, meet the team, etc. As such, they have already proven themselves to be a great place to be; the corporate culture is warm and friendly, and no one is walking around the hallways shaking their head and grumbling to themselves. But this is Monday, and I expect that this week, the gloves are off, I am an employee, and there is an expectation that I will be productive, and now.

Now for those who have never been an independent consultant, it may be difficult to understand, but for those who have, the question is, “Dude, WHY?!?!” Having once walked away from the myth of corporate security, why on earth would you go back and be a slave to the man, having tasted the sweet freedom of independence?

To truly understand that, let me explain the process of how I came to be here. It all began with an email from Michael Bolton; seems a manager had seen he and James Bach at StarEast, and asked if he knew anyone who shared their viewpoints of software testing and may be interested in working for a company in the Tampa area. As luck would have it, I live there, so he forwarded the email to me and suggested I call them.

Over the course of a few conversations with a few nice people there, it became apparent to all of us that I did not fit the cookie cutter mold for what they were looking for. I don’t fit anyone’s cookie cutter mold…this is why I have not had a real job in over 12 years. But here is where it gets interesting. Rather than thanking me for my time and telling me I did not fit the mold…they went off and made a new mold, based specifically on our conversations.

Now, a company who will do that is serious about bucking the status quo to reach their goals. A company who will do that is serious about getting what they want, not through the application of brute force, but by being smart, and thinking outside the box, and compromising when a really good solution is not necessarily what they thought it was gonna be. THAT got my attention.

For six years I have worked at a large government contract, and while we have done some good work, the culture there is simply too large, too entrenched, too spread out and decentralized, and too political to make any large and meaningful changes. On the other hand, this new position is ripe with opportunity to do really meaningful work, and I find myself at a level where I can be heard and effective, with the apparent backing of a group of folks who have already demonstrated a serious desire to achieve their goals above and beyond all other considerations. I am a very, VERY goal driven person…so this resonates strongly with me.

So as I sit here looking out of my cube, I am incredibly excited at the possibilities that lie in front of me. I am thankful for good friends who I have had the honor and opportunity to work with over the years who respect me as much as I respect them, and recommended me for this. I am deeply honored that a lifetimes work, including a lot of swimming upstream, has finally gotten me to a place where that contrarian but tenaciously grounded viewpoint is valued and desired, and can be put to very good use. And I am enthusiastic and ready to get started.

David

Now I have been tasked with creating the test plan…THE Test Plan…(Trumpets sound, the clouds part, a golden chalice descends from heaven containing a scroll upon which is written The Test Plan…you get the picture)… for a new upcoming project. And, since it is a large organization with lots of projects going on, and they like for their official documentation to have a consistent look and feel, there is a template for the test plan. Okay, no problem with that. So I go open the template to start crafting the test plan. The template…the empty structure, devoid of any meaningful input at all…is already 28 pages long. My response to this was not positive.

And to make matters worse, I really do not have much good information to put into the test plan, because, primarily, communication and collaboration among the teams is not strong right now. So I am frustrated that creating a test plan is deemed the highest priority at this point.

It is these kinds of situations that caused me, some time back, to come up with the theorem of the well formed test plan.

The amount of formatting and verbosity applied to a test plan is inversely proportional to the amount of actual good testing represented by such plan.

David

Last night, the Rebel Alliance gathered at StarEast. The original plan was to gather at Cafe TuTu Tango, a local watering hole that would have done the canteena scene from StarWars proud. But then, as happens in these epic stories, a stroke of unbelievable luck crossed paths with a stroke of genius. In a deus ex machina that even Shakespeare would be jealous of, one of the alliance members got comped into a massive suite; really, the official max occupancy for this room was 78. Why did this happen? I am not really sure…the story I heard was something along the lines of “The hotel just wanted to give him the room” But our plans instantly shifted, and we gathered in his suite, stocked the kitchen with beer and soda and snacks, and ordered Thai food for dinner.

After some general socializing, the lightning talks began. I must say, lightning talks are a marvelous icebreaker. The AA version of getting to know each other — “Hi, my name is David, and I am a software tester” — is feeble. But spend a couple hours listening to fast paced 5 minute talks about something someone is interested in, and then participate in the Q&A afterward, and in a couple hours, you really start to get to know people. I especially liked Lanette Creamer’s presentation, “Herding Cats”, where she used her experiences getting her collection of cats to live happily together as a metaphor for her professional passion, which is collaborative testing and development. Since my own home is somewhat like Noah’s ark, the metaphor resonated strongly with me, and I immediately felt like she and I could find so many things to talk about, rather than just how to write good tests.

She later proved she could not only talk the talk, but also walk the walk, as she paired up with Shmuel Gershon to work through The Dice Game, being proctored by Michael Bolton. Their ability to work very well together was noted as being exemplary of how such things should really be done.
Shmuel continued to work with Michael throughout the evening, sharpening his skills as Michael continued to challenge him more and more. His enthusiasm for learning, and meeting new people, and sucking every bit of goodness out of the evening that he could, was infectious. Kudos to Intel in Israel for believing in him, that he could come all the way from there to here, and actually find and bring back something of value, not only for himself personally, but for them as well. Trust me, he was successful…and he has also distinguished himself as a young tester of note.

It was these kinds of experiences that made the evening special. The room was just the right size, as was the crowd, thanx to Matt for being wise enough to cap attendance. Through the evening, people gathered in corners and at coffee tables, in small groups of a few to a dozen; they would talk for half an hour or so, then get a drink or snack, and find a new group. Through the evening, I was a part of probably a dozen or so different conversations. There was no pressure to watch the hall for someone else you might feel compelled to schmooze with, no other schedule to be mindful of, no other distraction; but there was plenty to keep your interest — no one was looking at the clock and wondering when it was okay to leave. It was a marvelous evening.
And lest you believe that I am just pimping my own opinionated view, I can tell you that based on the collective experiences, conversations, blogs, tweets and emails of the last 48 hours, as I write this I know of at least two other large national conventions which are suddenly putting together events modeled directly on what happened last night. I hope to be, and present, at one if not both of them. And if I am fortunate enough to make that, I truly look forward to gathering with another Rebel Alliance group.

If you are headed for a conference soon, jump on this bandwagon. Get in touch with someone from the original Alliance, and if the conference you are going to is not doing something like this, do your own…that is the Rebel part. Even if they are, still consider doing your own…again, be a Rebel. It will make a difference, trust me, I know. I Love conferences. I go to them as often as I can…but this…this was special.

Long live the Rebellion.

David (Ben Kenobi) Gilbert
Charter member, Master Brewer, Rebel Alliance

Ultimately, I have to justify the expenditure for going to a conference with some practical business benefit. And in this business climate, networking is always one possible benefit. But large conferences can be a double edged sword in this regard. While there are tons of folks there to meet, it can be difficult to meet them. And, there are tons of other folks there meeting them, so a week later, will they even remember you?
And if you have a network of colleagues you reconnect with at conferences, how do you manage to make new contacts as well?

Matt Heusser has come up with a wonderful idea to help resolve this…something he has referred to as a conference within a conference. You can listen to a podcast about it here. He has issued invitations to a collection of folks he knows, and they know, to create a quick sub group, which he has dubbed The Rebel Alliance. The whole point of this is that when the conference is over, in the evenings, for one or two evenings, we are going to get together and have a casual mini-conference, with our own lightning talks and possible presentations. We are all going to blog about it and tweet about it.
Participation is being purposefully limited, so that it does not just become another huge gathering (such as the conference sponsored socials), but rather, everyone will actually get to know each other, and learn a bit about each other, so that when we leave, we will have a collection of a dozen or so business cards that is unique from the 200 or so that we grabbed with a quick “Hi, do you have a card?” For that dozen or so, we will have dinner, lightning talks, conversation afterward, some are heading to Disney after the conference, all of us are already communicating online, and that hopefully will continue. Everyone knows someone in the group who knows someone…no one in the group will be a complete unknown, so introductions will flow naturally.

This is a first, and I am not sure how it will evolve moving forward; but I would imagine that in future conferences, the Rebel Alliance formal attendees will be different, since the point is to create new business relations, not simply create a new clique. So if you are looking to a conference in the future, keep your ears and eyes open for Rebels, and join up at the conference you attend; make some real new connections, and have some fun at the same time. Eschew the ice breakers and conference sponsored fun night out, and embrace something that can really provide some return on your investment. Be a Rebel!

David

When I first started in software QA, I was one of the “We are the gatekeepers of quality!” kinds of people, considering myself the last vanguard before unleashing unclean software upon an unwitting world. I eventually gave up that belief, and came to a place where I viewed testing as an information service, providing timely and accurate information to management concerning the quality of a piece of software. Whether or not that software got released was a business decision, and the current state of quality was only one of many considerations that went into that decision. Then I became a member of the Context Driven school of testing. And then I came to work here.

Now, in the time that I have worked here, I have seen a lot of good work done. I have done a lot of good exploratory testing, I have even gotten to do exploratory load testing, and I have stopped a lot of bad software from landing in users hands, which ultimately is job satisfaction of a kind for us testers. But I have also seen at least three different projects go incredibly bad, and when a government project of the size and scope we work on here goes bad, it goes really BAD. Estimates of the cost in dollars are millions…not quite a bank bailout, but still pretty serious coin.

As I have looked around and tried to understand what causes these collossal failures, I have come to the conclusion that it is not just bad testing. It is not just bad development. It is not just bad program management. I think it is a series of events that is almost a natural outcome of organizations that become this large. Projects become interdependant; communication through the cloud of beauracracy is unclear; people come and go; with almost inexhuastable resources, and very lax accountability, mediocrity sets in; and one day, you wake up with two projects that seem to work fine independantly, but won’t talk to each other, yet are interdependant in the real world. Welcome to the brave new frontier of Systems instead of Applications. Soon after, the finger pointing begins, everyone circles the wagons, and both projects proceed to fall out of the sky and become grey smoking holes in the ground. The casualty count is high.

So given that, how do I try to make things better? This question has led me to a new field of study, and a new era in my continued professional development; I, and my company, are now beginning to get involved in Scope Management. Scope managers are not common, per se, in American organizations, although they have been gaining prominence for the last decade in other places such as Finland, The Netherlands, and Australia. A scope manager is essentially a mediator between the various consumers and producers of IT products in an organization. The skills involved come from diverse set of more commonly understood professions; Development, Project Management, Business Analyst, Customer Advocate, and Quality Assurance Engineer. Strong analytical skills as well as strong communication skills are essential. If you have been an independent consultant for any amount of time and with some measure of success, you likely have all of these skills.

So why do I think a scope manager can make a difference, when a tester cannot? Two reasons; First, they answer a different set of questions, e.g. “How do we all, all stakeholders, define this project” and “How do we monitor the evolution of this project and keep our discreet definitions in synch?” Second, they function at a much higher level, where the input carries more weight, and is less likely to get lost in the local politics. A scope manager never blames anyone for anything…he simply states truth and reality, and presents options; management then makes choices.

Now, some will say, this is a very document heavy, process heavy thing; you have sold your soul and are no longer context driven; to them I would say, you are partially right. This can be, and likely will be, relatively process and document heavy. But these last four years have taught me that that is simply the reality of the world, in some contexts…see, still context driven. Context driven does not say document heavy and process heavy are always bad…it says there are no best practices, nor are there any worst practices…just practices that work better in some instances than others. Think of it this way; if you are out on the lake on your personal boat on bright summer day, and the boat starts leaking, process and documentation may not be important; you have a few choices, including simply jumping over and swimming the 50 yards to shore. But if you are on an oceanliner in the middle of winter miles from anything but more water and that ship starts leaking, wouldn’t you prefer that there was some documentation and process to keep you safe? Size seems to shift the context that way…so I would not think a startup dot com would see value in scope management. But huge organizations, like my current client, will.

Also, I am getting in on the ground floor more or less, and talking with some of the influential folks in the industry, and we have already begun to discuss how we may try to adjust the basic process into a variant designed with more agile organizations in mind. And so I am excited about the opportunities ahead, and the chance to try and continue to make a positive difference.

If you are interested in Scope Management, and the training and services we will be providing, please visit our website for more information, and feel free to leave comments here about what you think of this new emerging concept.

David

The basic idea was that Shrini’s colleague was looking for a way to make exploratory testing fit into his process.  These two hot topic buzzwords, “repeatable” and “predictable” quickly became part of the conversation, as they always do.  Having already discussed “repeatable”, today I want to look at “predictable”.

As a starting point, I must say, I believe that this is an industry wide mis-use of the word predictable.  For something to be predictable means it can be known in advance.  So for a test to be predictable means you know the result of it in advance.  If you know the result already, why execute the test?  That is just plain stupid.  And I don’t honestly think that’s what someone means when they say, “I need the tests to be predictable”.   I think they are trying to express a different idea altogether.  I think what they want to be able to predict is the cost of the test…cost, in terms of time, resources, and risk to upsetting the apple cart that is the rest of the production system.  They want to be able to budget for the test in advance…far in advance.  They want the test to be…well, Budgetable.  But budgetable is not a real word, and so they use “Predictable” as a poor substitute.

So let’s allow that this is really the concept being sought after here, the ability to know in advance not the outcome of the test, but the cost of it.  This now begins to relegate software testing (management) to an exercise in budgeting.  “We can’t run that test…it costs too much”.  This is where the smoke and mirrors of automation begins to work it’s magic.  Because I CAN tell you in advance, with little variation, how long one of my scripts should take to run.  And how many people and machines it will need.  And I can schedule it in advance to run wherever you want it to, so it will fit very neatly in your clean little process.  And I cannot do that with exploratory testing.

Okay, that last part…that was a lie.  But before I explain why, you have to give up some bad ideas about exploratory testing.  Lets take three examples.

1.  I am your “exploratory tester”.  You come to me and say “The dev team just added new functionality to search and purchase widgets by date of manufacture.  Run some quick exploratory tests by logging in, looking for all parts manufactured on the first of each month in the last year, and ordering all of them.  They have pre-loaded the database, so they know what the results should be.  We really need to get this out the door, so do this as fast as you can, and don’t waste time on anything else.  How long will it take you?”   Okay, second, I cannot answer that…but FIRST, this is not even exploratory testing.  This is scripted testing…you just didn’t bother to write it down . Just because it is not done with a written script, and just because it is done by a person…these things do not make it exploratory.  Now, I would probably personally turn it into an exploratory test, at least a little, because I am personally incapable of NOT paying attention to lots of other stuff while I carry out your script; but as requested, this is simply NOT exploratory testing.

2.  I am your “exploratory tester”.  You come to me and say “The dev team just released a new build.  How long will it take you to exploratory test it?”   I cannot answer that either.  There are lots of answers I can give you, but none of them is what you’re looking for and they are not gonna make you happy.  But let’s destroy another myth here while we are at it.  If I am your “automated tester” instead of your “exploratory tester” I could not answer that question either.  If I have a bank of regression scripts, and I know that the last time they ran they took 10 hours and I needed 5 machines and 2 testers to do it, and I answer your question by saying “I can test the app in 10 hours with 5 machines and 2 testers” then you should fire me because I am an idiot.  I have no idea how many of my scripts are going to need rework, how much new functionality I am going to have to create new tests for, and how many of my old tests are no longer applicable.  So if you think automation will solve this problem for you, keep dreaming.

3.  I am your “exploratory testing consultant”.  You have flown me in for a week of consulting.  Dev just released the new build.  You ask me how long it will take to test it.  Here is my answer — “A week — OR, less, if within that week I find something so bad that you ask me to stop so you can work on it.  But then when you fix it I can start again, so we are right back to … a week.”  “Wow ,Really!?!  You can be done in a week?  That’s great!  How can you be so sure?”  “Simple…that’s how long my contract is.”  See, the point is, exploratory testing is never constrained by the test, it is only constrained by the context.  We can start sooner, test through greater adversity, and be more flexible than automation ever dreamed.  And yes, we can stop when we damn well please.  An exploratory tester will never say “I can’t give you my results for another 5 hours…sorry”  Instead of saying “I need the system for this long in order to run my tests”  we say “How soon and how long can I get the system” and whatever that answer is, we squeeze testing value out of it.

Which brings us to the next big problem here.  Even if we could tell them that magical mythical answer of “How long”, and they try to change testing into budgeting, they can’t…because good budgeting looks at costs and benefits, and makes tough choices.  And they are only dealing with the costs.  There is no way of knowing what the benefit of running, or not running ,some test, will ever have been.

The perception that automation is much more budgetable than exploratory testing is largely not true, and ignores a possibly huge hidden factory in the background as your automation engineers scramble to get the old tests to run in the new system.  The perception that it is inherently more valuable than exploratory testing is just plain as wrong as can be, since there is an incredibly strong argument to be made that a whole lot of good exploratory testing happens in the process of creating a single automated test.

But I will give you this much.  Unless your developers broke something that used to work, it sure as hell is a lot more predictable.  And that is the worst reason in the world to use automated testing over exploratory testing.

David

The theme is repeatability, and for any practitioner of exploratory testing who has ever had to endure a debate with someone who was not a proponent of exploratory testing, it is the mantra of dissent.  “Exploratory testing is crap, cuz it’s not repeatable!”  Oh, really?  Let’s think about that a minute…

First, when did “repeatable” become the goal anyway?  Do you really test software so that you can be repeatable? “Hey, yall, watch theuhs…I’m'a gonna do the same thaing today I done yesterday..ain’t that cool?”

Are the best football teams great because they are repeatable?  Do they come out and run the exact same plays, week after week, in exactly the same order?  Of course not.
Do you go to your favorite restaurants because the food is repeatable…always exactly the same, week after week?  (Careful, there is a fallacy buried there you may be buying into)
Why is going to the symphony or orchestra generally considered to be better than just listening to the CD?  The CD is repeatable, the live performance is not.

These are just a few examples of situations where repeatability is not only not the goal, but is frankly looked down upon.  And what, you may ask, do they have in common with software testing?  This…they are all examples of performance, involving skill, typically in a time constrained and uncontrolled environment.

Great football teams have to go out week after week, in different stadiums, in adverse weather, with hostile crowds on the road, dealing with injuries and personalities that change the landscape every time they take the field…but they have one hour of playing time each Sunday to be great anyway.

Great restaurants buy fresh ingredients whose potency and size and strength vary from purchase to purchase, season to season, and market to market.  Yet each day they have a few hours from the end of lunch until the beginning of dinner to crank out consistently fantastic cuisine for their patrons.

Symphonies and orchestras deal with different venues, from perfect indoor theatres to outdoor band shells, different members, different directors, and maybe even different instruments depending on the performance, but they get from curtain up to curtain down to enthrall us with beautiful music, and they normally do just that.

And if that is not enough to convince you that repeatability is not even a good goal to have, lets consider if it is even a possible goal.

Could the football team do the same thing week after week?  Lets say they even had the ability to dictate whether or not they kick off, and they always choose to…no matter how good the kicker is, will the ball always leave his foot at precisely the same angle, go exactly the same height and distance, with no variation at all in the force applied, landing in precisely the same spot each week?  We have already said they can’t control the environment, so what if the wind is blowing differently?  What if the ball is inflated differently?  Does repeatable mean doing it the same way, or getting the same results?

And how about that chef — does repeatable mean you follow the recipe precisely every time, never deviating at all for any reason? (BTW, I have never seen any restaurant where that was the case)  Or does repeatable mean you adjust as needed to get that perfect dish in the end?

Do I want Chopin’s Nocturnes to sound the same as they did every other time I ever heard them, or do I welcome a new directors interpretation, and a new pianists exposition of emotional delivery to perhaps illuminate some expressive passage I had missed before?

I think one of the real problems here is an inappropriate use of the word “repeatability”.  I think a more accurate term to describe what people want for testing the same thing over and over would be “consistancy”  Mirriam-Webster defines consistancy as ” free from variation or contradiction”.  So lets look at this in the context of a real, albeit simple, testing problem.

Suppose I am testing your application via exploratory testing.  I come back to you and say “The print functionality is broken;  I get an access violation”.  You challenge me with “Is the test that gave you that result repeatable?”  Now, I can go one of two directions here — the first is, “Of course not.  No test is ever repeatable.  You can no more repeat a software test than you can stick your toe in the same river twice.  The water is rushing by at 6 knots on it’s way to the ocean…that same group of water molecules will never be at that same spot at that same time (assuming I could travel back in time to begin with) so that I could stick that same toe (discounting the effects of aging on the molecular structure of the toe) into them again.  What a silly idea.”

Or, I can ask “What do you mean by repeatable?”

If your answer is “By repeatable, I mean did you write down all the steps involved so you can do them exactly the same later”, well, there are normally 4 ways to ask any application to print something…the main menu, the toolbar, a pop up menu, or a hotkey.  Either one of them really only involves one or two steps.  None of them is so similar to the other that anyone would be likely to confuse them.  In theory, they all call the same code.  So even if memory failed me later, and I could not remember which one I originally did, it shouldn’t really matter which one I do again, and in a real jam, I could do them all in about two minutes.  So what if I did not write it down…does that make it not repeatable?  So what if I don’t do it exactly the same, if I can show the same results…consistently.  Like the football team, consistently winning.  Like the chef, consistently delighting your palate?  Like the orchestra, consistently thrilling you with their music?  I think being able to test something consistently is far more valuable than repeatably, and I think without realizing it, this is often what people are looking for when they use the phrase “repeatable”.

Shrini’s colleague was very concerned about being able to make exploratory testing fit into a process, and the process police tell us that a process should be repeatable…but winning football games, cooking great meals, and making masterful music are all processes, and as we have discussed, consistancy, not repeatability, is the goal.  But you can see where the flawed terminology originates, and people tend to accept what they are taught.

I encourage you to challenge this notion of repeatability in testing, and see how often consistancy is a much better idea.  Teach those around you to be careful of their word choice, and to clearly express their goals.  We test software to learn something, not simply to do something the same way we did it before.  Repeatability is not the goal.

David