Verdict
No
No, do not test every value a function accepts; test one value from each class of input that the code treats in its own way, plus the values on both sides of every boundary.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityimmediately
- Reversibilitywith-effort
- Test costheavy
Do not test every value a function accepts; test one value from each class of input and both sides of every boundary. The typical case is the untested titles of a function that turns product titles into URL slugs and already has those tests. Blast radius is users and Change frequency is regularly. Detectability is immediately, because the other titles run through the same branches as the tested ones, so a broken branch fails its test in CI. Reversibility is with-effort, since wrong slugs in stored URLs need a repair script, and Test cost is heavy, because each extra title needs a hand-written expected slug, so rule R13 gives Do not test.
| When | Decision | Why |
|---|---|---|
| No test covers a slug function yet, and a wrong slug is saved without an error | Test: one title from each class and both sides of the length limit | Detectability rises to eventually and Test cost falls to moderate: a few classes need few expected values |
| Each of a few values has a table entry, such as a retry rule per HTTP status code | Test: one parameterized test with a row for every value | Test cost falls to trivial and Detectability rises to eventually: a wrong entry silently stops retries |
| One rule holds for every input, such as a slug made only of lowercase letters, digits and hyphens | Test: a property-based test that generates titles and checks the rule | Detectability rises to eventually, since an emoji title gives a plausible broken slug, and Test cost falls to moderate, since the rule replaces expected values |
| The function computes a fee from an order amount in cents | Test mandatory: each tier boundary and one cent either side of it | Blast radius rises to money and Reversibility to costly: a wrong fee ends in refunds |
| The function decides which of a handful of roles may perform each action | Test mandatory: every pair of role and action with its expected answer | Blast radius rises to safety-or-legal and Detectability to never: a wrong allow raises no error |
| The inputs are the stored titles that a one-time migration turns into slugs | Test it differently: run the migration on a copy and check every new slug with a query | Change frequency falls to once: a test in the suite would never run again |
What breaks if you don't test
Failures come from a class nobody listed, not from untested values of a tested class. A slug function that turns each run of characters outside a-z and 0-9 into a hyphen makes "Größe 42 Jacke" into gr-e-42-jacke, and a German customer reports the URL weeks later. One test with an accented title catches it; no list of every title exists, because text has no last value.
What you lose if you over-test
At a minute per expected slug, a table of 5,000 titles takes 83 hours to write, and a rule change, such as spelling ß as "ss", edits hundreds of rows. Expected values generated by the slug function itself pass whatever it returns. A loop over every pair of 32-bit integers makes 2^64 calls; at one billion calls per second, it runs for about 585 years.
What to do instead
- List the classes of input the code treats in its own way. For a slug: plain words, digits, punctuation, accented letters, emoji, an empty title and an overlong title.
- Test one value from each class and both sides of each boundary, such as a title at the length limit and one character over it.
- Put the values in one parameterized test with pytest or JUnit, so a new class costs one line.
- When a rule holds for every input, add a property-based test with Hypothesis or fast-check.
When the answer changes
- Each of a few values has its own branch or table entry.
- A wrong result would move money or grant access.
- The function takes one 32-bit value, and a trusted reference implementation exists.
Counterexample + Code example
Four billion floats in 90 seconds
The usual answer is wrong for a function of one 32-bit float. Bruce Dawson compared fast ceil, floor and round functions against the C runtime versions on all 4,294,967,296 float bit patterns, in about 90 seconds. XMVectorCeiling in DirectXMath 3.03 claimed to handle all floats, yet gave wrong results for 880,803,839 of them, mostly tiny and odd numbers. Test cost falls from heavy to moderate, because the reference supplies every expected output, and Detectability rises to eventually, because a ceiling of 4 for the input 3 looks like a normal number. Even at Change frequency rarely, that gives Test minimally, with this loop as the one test:
#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
float fast_ceil(float x); /* the function under test */
int main(void) {
uint64_t failures = 0;
for (uint64_t i = 0; i <= UINT32_MAX; i++) {
uint32_t bits = (uint32_t)i;
float x, got, want;
memcpy(&x, &bits, sizeof x);
got = fast_ceil(x);
want = ceilf(x);
/* Any NaN passes for a NaN; otherwise the bits must match,
so a 0.0f where ceilf gives -0.0f is a failure */
failures += isnan(want) ? !isnan(got)
: memcmp(&got, &want, sizeof got) != 0;
}
printf("%llu failures\n", (unsigned long long)failures);
return failures != 0;
}
Related questions
FAQ
- Should I unit test multiple input values to a function?
Yes, test several input values when the function treats them in different ways: one value from each class and both sides of each boundary. Values that take the same branch add run time and no information.
- Should all possible counter-cases be tested?
No, test one counter-case for each rule the code enforces, at the edge of the rule, plus one valid value next to it. For a username of at most 20 characters, test an empty name, a name of 21 characters and a name of 20.
- How do I choose which input values to test?
Take one input value from each class the code treats in its own way, found in its branches and rules, plus both sides of each limit. For a slug function, that is one title per class plus titles at and just over the length limit.