API Testing / Advanced: versioned evidence and review / Lesson 13

Test properties and metamorphic relations

Extend example testing with justified invariants and transformations over small finite inputs.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Derive a property from a contract rather than from implementation convenience.
Use a transformation when an exact output is hard to enumerate.

State the quantified claim carefully

An example checks one chosen input. A property states a relationship intended to hold for a defined input domain, such as sorting preserving the number and multiplicity of catalog identifiers. The domain matters: a rule for unique nonempty strings does not automatically apply to nulls or mixed numeric types. Derive the property from the public contract and name its assumptions. A property that merely restates the implementation can repeat the same defect. Use a few explicit examples to explain the claim before expanding its input combinations.

Use a justified transformation

For a contract that sorts by a stable identifier, permuting the input fixture should not change the ordered output. This is a metamorphic relation: compare related executions instead of inventing a separate expected list for every permutation. The first program enumerates all six permutations of three unique identifiers and requires the same sorted result. It uses Python’s sort as the classroom subject, not an API implementation. The relation would be inappropriate for a contract that intentionally preserves insertion order, so the transformation must follow the promise being tested.

python Enumerate a small complete permutation domain

from itertools import permutations
expected = ["b1", "b2", "b3"]
checked = 0
for values in permutations(["b3", "b1", "b2"]):
    assert sorted(values) == expected
    checked += 1
assert checked == 6
print("six permutations accepted")

Expected output

six permutations accepted

The complete local program checks all permutations of this exact three-item domain. That is exhaustive for these permutations only. It supplies no evidence about a server query, collation, large inputs, or mutable collection traversal.

Test an algebraic property with a counterexample

A documented canonicalization function may promise that applying it twice gives the same result as applying it once. That algebraic property can detect a transformation that adds new decoration on every pass. It still does not prove that the canonical value is correct; an always-empty function also has the property. Pair relational checks with concrete expected examples and preservation constraints. The second program combines an idempotence check with an exact result for a fictional tag normalizer. It is not a recommendation to normalize every API identifier or user-visible name this way.

python Combine a relation with concrete expected outputs

def canonical_tag(value):
    return value.strip().lower()

for value in [" Essay ", "essay", "ESSAY", " poem"]:
    once = canonical_tag(value)
    assert canonical_tag(once) == once
assert canonical_tag(" Essay ") == "essay"
assert canonical_tag(" poem") == "poem"
print("canonicalization properties checked")

Expected output

canonicalization properties checked

The complete program checks four finite inputs and two exact outputs. Its normalization policy is explicitly fictional. It does not establish Unicode equivalence, locale-aware case rules, or safe handling of security-sensitive identifiers.

Generate within the meaningful domain

Generated cases should vary the attributes that could invalidate the property: duplicates, empty collections, boundary lengths, and equal sort keys where supported. Constrain generators to the documented domain, and write separate rejection properties for invalid inputs. If random generation is used, record the seed, generator version, and actual smallest failing input. Shrinking helps isolate the decisive difference but may discard contextual meaning if the generator is poorly designed. The finite examples here need no property-testing package and make no statistical claim about unexplored inputs.

Interpret a pass as bounded evidence

A property test explores cases; it is not a mathematical proof unless the entire claimed domain and implementation are formally covered. Report the property, assumptions, generated domain, and replay artifact. When a failure appears, preserve it as a focused regression example after fixing the defect. Avoid increasing case counts as the only response to an unclear property: review whether the relation is valid and whether the oracle can distinguish a wrong output. Properties complement contract examples, boundary checks, and integration evidence rather than replacing them.

Common errors and fixes

An always-empty implementation passes idempotence

Why it happens: The relation has no preservation or exact-output check.
How to fix it: Add concrete expected values and domain-preservation assertions.
Prevent it next time: Review simple wrong implementations against every property.

Permutation invariance rejects valid insertion order

Why it happens: The transformation contradicts the endpoint contract.
How to fix it: Use a relation justified by the documented ordering promise.
Prevent it next time: Record assumptions next to the property.

A random failure cannot be reproduced

Why it happens: Only the total case count was saved.
How to fix it: Preserve the actual input, seed, and generator version.
Prevent it next time: Promote reduced failures to stable regression cases.

Check your understanding

1. Does sorting idempotence alone prove that no books were lost?

Show worked solution

No. A function that always returns an empty list is also idempotent. Add a multiset-preservation property and concrete expected ordering examples, including duplicates if the domain permits them. Each property should rule out a different plausible defect rather than repeat the same weak claim.

2. A three-item permutation test checks six cases. What exactly is exhaustive?