Software QA / Advanced: strategy, resilience, and leadership / Lesson 16

Property-based and metamorphic testing

Go beyond individual expected outputs by testing invariants and relationships between executions. Derive valid properties for sorting and cart totals, then examine shrinking, replay, and misleading universal claims.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Express domain-limited properties and generators.
Design metamorphic relations when exact oracles are costly.
Preserve counterexamples and evaluate property strength.

A property is an assertion over a domain

Example-based testing names particular inputs and expected outputs. Property-based testing describes a domain of generated inputs and an invariant expected to hold for them. For a nonnegative integer cart without discounts or shipping, adding a nonnegative item must not decrease the total. The domain restrictions are essential: a discount threshold could invalidate monotonicity in a different product. Define generators that reach meaningful boundaries and combinations rather than producing only ordinary values. Generated tests still sample executions; passing them is not mathematical proof that the property holds universally.

Worked case: sorting needs more than ordering

A sorting function should return a nondecreasing sequence and preserve the multiset of input values. Checking only order would let a function returning an empty list pass. Add length and multiplicity preservation, including duplicate and negative values. Idempotence—sorting an already sorted result changes nothing—is another useful relation, but a function that always returns empty also satisfies it. Combine properties with a few exact examples and, where practical, an independent reference. The following deterministic bounded enumeration illustrates the idea without requiring a third-party framework or claiming exhaustive coverage of all lists.

python Forty bounded input lists for property illustration

from itertools import product
from collections import Counter
for size in range(4):
    for values in product([-1, 0, 1], repeat=size):
        result = sorted(values)
        assert all(a <= b for a, b in zip(result, result[1:]))
        assert Counter(result) == Counter(values)
        assert sorted(result) == result

Sizes 0–3 over three values produce 1+3+9+27=40 inputs. The demonstration checks standard-library sorting, not an unprovided application implementation.

Relate executions when an exact answer is hard

A metamorphic relation states how output should change, or remain unchanged, after a meaningful input transformation. For a search service with deterministic filters, narrowing the filter should not add identifiers outside the wider result set, assuming no pagination or changing data. For an export, permuting input rows should preserve the multiset of exported business records if order is irrelevant. State exclusions explicitly. A relation is a partial oracle: a broken search returning no results can satisfy subset checks, so retain positive examples that require known matching records to appear.

Worked case: permuting a simple cart

For a cart whose total is the sum of nonnegative integer prices, permuting items must preserve the total, adding a zero-priced item must preserve it, and concatenating two carts must add their totals. For [120, 80], exact total 200 anchors the relation; reversing still gives 200, and appending zero still gives 200. These properties fail to describe order-dependent promotions or shipping thresholds, so do not transfer them blindly. If a generated case fails, preserve the input and reduce it to a smaller counterexample without changing the rule that made it fail.

python Metamorphic relations for a deliberately simple price model

def total(items):
    return sum(items)

a, b = [120, 80], [50]
assert total(a) == 200
assert total(list(reversed(a))) == total(a)
assert total(a + [0]) == total(a)
assert total(a + b) == total(a) + total(b)

The exact 200-cent assertion prevents an always-zero implementation from satisfying every relation unnoticed.

Use failures to improve the specification

Tools such as Hypothesis generate and simplify failing cases, but shrinking can expose that the property itself was wrong. Triage whether the counterexample violates a true requirement, a generator precondition, or an unjustified assumption. Store a reproducible failing example with seed or replay information supported by the chosen tool, and turn important failures into stable regression cases. Review mutation ideas mentally or with controlled tooling: would returning an empty list or constant zero pass? This challenges oracle strength without treating any mutation score as proof of complete fault detection.

Common errors and fixes

A property ignores its domain restrictions

Why it happens: A simple model is applied to richer pricing rules.
How to fix it: State assumptions and split domains where rules differ.
Prevent it next time: Review properties with concrete counterexamples.

Weak properties accept an empty result

Why it happens: Only ordering or subset relationships are checked.
How to fix it: Add preservation and positive exact examples.
Prevent it next time: Ask whether constant implementations would pass.

A random failure cannot be reproduced

Why it happens: Input, seed, or replay data was discarded.
How to fix it: Preserve the counterexample and simplify it carefully.
Prevent it next time: Make replay evidence part of generated-test reporting.

Check your understanding

1. Why is “sorted output is ordered” insufficient?

Show worked solution

An empty output is ordered but loses every input. Add multiset preservation, including duplicates, plus at least one exact expected example. For input [2, 1, 2], require [1, 2, 2]; a set comparison alone would miss a lost duplicate.

2. Propose a relation for deterministic filtered export, with a limitation.