Secure Software Development / Intermediate: implementation boundaries and verification / Lesson 12

Design security verification that can detect failure

Match review and tests to concrete claims and check that the oracle rejects a wrong observation.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose evidence for a specific prevention, detection, or recovery claim.
Separate test presence, execution, and sufficient assertion strength.

Start with the claim and plausible defect

For the export requirement, a plausible defect is returning contacts from an unassigned workshop. A useful verification case must distinguish that wrong result from the permitted projection, not merely check that the handler returns an object. Decide what evidence is needed for prevention, what diagnostics support detection, and how recovery would be evaluated. These are separate claims. A static review can identify a missing scope predicate; an executed component case can show how the assembled handler behaves with owned fixtures. Neither alone proves the absence of every defect in the product.

Check that the oracle discriminates

An assertion can be present and still too weak. If it compares only a set of identifiers, duplicate protected records may disappear from the comparison. The first program evaluates a correct and an incorrect synthetic observation using multiplicity-aware equality. It verifies the oracle, not the application. Deliberately wrong fixture observations are a safe way to review assertion strength without modifying a live system. Include a minimal allowed result, a cross-scope result, and a duplicate result where those defects matter. Expected values should come from a reviewed contract rather than the same helper under test.

python Review the strength of a collection oracle

from collections import Counter
expected = ["registration-1", "registration-2"]
def accepted(observed):
    return Counter(observed) == Counter(expected)

assert accepted(["registration-2", "registration-1"]) is True
assert accepted(["registration-1", "registration-2", "registration-2"]) is False
assert accepted(["registration-1", "other-scope"]) is False
print("oracle rejects duplicate and cross-scope observations")

Expected output

oracle rejects duplicate and cross-scope observations

The complete program checks three local observations under a fictional unordered contract. It preserves duplicate counts. It cannot establish that a real endpoint applies scope, and it would be unsuitable if the endpoint promised a particular order.

Distinguish selected, executed, and passed

A suite can report success while omitting the important security case. Keep the intended inventory and actual executed inventory available for comparison. The second program identifies a missing denial case rather than assuming a green selected subset is complete. It is a bookkeeping calculation, not a test runner. Real execution records need the source revision, fixture, environment, exit result, and meaningful output. A test name containing security is not evidence of relevant assertions, and a coverage percentage does not show that the correct policy or side effect was checked.

python Compare intended and executed case inventories

intended = {"assigned-export", "different-organizer-denied", "duplicate-no-notice"}
executed = {"assigned-export", "duplicate-no-notice"}
missing = sorted(intended - executed)
assert missing == ["different-organizer-denied"]
assert not intended <= executed
print("missing denial case detected")

Expected output

missing denial case detected

The complete program detects one declared missing case. It does not execute any named test or inspect CI. The names are fictional and the missing result must remain pending until actual authorized execution supplies evidence.

Use tool findings as inputs to reasoning

Static analysis, dependency analysis, and dynamic verification see different parts of a system and have different false-positive and false-negative patterns. A tool finding requires context: affected code path, value control, destination interpretation, and existing defenses. A clean report does not establish complete security. Do not run scanners against configured applications or install packages for this lesson. Instead, learn to preserve the finding, form a bounded hypothesis, and choose an appropriate local reproduction or authorized review. Keep tool version and rule assumptions attached to any real evidence.

Verify repairs and adjacent behavior

A useful regression fails for the old defect and passes for the intended repair. If it never failed, investigate whether it exercises the same path or whether its oracle is too permissive. Add the legitimate neighboring behavior so a repair cannot pass by blocking all work or suppressing every event. Broaden verification when a shared mechanism or unresolved risk justifies it, not by endlessly repeating unrelated suites. State exactly which programs ran and which platform boundaries remain unexecuted. This discipline makes a verification package reviewable while preserving the uncertainty that responsible release decisions need.

Common errors and fixes

A green subset is called complete verification

Why it happens: Selection and execution are not compared with the intended inventory.
How to fix it: Record missing cases and execute the authorized relevant checks.
Prevent it next time: Keep selection rules and actual case lists reviewable.

A regression passes before the repair

Why it happens: The case may not discriminate the original defect.
How to fix it: Review path, fixture, and oracle with a wrong observation.
Prevent it next time: Preserve evidence that the regression detects the failure.

A clean tool report is treated as security proof

Why it happens: Tool scope and blind spots are ignored.
How to fix it: State the inspected boundary and add evidence for other claims.
Prevent it next time: Review findings and absences with explicit assumptions.

Check your understanding

1. Why test a deliberately wrong observation against the assertion helper?

Show worked solution

It demonstrates whether the oracle can reject a plausible defect without changing a live system. If duplicate or cross-scope values still pass, strengthen the comparison according to the contract. This review of the oracle complements, rather than replaces, execution of the application boundary.

2. A fix blocks every export and all denial cases pass. Is verification adequate?