Web Security / Advanced: assurance, supply chain, and recovery / Lesson 16

Security verification and regression evidence

Map versioned requirements to safe tests and reviews that verify the real boundary rather than merely producing a green report.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Choose unit, integration, browser, and review evidence by claim.
Verify refusal, unchanged state, and legitimate behavior.

A test proves only what its boundary observes

A pure policy test can exercise a rule without proving that every route calls it. A framework integration test can validate middleware while missing browser cookie behavior. A browser test can verify a user journey without establishing database atomicity. Choose the smallest boundary capable of revealing the failure and add complementary evidence where assumptions cross layers. Use ASVS 5.0.0 requirements with explicit applicability and version mapping rather than treating a tool’s test count as a security score. This course executes only benign local fixtures and does not run a scanner, attack, browser automation, or configured application.

Assert unchanged state on denial

The complete local test suite models one fictional owner-only edit operation. It checks legitimate change, unrelated-user denial with the record untouched, and missing-subject denial. These are behavior assertions, not private implementation call counts. The function is intentionally small and lacks a real route, session, data store, or transaction. In production tests, import the actual policy or service instead of maintaining a duplicate rule inside the test. The useful claim is that these supplied inputs produce the declared local results, while server wiring and concurrency need separate authorized verification.

python A complete harmless local access-regression suite

import unittest

def edit_title(subject, record, title):
    if not subject or subject != record["owner"]:
        return False
    record["title"] = title
    return True

class AccessTests(unittest.TestCase):
    def test_owner_can_edit(self):
        record = {"owner": "lab-a", "title": "Before"}
        self.assertTrue(edit_title("lab-a", record, "After"))
        self.assertEqual(record["title"], "After")
    def test_denial_preserves_state(self):
        record = {"owner": "lab-a", "title": "Before"}
        self.assertFalse(edit_title("lab-b", record, "After"))
        self.assertEqual(record, {"owner": "lab-a", "title": "Before"})
    def test_missing_subject_is_denied(self):
        record = {"owner": "lab-a", "title": "Before"}
        self.assertFalse(edit_title(None, record, "After"))

if __name__ == "__main__":
    unittest.main()

Runs with Python’s standard-library unittest and synthetic dictionaries only. Three tests should pass. It does not exercise an API, database, tenant system, session, or real identity.

Control fixtures and avoid operational targets

Use owned synthetic identities, records, files, and dependency fakes with explicit cleanup. Do not test another person’s service or use real credentials to reproduce a classroom scenario. Control time and asynchronous outcomes so expiry and race tests are repeatable. Keep expected failures distinct from harness errors and missing evidence. A retry that passes does not erase the first failure; preserve candidate identity and the relevant diagnostic context. Broad administrator fixtures can hide missing authorization, while overly realistic private data creates unnecessary exposure. Choose representative behavior with minimal sensitive material.

Bind verification to requirement and candidate identity

A useful evidence record includes the selected requirement version, candidate artifact or revision, method, environment, outcome, and limitations. The example uses a local label and explicitly planned evidence rather than inventing an ASVS requirement number. A real verifier must assess evidence authenticity and completeness, not merely accept a JSON field saying passed. Missing required checks should remain visible. Different tests can support complementary claims, and manual review can explain context not encoded in a fixture. Keep the review and exception owner accountable for residual risk instead of turning every incomplete result into a silent pass.

json A fictional requirement-to-evidence record

{
  "localRequirement": "LOCAL-DENIED-EDIT-NO-MUTATION",
  "standardMapping": "reviewed-ASVS-5.0.0-selection",
  "candidate": "synthetic-build-label",
  "method": "unit-and-authorized-integration",
  "requiredChecks": ["legitimate-edit", "denied-edit", "unchanged-downstream-state"],
  "evidenceState": "planned",
  "ownerRole": "verification-maintainer"
}

The schema is application-defined and contains no real approval or artifact digest. planned is not passing evidence. The record performs no tests and does not authorize deployment.

Review coverage and changes honestly

Security checks need maintenance when dependencies, routes, identity rules, response policy, or data flows change. A clean automated result supports the tool’s scope and configured cases, not absence of all vulnerabilities. Review exclusions, false positives, false negatives, and untested paths with the relevant owner. Add a focused regression case when a defect is fixed, and verify both protection and ordinary task completion. Keep assessments and operational security testing under separate explicit authorization. The practical outcome is traceable evidence for stated requirements with known gaps, not an unsupported certification claim or a test suite that only succeeds because it never exercises denial.

Common errors and fixes

A unit rule test is called complete route enforcement

Why it happens: The test boundary is wider in the claim than in execution.
How to fix it: Add appropriate authorized integration evidence.
Prevent it next time: State the boundary and limits of every test.

Denied status is checked but effects are ignored

Why it happens: The assertion does not observe protected state.
How to fix it: Verify record and downstream invariants.
Prevent it next time: Include unchanged-state checks in negative cases.

A green report hides missing checks

Why it happens: Only received results enter the denominator.
How to fix it: Compare expected evidence with actual completed checks.
Prevent it next time: Model missing and harness-failure states explicitly.

Check your understanding

1. Why include an allowed case in an authorization regression test?

Show worked solution

A system that denies everyone can pass denial-only checks while breaking the product. Verify the intended legitimate task as well as unrelated and missing subjects, then assess actual route and data behavior separately.

2. A scanner reports zero findings. What should the review record preserve?