Generative AI / Advanced: evaluation, reliability, and accountability / Lesson 17

Adversarial and boundary evaluation

Test whether application controls preserve task, access, and data boundaries under hostile-looking inputs without claiming a finite test set proves universal safety.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of adversarial and boundary evaluation.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

Start from assets and allowed outcomes

A useful security evaluation names the protected data, capabilities, user goal, trusted identity, and prohibited outcomes. Prompt-injection resistance, output safety, privacy, tool authorization, and robustness are related but distinct properties. A system can block an unauthorized write while still leaking a source title or producing unsupported advice. The first offline suite mutates a synthetic tool proposal and checks a narrow deterministic gate. It does not run an attack against a model, service, account, or real target. Define what each case can falsify and which important paths remain untested before reporting a security result.

python Offline proposal-mutation regression suite

def permitted(proposal):
    return (set(proposal) == {"tool", "source_id"}
            and proposal["tool"] == "lookup"
            and proposal["source_id"] == "lab-a")
cases = [({"tool": "lookup", "source_id": "lab-a"}, True),
         ({"tool": "lookup", "source_id": "lab-b"}, False),
         ({"tool": "send", "source_id": "lab-a"}, False),
         ({"tool": "lookup", "source_id": "lab-a", "approved": True}, False)]
for proposal, expected in cases:
    assert permitted(proposal) is expected
print(f"{len(cases)} boundary fixtures passed")

Expected output

4 boundary fixtures passed

Four hand-authored cases test one exact gate. The suite neither samples model behavior nor demonstrates protection against every data or output disclosure path.

Use negative cases that cross real trust boundaries

Include untrusted source instructions, wrong-tenant near-matches, extra approval fields, unsupported tool names, malformed output, conflicting citations, and stale resource versions. Keep protected actions fake and data public. A detector that searches for one suspicious phrase can miss alternative wording and can also block harmless quotations, so it is not a complete authorization layer. Test the actual consumer boundary independently of whether generation appears compliant. The first suite rejects extra fields, excluded sources, and unsupported operations even when their objects are syntactically valid. That is one defense property, not a statement that the model cannot be influenced.

Vary conditions without contaminating the holdout

Position, language, document formatting, retrieval order, tool output, and conversation history can change exposure. The second synthetic matrix identifies dimensions to review rather than supplying a real attack procedure. Separate development probing from protected evaluation and record cases discovered during investigation. A finite set that passes today does not prove all future inputs are safe, especially after model, prompt, index, or policy changes. Include benign and authorized cases so an overblocking system is not rewarded simply for refusing everything. Measure protected-boundary outcomes and user usefulness separately.

yaml Synthetic adversarial review matrix

evaluation_dimensions:
  - trusted-task-versus-retrieved-instruction
  - allowed-source-versus-other-tenant-neighbor
  - current-evidence-versus-stale-version
  - benign-quotation-versus-policy-changing-content
  - proposed-action-versus-actual-authorized-effect
status: synthetic-test-design-not-an-external-attack

The dimensions guide safe test design and do not run a red-team service, expose an exploit target, or certify that all variants have been evaluated.

Evaluate layered controls and residual risk

Input minimization, authorized retrieval, provenance labels, constrained output, tool allowlists, independent approval, destination policy, and monitoring can protect different boundaries. None should be advertised as a complete cure in isolation. If a model emits a prohibited proposal but the deterministic gate blocks it, record both events; do not call the model fully compliant or the system compromised without checking actual effects. If sensitive text reached the output, a denied tool action does not erase that disclosure. Preserve safe evidence, reviewer decisions, and ownership for remediation while keeping real secrets and customer data out of learning fixtures.

Authorize testing and report scope honestly

Real security testing needs explicit scope, authorization, data handling, stop conditions, and responsible response. This lesson performs only local fixture validation; it runs no scanner, model, network request, account action, or external tool. Report cases, configuration, expected outcomes, failures, and limitations without publishing sensitive artifacts. Re-run relevant regression cases when the controlled system changes, and prioritize fixes by actual consequence and exposure. Security evidence is strongest when it demonstrates an enforced boundary and an accountable recovery path, while acknowledging what has not been measured instead of claiming perfect injection resistance from a few successful prompts.

Common errors and fixes

A phrase blacklist is called complete injection defense

Why it happens: Pattern detection replaced enforced authority boundaries.
How to fix it: Validate access and effect policy independently of generated text.
Prevent it next time: Include varied and benign cases with actual consumer checks.

A blocked action is reported as full model compliance

Why it happens: Model proposal and system enforcement were conflated.
How to fix it: Report both the proposal failure and the blocked effect.
Prevent it next time: Use stage-specific security outcomes.

A passing finite suite is called universal safety

Why it happens: Untested inputs and future changes were ignored.
How to fix it: State scope, residual risk, and regression triggers.
Prevent it next time: Maintain versioned cases and accountable review.

Check your understanding

1. The model proposes an unauthorized write but the gate rejects it. What should evaluation record?

Show worked solution

Record a prohibited proposal and a successfully blocked effect as separate outcomes, then inspect other exposure paths. The model was not fully compliant, but the blocked action alone is not proof that a real write occurred. Keep the stage distinction and residual risk explicit.

2. Why include harmless quotations containing instruction-like words?