Apache Airflow / Advanced: historical processing, deployment and recovery evidence / Lesson 15

Secrets, authentication and access

Review identities and trust boundaries for authors, operators and executing tasks without exposing credentials.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish UI/API permissions from task access to external systems.
Explain why DAG authoring is a trusted code-execution capability.

The user and the task need different access reviews

A person viewing an Airflow run and a worker accessing an external warehouse can use entirely different identities. UI/API authentication establishes who interacts with orchestration controls, while task runtime credentials authorize external work. A viewer's ability to read logs does not mean they should see connection passwords, and a task's database write permission does not justify cluster administration. Map each required operation to its actual principal and resource. Keep deployment-manager, DAG-author and operations responsibilities explicit. Airflow supports pluggable authentication managers, so detailed role behavior depends on the selected manager and deployment policy rather than one universal role-name table.

A DAG file is executable code

DAG authors can supply Python that executes during processing and task runtime, with consequences determined by the deployment's isolation and credentials. A nice graph and serialized definition do not sandbox untrusted code. Shared workers and triggerers can expose common resource contexts, so visual task separation is not a strong tenant boundary by itself. Review code submission, package installation authority and execution permissions through the deployment's trusted process. Airflow 3's public Task SDK restricts direct task interaction with metadata internals, but that API boundary does not turn arbitrary user code into a safe untrusted plugin or remove the need for infrastructure isolation.

Keep retrieval and disclosure separate

Connections and supported secret backends let task code reference a configured identifier instead of carrying secret bytes in source. Retrieval still needs the correct backend, naming contract and runtime identity. A successful lookup is not evidence that the target permits the requested operation or that its certificate matches. Avoid dumping context, connection extras or complete environment dictionaries into logs. Masking can reduce accidental disclosure, but it is not permission enforcement and cannot guarantee arbitrary transformed secrets remain hidden. Rotation should verify both new task starts and long-running work under the approved overlap policy without publishing real credentials in examples.

Check a small expected-operation matrix

The offline set allows a synthetic viewer to inspect runs and a synthetic report task to read one input family. It deliberately denies the viewer a write and the task an administrative operation. This is plain set membership, not an evaluator for an Airflow authentication manager, cloud IAM or database grants. Its purpose is to make negative cases part of the review. Real authorization also includes resource scope, inherited permissions and identity mapping. A repair that makes a denied operation succeed should be checked against intended denials so a connectivity incident does not silently broaden the application's role.

python Offline access-review matrix

allowed = {('viewer-lab', 'view-runs'), ('task-report-lab', 'read-manifest')}
assert ('viewer-lab', 'view-runs') in allowed
assert ('viewer-lab', 'write-report') not in allowed
assert ('task-report-lab', 'administer-platform') not in allowed
print('expected grants and denials hold')

Expected output

expected grants and denials hold

This checks declared tuples only; no Airflow manager or external authorization system runs.

Record an access contract without handling secrets

The JSON design record states the secret reference and intended operation while omitting usernames, tokens and endpoints. It is a review artifact, not a runnable configuration or an instruction to create a connection. Keep actual secret provisioning in the deployment's approved system and include an evidence horizon for audits. Authentication, authorization, transport trust and code-execution trust need separate verification. This lesson reads no credential store, calls no API, modifies no role and performs no scan. The executed permission model checks only supplied synthetic tuples and makes no claim about real access enforcement.

json Synthetic credential-reference review record

{"task_principal":"task-report-lab","connection_reference":"lab_input_reader","operation":"read-manifest","resource_scope":"declared-business-interval","secret_material_in_source":false}

The reference is a logical label, not an existing connection or a verified permission grant.

Common errors and fixes

A read-only UI user receives sensitive connection details through logs.

Why it happens: Runtime diagnostic output crossed the intended disclosure boundary.
How to fix it: Remove unnecessary secret output and follow the approved incident and rotation process.
Prevent it next time: Review log content independently of UI permission names.

DAG author access is treated as harmless document editing.

Why it happens: The executable nature of Python definitions was ignored.
How to fix it: Apply trusted code review and deployment isolation appropriate to the environment.
Prevent it next time: Document author capabilities and shared runtime boundaries.

A secret lookup succeeds but the target operation is denied.

Why it happens: Retrieval was confused with external authorization.
How to fix it: Inspect the actual runtime principal and narrowly required resource operation.
Prevent it next time: Test intended grants and denials separately.

Check your understanding

1. Does masking a password in the interface prove the DAG author cannot use it?

Show worked solution

No. Display masking and execution capabilities are different controls. A trusted author can supply code that runs with the task environment’s available permissions. Review credential scope, code submission and runtime isolation together instead of inferring a strong trust boundary from hidden text in one screen.

2. Why does the permission model include denied operations?