Kubernetes / Advanced: extensibility, resilience, and lifecycle / Lesson 13

Controllers, custom resources, and Operators

Design convergent control loops with explicit schema, ownership, status, and cleanup rather than treating an Operator as automatic application expertise.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Explain the model, assumptions, and boundaries of controllers, custom resources, and operators.
Interpret both examples and distinguish declared intent from established evidence.
Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.

Extend the API with a contract

CustomResourceDefinitions add typed resources to the API; controllers implement behavior around them. Defining a resource does not by itself run reconciliation. An Operator packages operational logic for an application domain, but its guarantees depend on that logic and the underlying systems. The first schema fragment uses the stable apiextensions.k8s.io/v1 API and is deliberately incomplete. Validation should reject unsupported inputs before they trigger expensive or destructive work. Plan defaulting, versioning, conversion, and compatibility before evolving a custom resource. Do not expose an arbitrary shell command as an innocuous configuration field.

yaml Illustrative CRD schema excerpt

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
# Non-runnable excerpt: metadata, names, and version wrapper omitted.
spec:
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [replicas]
              properties:
                replicas: {type: integer, minimum: 0}

This incomplete excerpt demonstrates a nonnegative replica contract. It cannot be submitted as a complete CRD and supplies no controller or behavior.

Make reconcile safe to repeat

A reconcile invocation can be repeated, delayed, interrupted, or run after observations changed. Compute the desired action from current evidence and make external operations idempotent where possible. The second simulation keys a synthetic external allocation by object UID and returns the same result on repetition. Real APIs need durable ownership, failure handling, and reconciliation after uncertain timeouts. A successful network call followed by a crash before status update must not cause duplicate resources on the next loop. Avoid assuming one event maps to one invocation or that queue delivery is an exactly-once transaction.

python Harmless repeat-reconcile model

allocations = {}
def ensure(uid):
    if uid not in allocations:
        allocations[uid] = "lab-resource-" + str(len(allocations) + 1)
    return allocations[uid]
print(ensure("lab-uid-a"), ensure("lab-uid-a"))
print(len(allocations))

Expected output

lab-resource-1 lab-resource-1
1

Repeated calls for the synthetic UID return one allocation. The dictionary is not durable state and models neither distributed locking nor a real provider API.

Report status without creating a fight

Use spec for requested behavior and status for observations according to the resource schema and subresource configuration. Conditions should express meaningful states with enough generation context to distinguish stale evidence. A status update must not erase a user’s new intent. Handle optimistic-concurrency conflicts by reading current state and recomputing rather than repeatedly writing a stale object. Bound work per loop, use appropriate retry backoff, and avoid hot loops that overload the API server. Multiple controllers need explicit field and resource ownership, since two well-intentioned reconcilers can continuously undo each other’s changes.

Treat finalizers as cleanup obligations

A finalizer delays deletion until responsible cleanup is completed and the finalizer removed. It does not prove that cleanup will succeed or make the object immortal by design. Register cleanup ownership before creating external effects where the protocol requires it, and reconcile uncertain outcomes. A missing controller, revoked credential, or unavailable provider can leave deletion pending. Blindly removing the finalizer may orphan or leak external resources and bypass required retention. Provide an accountable recovery procedure with evidence and scope. Garbage collection and owner references also have scope and lifecycle rules that must be read before assuming automatic deletion is safe.

Test failure transitions, not only the happy loop

Useful tests cover invalid input, repeated reconciliation, conflict, partial external success, deletion, unavailable dependencies, and controller restart. Separate pure planning from effectful clients so local simulations can verify decisions without a cluster. Integration and scale behavior still require an authorized isolated environment. Review the Operator’s permissions, upgrade path, storage behavior, and application-specific recovery evidence rather than trusting its name. No API extension, controller process, finalizer mutation, or external resource is created here. A dependable extension narrows uncertainty and makes ownership visible; it does not eliminate the application’s responsibility for consistency and recovery.

Common errors and fixes

Reconcile duplicates external objects

Why it happens: Retries lacked durable idempotency and ownership.
How to fix it: Reconcile external identity and uncertain outcomes explicitly.
Prevent it next time: Test crash after external success before status update.

Deletion remains pending forever

Why it happens: A finalizer’s owner cannot complete cleanup.
How to fix it: Investigate ownership and use a scoped recovery procedure.
Prevent it next time: Document cleanup failure and controller availability requirements.

A controller overwrites newer intent

Why it happens: A stale object was written after a conflict.
How to fix it: Read current state and recompute the intended patch.
Prevent it next time: Separate spec and status ownership and test conflicts.

Check your understanding

1. An external create succeeds, then the controller crashes before status. What must the next loop do?

Show worked solution

Discover or reuse the external object through a durable ownership or idempotency key instead of creating another blindly. Verify actual state and update observations. A local boolean written after creation is not sufficient if the crash prevented persistence.

2. Why is removing a finalizer not a routine fix for a stuck object?