Proxmox VE / Advanced: coordination and recovery evidence / Lesson 15

High availability and fencing

Understand controlled guest recovery, exclusive ownership and failure capacity without promising continuous service from an HA label.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate restart-based recovery from application-level availability.
Explain fencing as protection against concurrent writers.
Assess placement rules and survivor capacity together.

Define the availability claim

Proxmox HA can coordinate recovery of managed guests when supported failure conditions and prerequisites are met. Restarting a guest elsewhere still includes detection, ownership resolution, startup and application recovery time. It does not turn a single-instance application into a continuously serving distributed system. Define the user-visible outage target and the data-loss tolerance separately. A guest marked HA can remain unavailable if no suitable node has capacity or data access. The first diagram identifies the recovery stages without assigning universal timing guarantees or triggering a failure.

text HA recovery is a sequence of claims

failure observation -> safe ownership decision -> suitable recovery placement
  -> guest start -> filesystem/application recovery -> user acceptance
separate prerequisites: quorum, fencing, storage access, spare capacity

The sequence is conceptual. It does not promise that any specific environment can complete every stage.

Understand exclusive ownership

If a node becomes unreachable but still writes shared storage, starting the same guest on another node can create conflicting writers. Fencing resolves that uncertainty by ensuring the former execution owner cannot continue in the unsafe state. Proxmox HA uses watchdog-related mechanisms in its supported design, with hardware and software assumptions requiring careful review. Fencing is not a cosmetic status check and should never be disabled casually to speed recovery. This lesson describes the invariant only and performs no watchdog access, node reset, service stop or HA configuration change.

Budget for surviving nodes

Failure shifts workloads onto survivors, so spare capacity must exist where placement rules permit it. The second program uses three synthetic nodes with 16GiB each and 30GiB of guest demand. Losing one leaves 32GiB gross capacity before any host reserve or topology constraints. That small arithmetic margin is not a proof of recoverability. Real placement also depends on CPU, storage I/O, network, reservations and application peaks. A cluster that looks comfortable in aggregate can overload one eligible survivor if affinity rules or local data restrict the recovery choices.

python Gross survivor-capacity arithmetic

nodes = 3
gross_gib_per_node = 16
guest_demand_gib = 30
after_one_failure = (nodes - 1) * gross_gib_per_node
print(after_one_failure)
print(after_one_failure - guest_demand_gib)

Expected output

32
2

32 and 2 omit host reserve and all other constraints. They are not HA admission or sizing recommendations.

Review placement and dependencies

Placement preferences and affinity rules can keep related workloads together or apart, but they must leave feasible recovery targets under the intended failure scenario. Keeping two application replicas on different nodes is useful only if shared power or storage does not defeat the failure independence. Conversely, requiring several large guests on one surviving node can make recovery impossible. Storage replication introduces its own recovery-point considerations; shared storage introduces access and failure-domain requirements. Read the selected release’s placement model because configuration concepts evolve, and avoid copying an old group-based example into a different release without review.

Validate the service contract

An HA review includes membership, fencing assumptions, disk access, survivor capacity, placement feasibility and application startup. An eventual guest-running status is weaker than user-service recovery and can hide corruption or missing dependencies. Define application acceptance and record observed timing under an explicitly authorized isolated scenario. This course runs no failover, simulates no actual node outage and touches no cluster service. Its capacity program proves subtraction only. The learning result is an evidence map that explains which prerequisites must be established before an HA claim becomes meaningful for a particular workload.

Common errors and fixes

HA is advertised as zero downtime for a single guest

Why it happens: Restart recovery was confused with continuous application replication.
How to fix it: Define outage stages and application recovery evidence.
Prevent it next time: State availability guarantees at the user-service boundary.

An unreachable node is assumed unable to write storage

Why it happens: Network visibility replaced exclusive-ownership evidence.
How to fix it: Follow the supported fencing and ownership recovery design.
Prevent it next time: Never equate missing contact with confirmed safe shutdown.

Recovery has no eligible node with enough capacity

Why it happens: Steady aggregate capacity ignored failure placement constraints.
How to fix it: Review survivor budgets and feasible affinity choices.
Prevent it next time: Model required failures before admitting HA workloads.

Check your understanding

1. Why is fencing relevant even when a failed node cannot answer management requests?

Show worked solution

Management reachability does not prove execution or storage writes stopped. A partition can leave the guest active on shared data. Before recovery starts another writer, the supported design must establish exclusive ownership. This is a data-safety requirement, not an optional delay to remove when an incident feels urgent.

2. The gross survivor model leaves2GiB. Can the cluster be approved for this workload?