MariaDB / Editorial solution guide

Diagnose replica lag: retrieval or application

Separate transport failures, stopped apply and resource-limited catch-up using read-only evidence. Do not skip events or reconfigure replication to hide the symptom.

About 3 min readContent reviewed

Read-only knowledge base ยท No public comments

Recognize the symptoms

A report on a replica is stale, Seconds_Behind_Master grows or becomes NULL, or monitoring reports a stopped replication thread. A zero lag value alone does not prove the replica contains the newest committed primary transaction, especially after restart, idling or an interrupted connection.

Understand the likely causes

  • Retrieval can fail because of endpoint, authentication, TLS or missing binary logs. Application can stop on divergent data or schema errors, or run slowly because of long transactions, locks and limited storage or CPU. An intentional delayed-replica policy can also explain stale data.

Capture a complete channel snapshot

Use an existing authorized monitoring session on the intended replica. SHOW REPLICA STATUS is available on modern releases; SHOW SLAVE STATUS remains familiar on older versions. Select the actual connection in multi-source deployments and record timestamps, server version and all relevant fields privately. Field names and monitoring privileges differ by release. The examples do not connect to a configured database and no status output is claimed.

sql Illustrative SQL; expected results are reasoned, not measured. 
SELECT VERSION(), @@hostname;
SHOW REPLICA STATUS;
SELECT @@global.gtid_slave_pos, @@global.gtid_binlog_pos;

Read-only diagnostics. Treat hostnames, source addresses and error text as operational information.

Diagnose the retrieval branch

When Slave_IO_Running is No or Connecting, inspect Last_IO_Errno and Last_IO_Error with the configured primary endpoint and retrieved log position. Distinguish a refused connection from rejected credentials or an unavailable required binary-log file. Compare existing network and certificate evidence through the established operations process. The contextual remedy may be correcting approved endpoint or credentials, restoring archived history or rebuilding consistently; repeatedly restarting the thread cannot manufacture a purged event.

Diagnose the apply branch

If retrieval is healthy but Slave_SQL_Running is No, preserve Last_SQL_Errno and Last_SQL_Error, relay position and failing object. Duplicate-key or missing-row errors can indicate local replica writes or inconsistent provisioning. Use the reviewed data-consistency runbook; do not blindly skip a transaction. If application is running but slow, compare received and applied progress over several timestamps, inspect long-running work and correlate storage and CPU saturation. A large incoming transaction can create temporary catch-up delay.

sql Illustrative SQL; expected results are reasoned, not measured. 
SHOW FULL PROCESSLIST;
SHOW GLOBAL STATUS LIKE 'Threads_running';

PROCESS privilege affects visibility. Query text may be sensitive; inspect it privately rather than publishing a full process list.

Resolve the cause and protect read semantics

Seconds_Behind_Master summarizes a version-dependent timing condition, not a complete consistency proof. Check configured delay, thread states, errors and progress together. MariaDB GTIDs contain domain, server and sequence components and must be interpreted per replication domain; they are not MySQL GTIDs or one globally comparable integer. Route freshness-sensitive requests according to the service policy while remediation proceeds. Capacity or topology changes belong in a separate reviewed procedure with a consistent source and rollback evidence.

Verify the fix

Take another timed status sample after the authorized remedy. Confirm both retrieval and apply are healthy, required history is advancing without new errors and backlog is shrinking or within the documented bound. Verify a known existing application transaction or sanctioned heartbeat using the service freshness policy. Avoid creating an ad hoc production write solely to make monitoring green.

Keep it from recurring

  • Monitor retrieval errors, apply errors, backlog trend and end-to-end freshness separately. Protect sufficient binary-log retention, prevent ordinary local replica writes and test recovery from a consistent source. Include intentional delay and clock assumptions in alert rules so expected behavior is not mistaken for an outage.

Official references