DNS Resilience: The DNS Setting That Fails Quietly

Symptom

A DNS zone that has been deliberately configured for resilience produces a complete resolution outage the first time its primary authoritative name server becomes unreachable. Before the outage, every internal test appeared to confirm redundancy: the secondary name server answered authoritative queries correctly when queried directly, zone transfers between primary and secondary completed without error, and the secondary held an up-to-date copy of every record. During a scheduled maintenance window on the primary server, external clients could no longer resolve the domain at all, even though the secondary was online, healthy and holding a complete, current zone.

False Assumption

The team assumed that once a secondary authoritative name server holds a correct, synchronised copy of the zone and answers queries correctly when tested directly, resolvers on the public internet will automatically use it if the primary becomes unavailable. This assumption treats zone-level replication (via AXFR/IXFR and matching SOA serials) as equivalent to delegation-level redundancy. It is not: a name server can be fully authoritative, fully synchronised and fully reachable, and still never be queried by a single production resolver if it has not been added to the delegation that resolvers actually follow.

Root Cause

The parent zone’s delegation – the NS record set held by the registry or parent zone and returned as a referral – had never been updated to include the secondary name server. The secondary’s hostname was added only to the NS records inside the child zone itself. Recursive resolvers performing an initial referral for the domain retrieve the NS set from the parent, not from the child’s own apex. Unless a resolver later requeries the child zone’s own NS records and treats that answer as authoritative for future referrals, it never learns that a second server exists. The result is two different NS record sets: one visible to parent-following resolvers, and a larger one visible only to anyone who queries the child zone directly.

Impact

Every recursive resolver that followed the standard referral chain from the parent zone continued sending queries only to the primary name server, so the moment that server stopped responding, resolution failed for the domain’s entire external audience regardless of the secondary’s health. Monitoring that queried the secondary directly, or tooling that queried the zone’s own apex NS set, continued to report a healthy, redundant configuration throughout the outage, which delayed diagnosis because the visible evidence contradicted the reported symptom.

Diagnosis

Confirm the mismatch by comparing the NS record set returned by the parent zone against the NS record set returned by the zone’s own apex, rather than trusting either source in isolation.

• Query the parent-delegated NS set from an external vantage point.
• Query the zone’s own apex NS set directly against its own name servers.
• Compare the two result sets; any name server present in the second list but absent from the first is not part of the effective delegation and will never receive production queries.
• Trace resolution from the root to confirm which servers a resolver actually reaches during a live referral.

Correction

Correct the trap by updating the domain’s delegation at the registrar or parent zone so that its NS record set includes every name server intended to serve production traffic, then verify the change from outside the organisation’s own infrastructure before relying on it.

• Submit a delegation update via the registrar or parent-zone control panel or API, adding the secondary hostname (and glue records if it is in-bailiwick) to the domain’s delegated NS set.
• Wait for propagation, bounded by the parent zone’s NS record TTL, before assuming the change is visible externally.
• Re-run the parent-versus-child NS comparison from diagnosis until both sets match exactly.
• Only after the sets match should the secondary be treated as load-bearing for resilience purposes.

Validation

Confirm the correction succeeded by repeating the parent-delegation query from a resolver outside the organisation’s own network and confirming both name servers now appear in the parent-returned NS set.

• Query an external, independent resolver and confirm both the primary and secondary appear in the parent-delegated NS answer.
• Confirm the zone apex NS answer matches the parent-delegated NS answer exactly.
• In a controlled, pre-announced maintenance window, take the primary offline and confirm representative records still resolve correctly using only the secondary before treating the fix as complete.

Rollback

If the delegation change causes unexpected resolution failures, revert the parent-zone NS record set to the previously verified configuration and restore the primary as the sole delegated authority until the discrepancy is resolved.

• Keep a documented snapshot of the pre-change NS RRset (both parent and child) captured before applying any delegation update.
• Revert the parent-zone delegation to that snapshot if the new configuration produces unexpected failures anywhere in the referral chain.
• Do not decommission or reduce reliance on the primary name server until the corrected delegation has been independently confirmed correct outside the organisation’s own network.

Prevention

Treat parent-zone delegation and child-zone NS records as two independent configurations that must be actively reconciled on every change, not a single setting maintained in one place. Add an automated comparison of parent-delegated NS records against the zone’s own apex NS records to routine DNS health checks, so a silent mismatch surfaces before it becomes an outage. Require that comparison as a mandatory, evidenced step before any DNS resilience change is certified complete, and design monitoring so that it follows the same referral path a production resolver would use, rather than querying a known-good server directly.