DNS Resilience Defaults Silently Override Expected Safeguards

Symptom

During a planned test of a secondary DNS path, client resolution continued to fail for several minutes after the primary resolver was confirmed reachable again. Applications reported intermittent NXDOMAIN and SERVFAIL responses even though direct queries against the authoritative and secondary resolvers returned correct answers immediately. The failure was inconsistent across hosts: some recovered within seconds, others took the full negative-cache lifetime to recover.

False Assumption

The operational assumption was that configuring a secondary resolver in /etc/resolv.conf or an equivalent stub-resolver list would cause clients to fail over promptly whenever the primary resolver became unreachable or returned an error, and that recovery would be equally prompt once the primary resolver was healthy again. This assumption treated resolver failover as symmetric: fast to fail away, fast to fail back. In practice, most stub resolvers and caching resolvers apply negative-caching TTLs and retry backoff independently of upstream health, and those defaults are not part of the resilience design that was reviewed.

Root Cause

The resolver’s negative-caching default holds a failed answer (NXDOMAIN or SERVFAIL) for a fixed period defined by the SOA record’s negative-caching TTL, or a resolver-specific floor when that field is absent or unusually low. This TTL is independent of whichever upstream resolver produced the failure and independent of whether an alternate, healthy resolver is available. Once a stub or caching resolver caches a negative result, it will continue returning that cached failure to the application until the TTL expires, even though a fresh query to a different, healthy resolver would succeed immediately. The resilience design assumed failover behaviour at the transport layer (which resolver is queried) without accounting for the caching layer (what answer is retained and served regardless of source).

Impact

Applications experienced extended, non-deterministic downtime windows bounded by the negative-cache TTL rather than by the actual outage duration, meaning the measured recovery time did not match the intended failover design. Because the effect was inconsistent across hosts depending on when each host’s cache entry was populated, the incident initially appeared to be a partial or flaky network problem rather than a deterministic caching effect, which extended diagnosis time and reduced confidence in the DNS resilience design during a live test window.

Diagnosis

Confirm the failure is a caching artefact rather than a live resolution problem by comparing a direct query against the authoritative or secondary resolver (bypassing the local cache) with a query through the affected client’s normal resolution path. If the direct query succeeds while the client-path query still fails, the local resolver or stub cache is serving a stale negative answer. Check the SOA negative-caching TTL for the zone and the resolver’s own negative-cache floor configuration, then compare that value against the observed recovery delay.

Correction

The correction is to align the negative-caching TTL with the actual recovery-time objective of the resilience design, and to make cache-clearing an explicit, tested step in any documented failover-recovery procedure rather than relying on the cache to expire naturally. Where the resolver software supports it, reduce the negative-caching ceiling to a value consistent with acceptable outage exposure, and treat clearing the negative cache after confirmed recovery as a normal recovery action rather than an emergency workaround.

Validation

Validate the fix by repeating the controlled failover test: intentionally make the primary resolver unreachable, allow one negative answer to cache, restore the primary resolver, and confirm that resolution recovers within the expected recovery-time objective without manual cache clearing. Cross-check by querying the authoritative or secondary resolver directly at the same moment to confirm the underlying data was already correct while the client path lagged.

Rollback

If a reduced negative-caching ceiling causes excessive query load on the authoritative or upstream infrastructure during normal operation, revert the resolver’s negative-cache floor to its previous value and reintroduce cache clearing as a manual step in the failover-recovery runbook instead. Any configuration change should be applied to a single non-production resolver instance first, with the previous configuration file retained for immediate restoration.

Prevention

Document the negative-caching TTL and any resolver-specific floor as an explicit, visible parameter in the DNS resilience design, alongside the failover trigger conditions, so that recovery-time expectations account for both layers. Include a cache-clearing or cache-bypass verification step in every failover test and in the production runbook, and re-verify the negative-caching configuration whenever the resolver software or its defaults change.