KBY Technologies Logo
runbooks/dns-certificate-lifecycle-security/nginx-continues-serving-pre-renewal-tls-certificate-after-renewal.md
CRITICAL SEVERITYDNS & Certificate Lifecycle SecurityTLS

NGINX Continues Serving a Pre-Renewal TLS Certificate After Renewal

Severity
CRITICAL
Target Time
30m
DomainDNS & Certificate Lifecycle Security
Verified
Overview

Confirm that automated certificate renewal succeeded on disk, validate NGINX configuration, and reload it safely so the renewed TLS certificate is served.

Share

Destructive Operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

01 // Diagnose

Symptom

Incident signalWhat responders observe

Clients receive the pre-renewal or expired certificate for an HTTPS hostname even though the ACME client reports that a newer certificate was successfully installed at the configured live path.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. From an external network, run openssl s_client -connect <HOSTNAME>:443 -servername <HOSTNAME> </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -serial -dates -fingerprint -sha256 and record the served certificate.

  2. On the server, run sudo nginx -T and identify the exact ssl_certificate and ssl_certificate_key paths for the matching server_name without copying key material.

  3. Run sudo certbot certificates and compare the intended lineage, domains, expiry, and live certificate path.

  4. Compare the served and on-disk certificate SHA-256 fingerprints.

  5. Check renewal and NGINX logs for a successful issuance followed by a failed deploy hook, configuration test, or reload.

  6. Verify DNS sends the affected hostname to this load balancer or server and check every active endpoint when traffic is distributed.

Root Cause Analysis

Causal chainWhy the incident occurred

  1. A renewed certificate exists at the NGINX-configured path, but the running NGINX workers still hold the pre-renewal certificate because the renewal deploy hook did not complete a successful reload.

  2. Renewal on disk and activation in the running service are separate operational steps.

02 // Contain & Prevent

Blast Radius

  • Browsers, APIs, agents, and service-to-service clients can reject the TLS connection.

  • Only some users may fail when one node or load balancer in a pool serves the expired certificate.

  • An incorrect certificate-path change can break NGINX reload or expose the wrong hostname certificate.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Run a post-renewal configuration test and controlled NGINX reload through the ACME deploy hook.

  • Monitor the certificate served externally at every endpoint rather than only the local certificate file.

  • Alert well before expiry and after any mismatch between served and expected fingerprints.

  • Keep certificate lineage and server_name ownership documented and reviewed.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Confirm the current hostname and server identity, prove this is an endpoint that serves <HOSTNAME>, and identify every other endpoint serving it.

  2. Run sudo -v and sudo -l, then confirm the listed policy explicitly authorizes the approved absolute NGINX binary for configuration testing and reload on this instance.

  3. Record the currently served certificate fingerprint and use sudo nginx -T to resolve the exact certificate path for the matching server_name.

  4. Calculate and record the on-disk fingerprint at that exact configured path; hard-stop unless it differs from the served fingerprint, covers <HOSTNAME>, has a later valid expiry, and matches its configured private key.

  5. Confirm the ACME account, renewal configuration, challenge path, and intended certificate lineage.

  6. Confirm a previous still-valid certificate or alternate healthy endpoint exists for rollback; never plan to restore an expired certificate.

  7. Obtain incident commander approval and require a fresh successful sudo nginx -t immediately before reload; abort without reloading if that test fails.

Execution CommandsCOMMANDS

set -e; sudo systemctl status nginx --no-pager; SERVED_CERT=$(mktemp); trap 'rm -f "$SERVED_CERT"' EXIT; openssl s_client -connect <HOSTNAME>:443 -servername <HOSTNAME> -showcerts </dev/null > "$SERVED_CERT" 2>/dev/null; SERVED_FP=$(openssl x509 -in "$SERVED_CERT" -noout -fingerprint -sha256); DISK_FP=$(sudo openssl x509 -in <NGINX_CERTIFICATE_PATH> -noout -fingerprint -sha256); printf 'served=%s
on_disk=%s
' "$SERVED_FP" "$DISK_FP"; test -n "$SERVED_FP"; test -n "$DISK_FP"; if test "$SERVED_FP" = "$DISK_FP"; then echo 'ABORT: served and on-disk fingerprints are identical'; exit 1; fi; sudo nginx -t; sudo nginx -s reload; openssl s_client -connect <HOSTNAME>:443 -servername <HOSTNAME> </dev/null 2>/dev/null | openssl x509 -noout -serial -dates -fingerprint -sha256

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. Repeat the external OpenSSL check and confirm the served fingerprint, subject alternative names, issuer, notBefore, and notAfter match the intended renewed certificate.

  2. Test the hostname from at least two independent network paths and every load-balanced endpoint.

  3. Confirm sudo nginx -t still succeeds and NGINX logs contain no reload or key-loading error.

  4. Confirm the application returns its expected HTTPS health response.

  5. Monitor handshake failures and certificate-expiry alerts for at least 15 minutes.

Rollback Protocol

Safe reversal path

  1. If NGINX cannot load the renewed certificate, do not restore an expired certificate.

  2. Keep or return traffic to the previously captured still-valid certificate or alternate healthy endpoint, restore the captured NGINX certificate paths if they referenced that valid certificate, run sudo nginx -t, reload, and escalate the issuance or key-pair problem.

  3. If no valid certificate remains, keep the affected endpoint out of service rather than bypassing TLS validation.

Escalation

Conditions requiring additional ownership

  • The certificate private key does not match the renewed certificate.

  • DNS or a load balancer directs traffic to an unmanaged endpoint.

  • ACME validation cannot complete before the recovery objective.

  • The only available rollback certificate is expired, revoked, compromised, or does not cover the hostname.

Authoritative Sources

Authoritative · NGINX

NGINX command-line parameters

NGINX configuration testing and graceful configuration reload behavior.

Authoritative · NGINX

Configuring HTTPS servers

NGINX TLS certificate and private-key configuration paths and HTTPS server behavior.

Authoritative · Internet Security Research Group

Let's Encrypt certificate profiles

Certificate validity periods, identifier coverage, and certificate profile behavior.

Authoritative · Electronic Frontier Foundation

Certbot: Renewing certificates

Certbot renewal behavior, certificate lineages, renewal hooks, and deploy hooks.

Authoritative · OpenSSL Project

openssl-s_client

Connecting to a TLS endpoint with an explicit server name to inspect the certificate served over the network.

Authoritative · OpenSSL Project

openssl-x509

Printing certificate serial numbers, validity dates, subject names, and fingerprints for deterministic comparison.