Building a Distributed TLS Expiry Scanner

A single openssl s_client cron job checking one endpoint scales fine until your inventory crosses roughly 2,000 TLS-terminating hosts. Beyond that threshold, sequential handshakes take longer than the scan interval itself, private CA roots start throwing false positives, and nobody notices a wildcard cert expiring on a load balancer that isn’t in the CMDB. A production-grade TLS expiry scanner is not a cron job with a grep pattern — it is a distributed system with its own rate-limiting, deduplication, and trust-store problems. This article walks through the architecture of a horizontally-scalable TLS expiry scanner built for fleets in the tens of thousands of endpoints.

#The Problem With Naive Expiry Checks

The obvious first implementation is a shell loop over a host list, piping openssl s_client output into grep 'notAfter'. This breaks down for several concrete reasons:

• Serial handshake latency — TLS 1.3 full handshakes average 40–120ms depending on cipher negotiation and network RTT. At 10,000 hosts, a serial scan takes over 20 minutes per pass, which is unacceptable for a 15-minute alerting SLO.
• SNI ambiguity — a single IP can front dozens of virtual hosts via SNI. Checking the IP without the correct ServerName returns the default vhost certificate, not the one your application actually presents.
• Private CA blind spots — internal services signed by an internal root CA will fail standard trust validation unless the scanner’s trust store is explicitly extended, generating noisy false-positive alerts that erode trust in the tool.
• OCSP and CRL drift — expiry date alone doesn’t tell you whether the certificate has been revoked. A cert with 60 days left but a revoked status via RFC 6960 OCSP is a live incident, not a future one.

A TLS expiry scanner built for real infrastructure has to solve concurrency, trust-store management, and revocation checking simultaneously — while staying polite enough not to trip WAF rate-limiting on the very endpoints it’s monitoring.

#Architectural Breakdown

The design splits into four logical planes: discovery, scheduling, execution, and state reconciliation. Discovery pulls the target list from a source of truth (DNS zone files, cloud provider tag APIs, or a CMDB export). Scheduling fans targets into a work queue with jittered intervals so scans don’t synchronise into a thundering-herd handshake burst. Execution workers perform the actual TLS dial, extract the certificate chain, and evaluate expiry plus OCSP status. State reconciliation deduplicates repeated failures so a flapping certificate doesn’t page on-call every five minutes.

The queue layer matters more than most teams expect. Using Redis Streams with consumer groups rather than a flat list allows horizontal worker scaling without duplicate scans — each worker acknowledges (XACK) its target after completion, and unacknowledged entries are automatically redelivered after a configurable claim timeout, handling worker crashes without a coordination service.

#Rate-Limiting the TLS Expiry Scanner Against WAF Detection

Scanning at volume from a small number of source IPs looks identical to a TLS reconnaissance sweep to any competent WAF or IDS. The scanner therefore needs a token-bucket limiter per destination CIDR block, not a global limiter, so it doesn’t inadvertently trip rate-based blocking on shared load balancer infrastructure. A sensible default is 5 handshakes per second per /24, tunable per environment via config rather than hardcoded.

#Implementation Logic

The execution loop for each worker follows a fixed sequence:

• Pop a target (host, port, SNI hint) from the consumer group.
• Establish a TLS connection with InsecureSkipVerify disabled but a custom RootCAs pool that merges the system trust store with internal CA bundles.
• Walk the returned certificate chain, extracting NotAfter, issuer, SAN list, and the OCSP responder URL from the leaf’s AIA extension.
• Fire an OCSP request against the responder if present; fall back to CRL distribution point parsing if the responder is unreachable.
• Compute days-until-expiry and compare against tiered thresholds (30/14/7/1 days).
• Write the result into the state store keyed by host:port:sni, with a TTL slightly longer than the scan interval so stale entries expire naturally.
• Emit a Prometheus gauge and, if a threshold is breached and the previous state wasn’t already alerting, push to the webhook queue.

#Go Worker: Chain Extraction and Threshold Evaluation

1func scanTarget(host string, port int, sni string, roots *x509.CertPool) (*ScanResult, error) {
2    conf := &tls.Config{
3        ServerName: sni,
4        RootCAs:    roots,
5        MinVersion: tls.VersionTLS12,
6    }
7    dialer := &net.Dialer{Timeout: 4 * time.Second}
8    conn, err := tls.DialWithDialer(dialer, "tcp", fmt.Sprintf("%s:%d", host, port), conf)
9    if err != nil {
10        return nil, fmt.Errorf("tls dial failed: %w", err)
11    }
12    defer conn.Close()
13
14    chain := conn.ConnectionState().PeerCertificates
15    if len(chain) == 0 {
16        return nil, errors.New("empty certificate chain")
17    }
18    leaf := chain[0]
19    daysLeft := int(time.Until(leaf.NotAfter).Hours() / 24)
20
21    return &ScanResult{
22        Host:      host,
23        SNI:       sni,
24        Issuer:    leaf.Issuer.CommonName,
25        NotAfter:  leaf.NotAfter,
26        DaysLeft:  daysLeft,
27        OCSPURLs:  leaf.OCSPServer,
28        SANs:      leaf.DNSNames,
29    }, nil
30}

#Scanner Fleet Configuration

1scanner:
2  concurrency_per_worker: 40
3  claim_timeout_seconds: 30
4  redis_stream: tls_scan_targets
5  redis_consumer_group: scanner_workers
6  rate_limit:
7    per_cidr: "5/s"
8    cidr_mask: 24
9  trust_stores:
10    - /etc/ssl/certs/ca-bundle.crt
11    - /etc/scanner/internal-ca-bundle.pem
12  thresholds_days:
13    warning: 30
14    critical: 14
15    page: 7
16  ocsp:
17    enabled: true
18    timeout_ms: 800
19    fallback_to_crl: true

#OCSP Verification via OpenSSL for Manual Spot Checks

1openssl s_client -connect edge-lb-03.internal:443 -servername app.example.com -status /dev/null | 
2  openssl x509 -noout -dates -issuer
3
4# Explicit OCSP responder query
5openssl ocsp -issuer chain.pem -cert leaf.pem 
6  -url $(openssl x509 -in leaf.pem -noout -ocsp_uri) 
7  -text -no_nonce

These manual commands double as the exact validation path used inside the Go worker, which keeps the automated scanner and manual debugging consistent — a debugging session against production shouldn’t require a different toolchain than the scanner itself uses.

#Failure Modes and Edge Cases

A TLS expiry scanner deployed at fleet scale surfaces failure classes that never show up in a single-host test. The table below captures the diagnostic signatures worth building explicit handling for.

The rate-limit and OCSP timeout cases interact in an unpleasant way: if the token bucket is too conservative, OCSP checks queue up behind rate-limited handshakes and the scan cycle overruns its interval, causing the next cycle’s targets to be enqueued before the previous cycle finishes. This is the most common cause of unbounded queue growth in production deployments, and it is why claim_timeout_seconds needs to be shorter than the scan interval, not longer.

#Scaling and Security Trade-offs

Operating a TLS expiry scanner at fleet scale forces a set of explicit trade-offs between coverage, blast radius, and operational overhead:

• Centralised vs. distributed scanning nodes — a single scanning cluster is simpler to secure and audit, but any egress ACL misconfiguration blinds the entire fleet at once; distributing scanners per network segment adds resilience at the cost of certificate bundle sprawl.
• Active handshakes vs. passive TLS metadata capture — capturing certificate metadata from existing mTLS
  The KBY Lexicon

  mTLS (Mutual Transport Layer Security)

  mTLS is a TLS handshake extension in which both the client and server present X.509 certificates to cryptographically authenticate each other before establishing an encrypted channel, rather than only the server proving its identity as in standard TLS. It forms the cryptographic identity layer underpinning zero-trust network architectures, replacing implicit network-location trust with explicit, per-connection workload authentication.

  Read Full Context
  proxy logs (e.g. Envoy access logs) avoids generating new handshake traffic entirely, but only covers traffic that’s actually flowing, missing cold or low-traffic endpoints.
• Trust bundle staleness — pinning the scanner’s CA trust store to a fixed image build simplifies reproducibility but risks false positives when an internal CA rotates; a sidecar that pulls the bundle on a schedule adds a small supply-chain surface in exchange for accuracy.
• Credential exposure for internal endpoints — if the scanner needs client certificates to reach mTLS-only services, those credentials become a high-value target; short-lived certificates issued via a workload identity system reduce the blast radius versus a long-lived static client cert.
• Alert fatigue vs. detection latency — a 15-minute scan interval catches revocation events quickly but multiplies handshake volume by 4x compared to an hourly interval; most fleets land on a tiered approach, scanning customer-facing edge certificates every 15 minutes and internal service certificates hourly.

These trade-offs are the same ones that show up when designing any monitoring subsystem that has to touch production traffic paths to gather its own signal — the scanner is, in effect, a synthetic client, and it inherits every constraint a real client would face plus the added burden of not being allowed to degrade the service it’s checking. Teams building this as part of a broader observability stack should treat trust-store management and rate-limit configuration as first-class architectural patterns rather than deployment afterthoughts, since both directly determine whether the scanner produces signal or noise at scale. Get the queue semantics and OCSP fallback logic right once, and the same scanner design scales cleanly from a few hundred endpoints to a multi-region fleet without a rewrite.