Incident Overview
KBY Technologies runs a 12-node Apache Cassandra 4.1 cluster (RF=3, vnodes, STCS) on i3en.2xlarge instances in AWS, backing an events-ingestion service handling roughly 500k writes/sec for a fintech telemetry platform. At 02:14 UTC, ops completed a scheduled scale-out ahead of Black Friday, bootstrapping three new nodes (13, 14, 15) via nodetool bootstrap.
Streaming finished cleanly per nodetool netstats, and the cluster showed 15 UN nodes. At 02:40 UTC, p99 read latency on the events table rose from a 45ms baseline to 310ms.
By 03:05 UTC, Grafana fired an alert showing client-side ReadTimeoutException rates up 8x (DataStax driver metrics). At 03:10 UTC, system.log on node14 began logging "Scanned over 100000 tombstones in keyspace1.events for query SELECT * FROM events WHERE partition_key = ?
AND user_id = ? - query aborted" roughly 40 times per minute. At 03:15 UTC, nodetool tpstats on nodes 13-15 showed ReadStage pending queues exceeding 200, with G1GC pause logs averaging 850ms versus a 120ms baseline.
At 03:20 UTC, nodetool compactionstats reported pending compactions of 180, 210 and 195 on nodes 13, 14 and 15 respectively, while nodes 1-12 remained under 10. CPU on the new nodes sat at 92%; existing nodes remained near 55%.
Two hypotheses compete. First, tombstone-heavy partitions—likely from TTL-expired events or frequent deletes—are exceeding the tombstone_failure_threshold on read scans, and this has only now become visible because increased read fan-out is hitting the new replicas.
Second, the new nodes inherited disproportionate token ranges or unmerged SSTables from bootstrap streaming, producing a compaction backlog that drives GC pressure and ReadStage queuing, which then surfaces as tombstone-scan timeouts because reads are aborting mid-scan on overloaded replicas rather than because of genuinely pathological partitions. Operational constraints are tight: the cluster cannot be taken offline, the write-availability SLA is 99.95%, a change freeze begins in 96 hours prohibiting further large topology changes, and no additional engineers are available for the next four hours.
Investigation Options
Review the available operational moves and select the best immediate action.
Correlate nodetool compactionstats, cfstats and tpstats across all 15 nodes with the GC pause timeline to establish whether tombstone scan aborts are concentrated on the three new nodes specifically, before making any cluster-wide compaction or streaming throughput changes.
Immediately identify the hot partition from the logged query pattern, run nodetool tablehistograms and cqlsh tracing against it, and switch the events table to TimeWindowCompactionStrategy without first checking whether node health metrics differ between old and new nodes.
Restart the three new Cassandra nodes to force garbage collection and interrupt the compaction backlog, expecting the read latency spike to resolve immediately given the limited time before the Black Friday change freeze begins.
Decommission the three newly added nodes and revert the cluster to its prior 12-node topology, treating the capacity increase itself as the root cause without further correlating compaction, GC or tombstone data first.