Skip to main content
Systems Engineering

Diagnosing NVMe Write Amplification in Multi-Tenant SSD Deployments

Measuring write amplification on consumer NVMe drives with nvme-cli and fio, and why HMB-less SSDs burn through TBW ratings years early.

Diagnosing NVMe Write Amplification in Multi-Tenant SSD Deployments
KBY Technologies12 min read

In this guide

Share

A consumer NVMe drive rated for 600 TBW hitting its endurance ceiling in eighteen months instead of the projected five years is not necessarily a manufacturing defect. It is often a symptom of high write amplification at the flash translation layer, compounded by a filesystem and application stack that has no concept of how NAND actually works. This failure mode is easy to misdiagnose in home lab storage and student workstation builds, and is often blamed on “bad batches” of flash when the real cause may be a mismatch between logical write patterns and physical erase-block geometry.

This article breaks down how to measure write amplification on consumer NVMe hardware, why it can get worse on DRAM-less drives, and how to build a diagnostic pipeline using nvme-cli, fio, and kernel-level I/O accounting rather than relying on vendor dashboards that round everything to zero.

#The Physics Behind Write Amplification

NAND flash cannot perform in-place overwrites. A page can only be written after the entire block containing it has been erased, and erase operations happen at block granularity (a block contains many pages), not page granularity. When the host issues a small, misaligned 4KB write, the drive’s Flash Translation Layer (FTL) may need to read an entire block, merge the new data with the old, and rewrite the whole block elsewhere. That single logical write can trigger megabytes of physical NAND activity. This ratio of physical bytes written to NAND versus logical bytes requested by the host is the write amplification factor, and it is a primary driver of premature flash wear.

Write amplification is never zero, even on an empty drive, because of background garbage collection, wear-levelling redistribution, and metadata table updates. As a rule of thumb (not a documented vendor limit), a well-tuned enterprise NVMe deployment might see a WAF between 1.5 and 3.0 under mixed random workloads. Consumer drives under unmanaged conditions, particularly DRAM-less SKUs, may reach a much higher WAF during sustained 4K random write workloads, which can help explain why some cheap NVMe drives wear out well ahead of their rated endurance.

#
Quantifying Write Amplification Factor

WAF is calculated as:

1WAF = (Total NAND bytes written) / (Total Host bytes written)

The host-side figure is easy to obtain from the standard SMART/Health log or block layer statistics. NAND-side writes are available only from vendor-specific log pages or proprietary vendor attributes exposed through nvme-cli; the standard SMART/Health log (also called the Media and Data Integrity log) has no NAND-write counter. The NVMe Base Specification defines the standard SMART/Health log and only reserves space for vendor-specific pages, whose formats are in vendor documentation. Without correlating both sides, you are only measuring host I/O, which tells you nothing about actual flash wear.

#Architectural Breakdown: Where Amplification Actually Originates

There are four main amplification sources. They compound in workload-dependent ways rather than combining as simple independent multipliers, and diagnosing write amplification means isolating which layer is contributing:

  • Filesystem journaling — ext4 and XFS journal metadata before committing, meaning every metadata update generates a second physical write. Btrfs and ZFS add copy-on-write overhead on top of that, which under fragmented conditions can double logical write volume before it even reaches the block layer.
  • Alignment mismatch — partitions or LVM volumes not aligned to the drive’s erase block size force the FTL into read-modify-write cycles for every boundary-crossing operation.
  • Garbage collection pressure — as a drive fills up, the FTL has fewer free blocks to use as GC targets, forcing more aggressive compaction and increasing internal write amplification.
  • Absent or delayed TRIM — without fstrim or continuous discard, the controller believes deleted blocks are still live data, forcing it to preserve and relocate stale pages during wear-levelling instead of simply erasing them.

These effects compound. A drive at 85% capacity with a misaligned partition table and no TRIM scheduling can see considerably higher write amplification than a healthy one, though the combined figure depends on the workload and must be measured rather than assumed. This helps explain why identical hardware in two different deployments can show wildly different failure timelines. This is the same class of layered inefficiency you see in poorly designed architectural patterns where each abstraction layer independently re-solves a problem the layer below it already handled.

Diagnosing NVMe Write Amplification in Multi-Tenant SSD Deployments architecture diagram 1

#Implementation Logic: Building the Diagnostic Pipeline

You cannot manage what you cannot measure at the correct layer. The diagnostic pipeline requires three data points captured over an identical time window: host logical writes, controller-reported NAND writes, and filesystem-level write multiplication.

#
Step 1: Baseline Controller Statistics

Pull the SMART/health log and vendor-specific attributes before starting any workload. This is a read-only check:

sudo nvme smart-log /dev/nvme0n1 --output-format=json | jq '.data_units_written'

data_units_written counts units of 1000 512-byte blocks (512,000 bytes each). It is a host-side counter, so it gives you the host-written baseline only, not NAND writes.

The NAND-side write counter is vendor-specific: its name and availability depend on the controller vendor, and many consumer drives do not expose it at all. Where supported, use the matching nvme-cli vendor plugin (for example nvme micron vs-smart-add-log or nvme intel smart-log-add; nvme help lists the plugins your version includes) or the vendor’s documentation, and record that counter before the workload as well.

#
Step 2: Generate a Controlled Workload

Warning: the fio job below destroys all data on its target if pointed at a raw device. Back up anything you need, verify the device name with lsblk, and run it only against a dedicated empty drive or against a file on a mounted filesystem, as shown here.

Use fio to generate a deterministic 4K random write pattern that mimics database or VM disk activity, the workload class most sensitive to write amplification:

1[global]
2ioengine=libaio
3direct=1
4bs=4k
5rw=randwrite
6size=10G
7runtime=300
8time_based
9iodepth=32
10
11[job1]
12filename=/mnt/test/fio-testfile
13numjobs=1

Run this, capture the total bytes written by fio from its summary output, then immediately re-pull the SMART log (and the vendor NAND counter, if available):

sudo nvme smart-log /dev/nvme0n1 --output-format=json | jq '.data_units_written'

#
Step 3: Calculate and Correlate

Subtract the baseline from the post-workload figure for both counters over the same window. WAF is the NAND-side delta divided by the host-side delta. The script below captures the host counter around the fio run and takes the NAND-side delta from a vendor counter you supply in bytes; without that counter it cannot compute WAF.

1#!/bin/bash
2# waf_calc.sh — approximate write amplification factor
3NAND_BYTES=$1   # vendor-specific NAND bytes written during the run (delta)
4JOBFILE=$2      # fio job file
5BEFORE=$(sudo nvme smart-log /dev/nvme0n1 --output-format=json | jq '.data_units_written')
6fio "$JOBFILE"
7AFTER=$(sudo nvme smart-log /dev/nvme0n1 --output-format=json | jq '.data_units_written')
8HOST_BYTES=$(( (AFTER - BEFORE) * 512000 ))
9WAF=$(echo "scale=2; $NAND_BYTES / $HOST_BYTES" | bc)
10echo "Write Amplification Factor: $WAF"

As a rule of thumb (not a vendor-documented limit), a figure sustained above roughly 4.0 under a 4K random write test on a drive with headroom (under 70% capacity, TRIM enabled) suggests a structural problem rather than workload noise. The NVMe Base Specification defines the standard log pages; vendor-specific pages are only reserved there, so consult the drive vendor’s documentation to interpret their attributes.

#Failure Modes & Edge Cases

Several conditions cause write amplification measurements to mislead engineers who only look at surface-level metrics:

Diagnosing NVMe Write Amplification in Multi-Tenant SSD Deployments architecture diagram 2

DRAM-less controllers under sustained load. Drives lacking an onboard DRAM cache may use Host Memory Buffer (HMB) allocation from the OS to hold part of the FTL mapping table. If the OS denies or throttles the HMB allocation (for example under memory pressure or on certain hypervisor configurations), mapping-table handling is controller-specific and vendors generally do not document it. It can degrade performance and possibly increase write amplification, so compare behaviour with and without HMB on your own hardware.

TRIM gaps on layered storage. LVM, LUKS, and software RAID layers do not always propagate discard commands transparently. A LUKS volume without discard explicitly enabled in /etc/crypttab silently disables TRIM propagation, and the drive may accumulate stale pages, with write amplification climbing over weeks even though the filesystem itself reports free space correctly. Enabling discard on LUKS is a deliberate trade-off: it reveals which blocks are free, leaking information about filesystem usage patterns.

Thermal throttling masking the real signal. When a controller throttles due to thermal limits, it may also defer garbage collection, which can temporarily lower measured write amplification while building a backlog of stale blocks. WAF can then rise once thermal headroom returns and the controller catches up on deferred garbage collection, which is frequently misread as a sudden hardware fault rather than deferred maintenance.

Filesystem-level double accounting. On ZFS with default recordsize mismatched against workload I/O size (e.g. 128K recordsize serving a database issuing 8K random writes), read-modify-write at the ZFS layer inflates logical writes before they even reach the block device, meaning your baseline host-write figure is already amplified before NAND-level WAF is applied on top.

#Scaling & Security Trade-offs

Mitigating write amplification involves genuine trade-offs rather than free optimisations. Every technique below reduces flash wear at a measurable cost elsewhere:

  • Over-provisioning (reserving 10-20% of raw capacity unpartitioned) reduces write amplification substantially by giving the FTL more free blocks for garbage collection, at the direct cost of usable capacity.
  • Enabling continuous discard (discard mount option) keeps stale pages low but can add latency on some filesystems and drives, particularly metadata-heavy workloads; scheduled fstrim.timer weekly is the better trade-off for most desktop and lab use.
  • Self-encrypting drives (SEDs) with hardware AES-XTS introduce negligible write amplification overhead since encryption happens inline at the controller, whereas software-layer encryption (LUKS) can obscure discard propagation and indirectly worsen amplification unless explicitly configured.
  • Secure erase / sanitize (nvme sanitize) before redeployment irrecoverably erases all data on the device and cannot be undone. Only run it after verifying your backups restore correctly, and confirm the target device with lsblk and nvme list. Check nvme sanitize --help and the drive’s supported sanitize actions (nvme id-ctrl reports them) before choosing the action flag, for example --sanact=2 for block erase where supported. Do not expect it to fix write amplification; it only returns the drive to a clean state.
  • Aggressive wear-levelling algorithms distribute amplification evenly across all cells, extending median drive life but slightly increasing worst-case write amplification per operation compared to naive sequential allocation on a fresh drive.

Treat write amplification as a continuously monitored metric, not a one-time benchmark. A drive passing today’s WAF test at 60% capacity will behave completely differently at 90% capacity six months later, and the diagnostic pipeline above should be rerun after any significant change to partition layout, encryption configuration, or filesystem tuning. Pilot such changes on a single machine before rolling them out across a fleet.

Evidence trail

Sources and verification

Primary documentation and external technical references used in this article.

  1. 01NVMe Base Specificationnvmexpress.org

About this page

Credited to

Dates

Published

Last updated

Corrections

  • : Corrected during a review of published articles. Details
Reader Interaction

Comments

Add a thoughtful note on Diagnosing NVMe Write Amplification in Multi-Tenant SSD Deployments. Comments are checked for spam and held for moderation before appearing.

Loading comments...

Learn More About KBY

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

Engineering insights, direct to you.

Receive the latest Systems Engineering tutorials, production guides, Engineering Labs and operational best practices.