Reading SMART Attribute Thresholds Before Failure

A drive reporting SMART Health Status: PASSED tells you almost nothing about imminent failure. That single bit is the output of firmware-level SMART attribute thresholds comparing a handful of pre-fail counters against manufacturer-defined limits — limits set conservatively to avoid warranty churn, not to protect your dataset. By the time the overall-health self-assessment flips to FAILED, the window to migrate data off gracefully has usually already closed. For home NAS builders, workstation owners and students running research data on consumer HDDs and SSDs, the actionable signal lives in the raw attribute values and their trend, not the binary pass/fail flag most GUI utilities surface.

#The Structure Behind SMART Attribute Thresholds

Self-Monitoring, Analysis and Reporting Technology (SMART) is an ATA/ATAPI firmware feature, not an operating-system service. The drive controller maintains an internal table of up to 30 attributes, each identified by a numeric ID (0–255, with vendor-specific ranges overlapping heavily). Every row exposes ten columns — ID, Attribute Name, Flag, Value, Worst, Thresh, Type, Updated, When_Failed, Raw_Value — and understanding how firmware compares the normalised Value against Thresh is the entire basis of SMART attribute thresholds monitoring. The normalised value is a vendor-scaled figure (typically 1–253, higher is healthier); the raw value is the counter you actually care about, such as an uncorrected sector count or a temperature in degrees Celsius disguised inside a 48-bit field.

Querying a drive directly with smartmontools exposes the full table rather than the summarised pass/fail bit:

1$ sudo smartctl -a /dev/sda | grep -A20 "ID# ATTRIBUTE_NAME"
2ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  RAW_VALUE
3  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always   0
4  9 Power_On_Hours          0x0032   078   078   000    Old_age   Always   19842
5 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always   412
6187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always   0
7188 Command_Timeout         0x0032   100   099   000    Old_age   Always   0
8197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always   0
9198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Always   0
10199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always   0

#Pre-Fail vs Old-Age Attributes

Attributes flagged as Pre-fail (Reallocated_Sector_Ct, Spin_Up_Time, Seek_Error_Rate) carry SMART attribute thresholds that, once the normalised value drops to or below Thresh, trigger an immediate FAILED status. Old_age attributes (Power_On_Hours, Temperature, most CRC/timeout counters) never trip the pass/fail bit regardless of how bad the raw value gets — they are informational only. This is the root cause of drives showing PASSED with tens of thousands of UDMA CRC errors: the vendor firmware simply doesn’t wire that counter into the health verdict. Any monitoring built purely on the firmware’s own verdict inherits this blind spot.

#Implementation: Building a Monitoring Pipeline Around SMART Attribute Thresholds

Default SMART attribute thresholds set by drive vendors are tuned for warranty liability, not early warning. The practical fix is running smartd as a persistent daemon with attribute-specific overrides, then piping the raw values into a time-series store so trend, not point value, drives the alert.

1. Install smartmontools and enable the daemon at boot.
2. Define per-device policies in /etc/smartd.conf with explicit attribute tracking rather than relying on -a defaults.
3. Schedule short self-tests daily and extended (long) self-tests weekly during low-I/O windows.
4. Export raw attribute values via a textfile collector into Prometheus for delta-based alerting.

1# /etc/smartd.conf
2DEVICESCAN -a -o on -S on 
3  -s (S/../.././02|L/../../6/03) 
4  -W 4,45,55 
5  -C 5+ -C 197+ -C 198+ -U 199+ 
6  -m ops@example.internal 
7  -M exec /usr/local/bin/smart_notify.sh

The -C 5+, -C 197+ and -C 198+ directives tell smartd to email whenever the raw value of Reallocated_Sector_Ct, Current_Pending_Sector or Offline_Uncorrectable increases at all — bypassing the firmware’s normalised SMART attribute thresholds entirely and reacting to the counter itself. -U 199+ covers UDMA CRC errors, which almost always indicate a failing SATA cable or backplane connector rather than the platter, and is worth isolating from disk-health alerts.

1#!/usr/bin/env bash
2# smart_to_prometheus.sh — textfile collector for node_exporter
3OUT=/var/lib/node_exporter/textfile_collector/smart.prom
4: > "$OUT"
5for dev in /dev/sd?; do
6  json=$(smartctl -a -j "$dev" 2>/dev/null)
7  [ -z "$json" ] && continue
8  serial=$(echo "$json" | jq -r '.serial_number')
9  echo "$json" | jq -r '.ata_smart_attributes.table[] |
10    "smart_attribute_raw{device="'$dev'",serial="'$serial'",id="(.id)",name="(.name)"} (.raw.value)"' >> "$OUT"
11done

With raw values landing in Prometheus every five minutes, an alerting rule such as increase(smart_attribute_raw{id="5"}[7d]) > 0 catches sector reallocation growth long before the firmware’s own SMART attribute thresholds have any reason to intervene.

#Failure Modes and Edge Cases

Several practical scenarios break naive SMART attribute thresholds monitoring outright.

• Counter resets on power cycle: some Seagate firmwares zero Raw_Read_Error_Rate on spin-up, making week-over-week delta checks meaningless for that specific attribute on those families.
• Hardware RAID abstraction: controllers such as LSI/Broadcom MegaRAID or Dell PERC intercept ATA passthrough. smartctl against /dev/sda in these environments queries the virtual disk, not the physical spindle — you need storcli/perccli or SAS passthrough mode to reach real attribute tables.
• NVMe SSDs bypass the classic table entirely: there is no attribute ID/Value/Thresh structure. Health is exposed via the NVMe Health Information log page, read with nvme smart-log /dev/nvme0, using fields like percentage_used and media_errors instead.
• USB bridge chips drop SMART passthrough: many UAS/USB-SATA enclosures used in home NAS builds silently discard SMART commands, returning stale or fabricated data — verify with smartctl -i before trusting any subsequent read.
• Non-standard SSD wear IDs: attribute 177 (Wear_Leveling_Count) on Samsung maps roughly to attribute 173 on Micron/Crucial and 233 on some Intel drives — cross-vendor fleets need a lookup table, not a single hardcoded ID, when applying custom SMART attribute thresholds.

#Scaling and Security Trade-offs

Rolling this out across a fleet, or even a single home NAS with eight bays, introduces trade-offs that pure per-disk monitoring doesn’t surface. Choosing how aggressively to poll and how far to relax vendor-default SMART attribute thresholds has real operational cost:

• Polling frequency vs spin-down power management: frequent smartctl polling wakes drives configured for APM spin-down, defeating power-saving profiles common on home NAS builds — align poll intervals with existing spin-down timers.
• Self-test I/O contention: extended self-tests consume the full seek/read pipeline; scheduling them concurrently across a RAID array during a rebuild window compounds latency exactly when the array is least tolerant of it.
• Alert fatigue from Old_age counters: Temperature and Power_On_Hours change constantly and generate noise if wired into the same alerting path as Pre-fail attributes — separate the rule groups.
• Fingerprinting exposure: exporting raw SMART data (serial numbers, firmware revision) via an unauthenticated metrics endpoint leaks hardware identity useful for physical asset correlation; scope the node_exporter textfile endpoint behind the same network ACLs as other internal telemetry.
• False negatives on refurbished/relabelled drives: some resellers reset Power_On_Hours or Reallocated_Sector_Ct counters via vendor tools before resale — treat a suspiciously pristine attribute table on a “new” drive as a data point, not a guarantee.

None of this replaces a proper backup