Software RAID and SMART checks
Software RAID with mdadm and SMART monitoring: two tools so a failing disk never catches you by surprise. You learn to keep the server running through a failure, then to see it coming.
- Course
- Tutorial 32 of 35
- Time
- 35 min to read
- Tested with
- Debian 13.7 (trixie), mdadm 4.4, smartmontools 7.4
What you will be able to do
- Say what RAID protects, and what it doesn't.
- Build a RAID 1 with mdadm, simulate a failure and rebuild, with no risk to your real disks.
- Get an email when a disk in the RAID fails.
- Read a disk's health with smartctl, schedule tests with smartd and know when to replace it.
Before you start: Read your logs with journalctl, Your first backup with rsync.
Contents
- What RAID protects, and what it doesn't
- RAID 1, RAID 5, RAID 6: which one to pick
- 1. Prepare fake disks
- 2. Create the RAID 1 and read it
- 3. Simulate a failure
- 4. Replace the disk and rebuild
- 5. Get an email when a disk fails
- 6. Clean up the experiment
- 7. Read a disk's health with smartctl
- 8. Run a short or long test
- 9. Let smartd keep watch
- When to replace a disk
- If it fails
- Going further
- What's next
- Sources
What RAID protects, and what it doesn't
RAID protects against one thing: a disk failure. A hard disk is a part that spins and wears out, so one day it will break. RAID groups several disks so that the failure of one doesn't stop the server.

It is software RAID when Linux does the grouping, with a tool called mdadm, instead of a special card. That is the case in this tutorial. Here is what it covers, and above all what it doesn't.
- Covered: a dying disk. The server keeps answering, and you swap the part whenever you like.
- Not covered: your own mistakes. A deleted file is deleted on every disk at once, immediately.
- Not covered: ransomware. This program encrypts your files to demand money. It encrypts them on every disk of the RAID.
- Not covered: fire, theft, power failure. All the disks are in the same machine.
So RAID is a parachute against one single risk, to add to your backups and not to replace them. There are several ways to group disks, called levels. Let's look at the three main ones.
RAID 1, RAID 5, RAID 6: which one to pick
Each form of RAID is called a level, and the level sets the price of safety in disks. RAID 1 is a mirror: two disks hold exactly the same data. RAID 5 and RAID 6 spread the data across several disks, with extra parity. Parity is a piece of control information that lets the RAID recompute the content of a lost disk.
| Level | Minimum disks | Usable space | Failures survived |
|---|---|---|---|
| RAID 1 | 2 | One disk's worth: 2 disks give 4 TB | 1 |
| RAID 5 | 3 | All disks but one: 3 disks give 8 TB | 1 |
| RAID 6 | 4 | All disks but two: 4 disks give 8 TB | 2 |
The rest of the article follows RAID 1, the easiest to read. To learn it without touching your real disks, we will build fake ones.
1Prepare fake disks
A file can play the part of a disk: Linux can present it as a device, called a loop device. We create 200 MB files, which are pretend disks. Only these files are affected from here on.
First install the tool, then build two fake disks. The truncate command creates a file of the requested size, and losetup plugs it in as a disk.
sudo apt install -y mdadm
mkdir -p ~/essai-raid
cd ~/essai-raid
truncate -s 200M disque1.img disque2.img
D1=$(sudo losetup --find --show disque1.img)
D2=$(sudo losetup --find --show disque2.img)
echo "$D1 $D2"The last line prints two names like /dev/loop0, and these names may differ on your machine. The variables D1 and D2, small named memories of the terminal, keep these two names. Keep this terminal open until the end of the experiment. The disks exist, now to join them into a RAID.
2Create the RAID 1 and read it
The mdadm --create command joins the two fake disks into a single disk, /dev/md9, where every piece of data is written twice. Three settings describe it.
--level=1picks the mirror.--raid-devices=2announces two disks.--runskips the confirmation question about starting.
sudo mdadm --create /dev/md9 --level=1 --raid-devices=2 --run "$D1" "$D2"
sudo mdadm --wait /dev/md9
cat /proc/mdstat
sudo mdadm --detail /dev/md9The creation prints a note about booting the machine (/boot): it doesn't concern us here. Right after, the RAID copies one disk onto the other for the first time, and --wait waits for that copy to finish. The /proc/mdstat file is the RAID dashboard: the kernel writes the state of each set there. Here is what you will read for a healthy RAID 1.
Personalities : [raid1]
md9 : active raid1 loop1[1] loop0[0]
203776 blocks super 1.2 [2/2] [UU]
unused devices: <none>Remember the end of the line of numbers: [2/2] says two disks are expected and two are there, and each U is a disk that is up. The --detail command gives the same thing in sentences. Look for State : clean (the RAID is in good shape) and Failed Devices : 0 (no failed disk). Now let's put this RAID to the test.
3Simulate a failure
To see the RAID hold up, we declare a disk failed and check that the data stays readable. First let's store a file on the RAID. That needs a filesystem, the way a disk organizes its files.
sudo mkfs.ext4 -q /dev/md9
sudo mkdir -p /mnt/essai
sudo mount /dev/md9 /mnt/essai
echo "contenu precieux" | sudo tee /mnt/essai/important.txtHere is what comes next as a diagram: one disk drops, the other answers alone, a third replaces it.
On to the failure itself. The --fail option declares the disk failed, and --remove takes it out of the RAID, as you would pull it from the machine. Then read the file: it should still answer.
sudo mdadm /dev/md9 --fail "$D1"
cat /proc/mdstat
cat /mnt/essai/important.txt
sudo mdadm /dev/md9 --remove "$D1"In /proc/mdstat, the failed disk carries the mark (F), and the line of numbers becomes [2/1] [_U]. The underscore marks the missing disk, and the RAID is then called degraded: it works, but with no safety net. Your file still answers: cat should print contenu precieux. Now to give the RAID its safety net back.
4Replace the disk and rebuild
Replacing a disk means plugging in a new one and asking the RAID to fill it. Let's build a third fake disk, then add it with --add. Copying the data from B to C is called the rebuild.
truncate -s 200M disque3.img
D3=$(sudo losetup --find --show disque3.img)
sudo mdadm /dev/md9 --add "$D3"
cat /proc/mdstat
sudo mdadm --wait /dev/md9
cat /proc/mdstat
sudo mdadm --detail /dev/md9During the copy, the first /proc/mdstat contains a recovery line with a percentage. On our 200 MB fake disks, it is very short. On a real disk of several terabytes, it takes hours, and the server stays usable. The --wait command waits for the end, then you get [2/2] [UU] and State : clean back. The RAID is repaired. Next time, you want to be told without looking yourself.
5Get an email when a disk fails
mdadm watches the RAIDs in the background and writes to the address on the MAILADDR line of its settings file. On Debian, this line points to root at first, the server's administrator account. Let's replace it with alice, the server's own mailbox, as in Simple monitoring. The watching service, mdmonitor, only rereads this file at startup, so we restart it.
sudo sed -i 's/^MAILADDR .*/MAILADDR alice/' /etc/mdadm/mdadm.conf
grep MAILADDR /etc/mdadm/mdadm.conf
sudo systemctl restart mdmonitor
systemctl is-active mdmonitorMAILADDR alice
activeTo check that the message goes out, mdadm can send a test email for each RAID it finds. The message lands in /var/mail/alice, and the grep command prints its subject.
sudo mdadm --monitor --scan --oneshot --test
grep ^Subject /var/mail/aliceSubject: TestMessage event on /dev/md9:mon-serveurIf /var/mail/alice doesn't exist, the mail command is missing: install it as in Simple monitoring. In a real failure, a message whose subject starts with DegradedArray event arrives within a minute. The experiment is over: let's clean up so we don't leave fake disks behind.
6Clean up the experiment
Clean up in the reverse order of creation: unmount, stop the RAID, unplug the fake disks, delete the files.
sudo umount /mnt/essai
sudo mdadm --stop /dev/md9
sudo losetup -d "$D1" "$D2" "$D3"
cd ~
rm -r ~/essai-raid
sudo rmdir /mnt/essaiYour machine is back to its earlier state. On real disks, the steps are the same, with different names.
On real disks
First find the disks with lsblk, then check their names twice. The create command erases both disks it names. Here they are /dev/sdb and /dev/sdc, two empty disks you no longer need.
lsblk -o NAME,SIZE,MODEL,SERIAL
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
sudo mkfs.ext4 /dev/md0
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -uThe creation asks two questions. The first offers an option to speed up rebuilds: answer n. The second, Continue creating array, asks for confirmation: answer y. The last two lines record the RAID in the configuration, so that it reassembles at reboot. Then mount /dev/md0 wherever you want to keep your data.
RAID can repair a failure that arrives suddenly. But a disk often warns you before it breaks, if you know how to listen. That is SMART's job.
7Read a disk's health with smartctl
SMART is a health record that every disk keeps about itself, and the smartctl command reads it. The smartmontools package provides it. Find your disk's name with lsblk, then read its record. Here we take /dev/sda.
sudo apt install -y smartmontools
lsblk -d -o NAME,SIZE,MODEL
sudo smartctl -a /dev/sdaThe output is long, but two places matter: the verdict at the top and the attribute table below it. Each line of the table is a counter kept by the disk. Here is an excerpt for a healthy disk, with sample values.
SMART overall-health self-assessment test result: PASSED
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 11874
194 Temperature_Celsius 0x0022 064 052 000 Old_age Always - 36
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0Look only at the last column, RAW_VALUE: it is the real counter, with no manufacturer calculation. Four attributes deserve your attention.
| Attribute | What it counts | If the counter goes up |
|---|---|---|
| Reallocated_Sector_Ct | Damaged areas replaced by spare areas | The disk is wearing out: prepare to replace it |
| Current_Pending_Sector | Doubtful areas it can no longer read | Keep watching: if it stays above zero, replace |
| Offline_Uncorrectable | Unreadable areas found during a test | Same answer as the previous one |
| UDMA_CRC_Error_Count | Transmission errors between the disk and the motherboard | It isn't the disk: change the cable first |
A large online backup service tracked five counters across nearly 70,000 disks. The first three in the table are among them. For that service, a counter above zero deserves an investigation. Above zero, 76.7% of the dead disks showed it, against 4.2% of the disks still healthy. The other 23.3% died without warning: a health record helps, but it guarantees nothing. That is one of the reasons to keep backups.
What about an NVMe SSD?
An NVMe SSD doesn't have the same attributes. The command is the same, with the NVMe disk's name, for example sudo smartctl -a /dev/nvme0. Look for the wear percentage and the data integrity error counter, which are the equivalent of the counters above.
Reading the record is a snapshot. To make it complete, the disk must also test itself.
8Run a short or long test
A disk can test itself without leaving service, and there are two tests. The short test takes a few minutes and checks the essentials. The long test reads the whole surface of the disk and takes several hours on a big disk. Start the short one: the command announces the duration, for example Please wait 2 minutes for test to complete. Wait that long, then read the test log and the verdict.
sudo smartctl -t short /dev/sda
sleep 150
sudo smartctl -l selftest /dev/sda
sudo smartctl -H /dev/sdaHere is the shape of the log, with sample values. The test line should read Completed without error.
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 11874 -A Completed: read failure line instead means the disk couldn't reread an area: take it seriously. For the long test, replace short with long and let it work. Running these tests by hand every week is quickly forgotten. A service can do it for you.
9Let smartd keep watch
The smartd service rereads the counters in the background, runs the tests at the scheduled times and writes to you when a counter gets worse. On Debian, it starts with the package and reads the /etc/smartd.conf file. Its DEVICESCAN line watches every disk it finds. We keep that line and add three settings to it.
-a: watch the disk's health, counters and error logs.-s (S/../.././02|L/../../6/03): a short test every day at 2 AM, a long test every Saturday at 3 AM (the 6 means Saturday).-m alice: the mailbox that receives the alerts, instead ofroot.
sudo sed -i 's#^DEVICESCAN.*#DEVICESCAN -a -d removable -n standby -s (S/../.././02|L/../../6/03) -m alice -M exec /usr/share/smartmontools/smartd-runner#' /etc/smartd.conf
grep '^DEVICESCAN' /etc/smartd.conf
sudo systemctl restart smartmontools
systemctl status smartmontools --no-pagerThe line printed should be the one you just wrote, and the service should be active (running) on a physical machine. To check that a message goes out, temporarily add -M test at the end of the line: smartd then sends a test email at every start. Remove it afterwards. The alerts arrive, and what is left is knowing what to do with them.
When to replace a disk
Replace a disk as soon as a serious sign appears, without waiting for the complete failure. Here are the signs that matter.
- A counter going up:
Reallocated_Sector_Ct,Current_Pending_SectororOffline_Uncorrectableincreases from one day to the next. - A failed long test:
Completed: read failurein the test log. - A negative verdict:
smartctl -HprintsFAILED. - A RAID that kicks out the same disk:
mdadmmarks it as failed a second time.
One exception: if only UDMA_CRC_Error_Count goes up, change the cable before the disk. My advice: a disk that has started to reallocate doesn't heal, and it costs less than a failed rebuild.
If it fails
Smartctl open device: /dev/sdz failed: No such device: the disk name is wrong, or the disk is virtual. Reread the name withlsblk -d.losetup: disque1.img: failed to set up loop device: No such file or directory: the file doesn't exist where you are. Move into~/essai-raidwithcd ~/essai-raid, or redo step 1.mdadm: cannot open /dev/loop9: No such file or directory: the fake disk name doesn't exist. Redo step 1 and check withecho "$D1 $D2".
Going further
Regular RAID checks and a spare disk
Debian provides mdcheck timers, tasks that systemd runs at set times, which reread the whole RAID to find hidden errors. See what is active on your machine with systemctl list-timers 'mdcheck*'. You can also add one more disk to a healthy RAID with --add: it stays in reserve and takes the place of the first disk that fails.
Other levels and other tools
RAID 5 and RAID 6 are created like RAID 1, with --level=5 or --level=6 and the right number of disks. Other filesystems, like Btrfs or ZFS, handle the grouping themselves. That is a different choice, with its own rules.
What's next
Your disk is now doubled and watched, and you know when to change it. What remains is the case where everything is healthy but the server crawls, because the disk is too busy. The next article, Diagnose a saturated disk, teaches you to measure that saturation. It comes now because you can only diagnose a disk well once you know it is healthy.