Not every file system do data block checksumming. And the most popular one on Linux ecosystem – ext4 file system – is one of them. That means if there is a silent data corruption occur on the disk, it may not get noticed. The corrupted data may get backed up to the off-site storage device, and by time, the backup will get corrupted after it’s been replaced by up-to-date (corrupted) version. This is potentially dangerous if the data need to be stored for a long period of time.
Silent corruption did happen, and I have real-life experience on it. While I was working for an industrial-grade network device manufacturer, one of my colleagues asks me for help on failure analysis on one of our engineering samples. I found the silent corruption on mSATA storage device. The data can be written to the mSATA storage device without error, and be read from it without error – but it’s not the same data it’s been written. This indicates a silent data corruption is possible – in theory and in real life. And subsequently, I advise my company to change the supplier of mSATA.
Here are the quick facts, based on my survey and experiments
- Ext4 filesystem has metadata checksumming but it does not have data block checksumming.
- Linux MD (Linux software RAID) don’t do data block checksumming.
- Linux LVM doesn’t do data block checksumming.
And my solution
- Use Btrfs on my laptop. It works fine on Lenovo X280 running Ubuntu 18.04 LTS. Btrfs also enable me to create a snapshot for rootfs for fast and easy recovery. The only downside is that it’s not possible/practical to put the swap file on Btrfs due to its copy-on-write nature. Having a swap partition is recommended.
- On some legacy system using MD and LVM, I change the filesystem from ext4 to Btrfs. This avoids changing the whole storage stack but enable me to detect data corruption.
- Use ZFS on new-built systems. ZFS is really powerful as it combines RAID-Z, LVM, snapshot and data integrity check on data blocks.
But Btrfs (and ZFS) is not bullet-proof on every case. If there is data corruption on the RAM, and it’s been written back to disk, Btrfs will not able to detect them. I believe using EEC RAM on the server is a good idea, but it’s still an issue on the laptop.