Filesystems Linux Storage Storage Management Tools

The secret to fixing a mangled filesystem

David Both

Modern computers and the storage devices we use in them, both HDDs and SSDs, are quite resilient. It’s seldom that a power failure, tripping over a power cord and pulling it out of the wall, or even an accidental bump on the reset button, can damage an entire filesystem. For various reasons, it just doesn’t happen much.

Except when it does

But nothing’s perfect and sometimes a filesystem can be damaged — that is, become inconsistent. This just means a file may be partially written to the disk, or its data written but the list of inodes doesn’t get updated, or new inodes don’t get written to the disk, to list just a few potential issues.

These inconsistencies can cause individual files to be unreadable, or prevent access to an entire filesystem or directory structure. Depending upon the location of the inconsistencies, they could prevent the system from booting.

But there’s an app for that.

The secret sauce

The fsck program has been around ever since the beginning of Linux. It’s function is to verify the structural integrity of a filesystem and its metadata, and to repair it if possible and if requested. Like many Linux tools, it’s name is meaningful, in this instance it stands for “filesystem check.”

According to the man page for fsck, it can check and repair the following list of filesystems.

  • EXT2
  • EXT3
  • EXT4
  • CramFS
  • JFS
  • NFS
  • Minix
  • MSDOS
  • VFAT
  • XFS
  • ReiserFS

The real secret is that the fsck utility is run against all supported filesystems early in the startup process, and before the filesystems are mounted. This will usually locate and resolve problems that occurred prior to or during the last power off or reboot.

Run fsck manually

Sometimes it’s possible to use fsck on a running system. If the filesystem can be unmounted, fsck can be run against it. The fsck command will not run against a mounted filesystem because changes generated by other programs might conflict with the changes being made by fsck. This could lead to additional inconsistencies and result in the filesystem being unrecoverable. The -y option causes fsck to assume “yes” to all questions about whether an inconsistency should be corrected. On a badly damaged filesystem, failure to use this option can result in the need fo a lot of “y” responses. 

# fsck -y /dev/mapper/vg01-root
fsck from util-linux 2.41.1
e2fsck 1.47.3 (8-Jul-2025)
/dev/mapper/vg01-root is mounted.
e2fsck: Cannot continue, aborting

For example, I use a separate LVM/EXT4 filesystem for Ansible files such as the Ansible YAML configuration files, files to be updated, and more. If there is an inconsistency on that volume, I can unmount it and run fsck against it.

# lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                      8:0    0 931.5G  0 disk
└─vg03-Virtual         252:6    0 931.5G  0 lvm  /Virtual
<SNIP>
nvme1n1                259:0    0 476.9G  0 disk
└─vg02-home            252:2    0   250G  0 lvm  /home
nvme0n1                259:1    0 476.9G  0 disk
├─nvme0n1p1            259:2    0     5G  0 part /boot/efi
├─nvme0n1p2            259:3    0     5G  0 part /boot
└─nvme0n1p3            259:4    0 466.9G  0 part
  ├─vg01-root          252:0    0    10G  0 lvm  /
  ├─vg01-usr           252:1    0    60G  0 lvm  /usr
  ├─vg01-var           252:3    0    50G  0 lvm  /var
  ├─vg01-tmp           252:4    0    15G  0 lvm  /tmp
  └─vg01-ansible       252:5    0    15G  0 lvm  /home/dboth/development/ansible

# umount /dev/mapper/vg01-ansible
# fsck -y /dev/mapper/vg01-ansible
fsck from util-linux 2.41.1
e2fsck 1.47.3 (8-Jul-2025)
ansible: clean, 41654/983040 files, 143455/3932160 blocks
# mount /dev/mapper/vg01-ansible

In this case the filesystem is clean so no fixes are made. They would be listed if any had been made.

The good news is that it’s been years since I’ve had a need to run fsck manually. But it could still happen and having that ability can minimize the impact of using fsck to recover a mangled filesystem.

The role of /etc/fstab

The /etc/fstab file is used to define the mount points for each filesystem on the host. It’s still in use, although systemd is changing things a bit, but that discussion is beyond the scope of this article.

The fstab file also specifies the sequence in which fsck is run on the host’s filesystems during startup. The last column of each filesystem entry in this example from my primary workstation is a number from 1 to 3. The root filesystem (/) has the number 1 because it must be the first filesystem mounted; all other filesystems are mounted somewhere on the root directory tree. So fsck must be run first against the root filesystem so it can be mounted first.borked

The /boot, /boot/efi, /home, /tmp, /usr, and /var, all have the number 2. After the root filesystem is mounted, fsck is run in parallel against the filesystems marked by the number 2 and each is mounted as the check is completed.

My ansible filesystem is located on /home/dboth/development/ansible, and both root (/) and /home must be mounted before the ansible filesystem can be mounted. So it has a number of 3 to ensure that it is checked and mounted after the other filesystems in that specific tree.

#############################################################################################
#
# /etc/fstab
# Created by anaconda on Thu Sep 10 11:12:33 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/vg01-root                     /                  ext4    defaults        1 1
UUID=73b04be9-ff03-4583-877a-9b6e556eecb5 /boot              ext4    defaults        1 2
UUID=1AE1-6B0D                            /boot/efi          vfat    umask=0077,shortname=winnt 0 2
/dev/mapper/vg02-home                     /home              ext4    defaults        1 2
/dev/mapper/vg01-tmp                      /tmp               ext4    defaults        1 2
/dev/mapper/vg01-usr                      /usr               ext4    defaults        1 2
/dev/mapper/vg01-var                      /var               ext4    defaults        1 2
/dev/mapper/vg01-ansible    /home/dboth/development/ansible  ext4    defaults        1 3
<SNIP>

Because fsck is run on all supported filesystems during the startup phase, it’s invisible to most users who will never know that there was a mangled filesystem on their computer.

BtrFS

Although there is a program, fsck.btrfs, it does nothing and returns a success code. BtrFS has it’s own consistency check, btrfs check. But that’s another story.

In,

Leave a Reply

More Stories