war story: power failure makes CentOS system unbootable

tl;dr / spoiler: change fstab so that /boot/efi is not automatically mounted. See the recommendation at the end of this message. My gateway computer is a little PC running CentOS. It does not come back after a power failure. The reason (as best I can tell) is interesting and I think that I have a fix. My system is UEFI. It boots from a UEFI system partition, known as /boot/efi to Linux. If this gets corrupted, it won't boot. It is a VFAT partition. On my gateway (a Zotac Zbox RI321, a cute little box with two ethernet ports), the UEFI firmware apparently won't boot if the dirty bit is set in the system partition. CentOS normally runs with /boot/efi mounted. So when the system isn't shut down cleanly, the dirty bit will be on. Consequence: the system is not going to boot after a power failure. Odd observation 1: if I put a live Fedora 25 USB stick in the system and try to boot in this otherwise-unbootable state, CentOS boots from the main disk. So this really looks like a firmware bug/quirk. Odd observation 2: fsck doesn't seem to automatically fix the system partition. Once CentOS 7 booted, I dismounted /boot/efi and did an fsck on it. $ sudo fsck /dev/sda1 fsck from util-linux 2.23.2 fsck.fat 3.0.20 (12 Jun 2013) 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. 1) Remove dirty bit 2) No action ? 1 Leaving filesystem unchanged. /dev/sda1: 16 files, 2420/51145 clusters Googling got me to <https://www.centos.org/forums/viewtopic.php?t=50917> In particular, this advice seemed quite good: ==== recommendation ==== Two other changes I recommend for UEFI systems, to each OS's /etc/fstab. - For the /boot/efi mountpoint, add the mount options x-systemd.automount,noauto - Change fs_passno (last column) to a 1 or 2; the canonical fstab instructions suggest 2, but systemd treats 1 and 2 as the same. The first change means the EFI System partition will not be automatically read-write mounted at boot time; it's a bad idea this is the default because it puts the ESP at risk especially if ther are crashes, FAT has no journal and will therefore always be marked dirty in such a case; no other UEFI OS mounts the ESP by default. Second, if anything tries to access /boot/efi (read or write), systemd will automatically mount it, and because of the fs_passno 1 or 2, it will fsck it first and that fixes and clears the dirty bit in case it's set. Right now without these changes, it's just a matter of having the right number and bad timing of crashes to render the EFI System partition corrupt.

Another reason why I don't use UEFI is that the boot manager is written to motherboard's non-volatile memory (what used to be "BIOS"). So, you can't simply take the harddisk and put into another machine. Found out the hard way... -- William On Thu, Jul 06, 2017 at 01:12:11PM -0400, D. Hugh Redelmeier via talk wrote:
tl;dr / spoiler: change fstab so that /boot/efi is not automatically mounted. See the recommendation at the end of this message.
My gateway computer is a little PC running CentOS. It does not come back after a power failure. The reason (as best I can tell) is interesting and I think that I have a fix.
My system is UEFI. It boots from a UEFI system partition, known as /boot/efi to Linux. If this gets corrupted, it won't boot. It is a VFAT partition.
On my gateway (a Zotac Zbox RI321, a cute little box with two ethernet ports), the UEFI firmware apparently won't boot if the dirty bit is set in the system partition.
CentOS normally runs with /boot/efi mounted. So when the system isn't shut down cleanly, the dirty bit will be on.
Consequence: the system is not going to boot after a power failure.
Odd observation 1: if I put a live Fedora 25 USB stick in the system and try to boot in this otherwise-unbootable state, CentOS boots from the main disk. So this really looks like a firmware bug/quirk.
Odd observation 2: fsck doesn't seem to automatically fix the system partition. Once CentOS 7 booted, I dismounted /boot/efi and did an fsck on it.
$ sudo fsck /dev/sda1 fsck from util-linux 2.23.2 fsck.fat 3.0.20 (12 Jun 2013) 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. 1) Remove dirty bit 2) No action ? 1 Leaving filesystem unchanged. /dev/sda1: 16 files, 2420/51145 clusters
Googling got me to <https://www.centos.org/forums/viewtopic.php?t=50917> In particular, this advice seemed quite good:
==== recommendation ====
Two other changes I recommend for UEFI systems, to each OS's /etc/fstab.
- For the /boot/efi mountpoint, add the mount options x-systemd.automount,noauto
- Change fs_passno (last column) to a 1 or 2; the canonical fstab instructions suggest 2, but systemd treats 1 and 2 as the same.
The first change means the EFI System partition will not be automatically read-write mounted at boot time; it's a bad idea this is the default because it puts the ESP at risk especially if ther are crashes, FAT has no journal and will therefore always be marked dirty in such a case; no other UEFI OS mounts the ESP by default. Second, if anything tries to access /boot/efi (read or write), systemd will automatically mount it, and because of the fs_passno 1 or 2, it will fsck it first and that fixes and clears the dirty bit in case it's set.
Right now without these changes, it's just a matter of having the right number and bad timing of crashes to render the EFI System partition corrupt. --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk

| From: William Park via talk <talk@gtalug.org> | Another reason why I don't use UEFI is that the boot manager is | written to motherboard's non-volatile memory (what used to be "BIOS"). | So, you can't simply take the harddisk and put into another machine. | Found out the hard way. I wrote and discarded several long responses. Here's a shorter one. There are conventionally two kinds of relevant non-volatile memory on a motherboard; - the eeprom that contains the firmware. It is only updated when one updates the BIOS - what was called the CMOS memory: battery-backed up RAM. It was used to store the "BIOS settings". I imagine this is quite a bit larger now so that it can accommodate the UEFI settings. I think that you are referring to the second. The UEFI settings include public keys for Secure Boot; if you've added some, you have to install them on the new machine too. I'll ignore this issue. The key mystery in the UEFI settings is the boot choices. I don't really understand them, so I've been looking at a gentle introduction <https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/> Try the command sudo efibootmgr -v On my machine, some of the entries are pretty ugly, possibly because some file names are in UTF-16 (an abomination). Here's the first entry: Boot0000* Fedora HD(2,GPT,f66e4ede-1301-47fd-af96-7f45aee7bc28,0x40800,0xb4000)/File(\EFI\fedora\shim.efi) This means: - present it as "Fedora" in the boot prompt - if selected, run it from + hard drive 2, + with GPT partitioning, + from the partition with Globally Unique ID f66e4ede-1301-47fd-af96-7f45aee7bc28 + from file \EFI\fedora\shim.efi + mystery: 0x40800,0xb4000 If you move this disk to a new machine, there is no way that the firmware will boot this entry without reconfiguring. Apparently it will be able to boot \EFI\BOOT\BOOTX64.EFI How to tell the setup screen that you wish to add something to the boot menu is not standardized and is often difficult to figure out.

I'm replying to a really old posting I made. [Mystery: I'm replying to the copy I sent. Formatting is screwed up when I reply to the copy I got back from the list server.] [I'm top posting because I'm just including it whole, for reference. I'm sure most of you have deleted it from you mailboxes after 4 years.] I followed the instructions on how to make /boot/esp mounted on demand, but I still have problems with my CentOS box not rebooting after a crash because the ESP is marked as dirty. "journalctl -b" says that it is mounting /boot/efi because packagekitd is requesting it Aug 07 10:38:35 reddoor.mimosa.com PackageKit[1855]: daemon start Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Got automount request for /boot/efi, triggered by 1855 (packagekitd) Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Starting File System Check on /dev/disk/by-uuid/A9B4-1C74... Aug 07 10:38:35 reddoor.mimosa.com systemd-fsck[1866]: fsck.fat 3.0.20 (12 Jun 2013) Aug 07 10:38:35 reddoor.mimosa.com systemd-fsck[1866]: /dev/sda1: 25 files, 2913/51145 clusters Aug 07 10:38:35 reddoor.mimosa.com chronyd[724]: Selected source 207.34.48.31 Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Started File System Check on /dev/disk/by-uuid/A9B4-1C74. Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Mounting /boot/efi... Aug 07 10:38:36 reddoor.mimosa.com spice-vdagent[1942]: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0 Aug 07 10:38:36 reddoor.mimosa.com gnome-session-binary[1460]: WARNING: App 'spice-vdagent.desktop' exited with code 1 Aug 07 10:38:36 reddoor.mimosa.com gnome-session-binary[1460]: Entering running state Aug 07 10:38:36 reddoor.mimosa.com gnome-session[1460]: gnome-session-binary[1460]: WARNING: App 'spice-vdagent.desktop' exited with code 1 Aug 07 10:38:36 reddoor.mimosa.com systemd[1]: Mounted /boot/efi. Fix: why the heck am I running packagekit? Stop that. $ sudo systemctl disable packagekit $ sudo systemctl stop packagekit $ sudo umount /boot/efi See https://bugzilla.redhat.com/show_bug.cgi?id=1991228 | From: D. Hugh Redelmeier <hugh@mimosa.com> | To: Toronto Linux Users Group <talk@gtalug.org> | Date: Thu, 6 Jul 2017 13:12:11 -0400 (EDT) | Subject: war story: power failure makes CentOS system unbootable | | tl;dr / spoiler: change fstab so that /boot/efi is not automatically | mounted. See the recommendation at the end of this message. | | My gateway computer is a little PC running CentOS. | It does not come back after a power failure. | The reason (as best I can tell) is interesting and I think that I have a | fix. | | My system is UEFI. It boots from a UEFI system partition, known as | /boot/efi to Linux. If this gets corrupted, it won't boot. It is a | VFAT partition. | | On my gateway (a Zotac Zbox RI321, a cute little box with two ethernet | ports), the UEFI firmware apparently won't boot if the dirty bit is | set in the system partition. | | CentOS normally runs with /boot/efi mounted. So when the system isn't | shut down cleanly, the dirty bit will be on. | | Consequence: the system is not going to boot after a power failure. | | Odd observation 1: if I put a live Fedora 25 USB stick in the system | and try to boot in this otherwise-unbootable state, CentOS boots from | the main disk. So this really looks like a firmware bug/quirk. | | Odd observation 2: fsck doesn't seem to automatically fix the system | partition. Once CentOS 7 booted, I dismounted /boot/efi and | did an fsck on it. | | $ sudo fsck /dev/sda1 | fsck from util-linux 2.23.2 | fsck.fat 3.0.20 (12 Jun 2013) | 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. | 1) Remove dirty bit | 2) No action | ? 1 | Leaving filesystem unchanged. | /dev/sda1: 16 files, 2420/51145 clusters | | Googling got me to <https://www.centos.org/forums/viewtopic.php?t=50917> | In particular, this advice seemed quite good: | | ==== recommendation ==== | | Two other changes I recommend for UEFI systems, to each OS's /etc/fstab. | | - For the /boot/efi mountpoint, add the mount options | x-systemd.automount,noauto | | - Change fs_passno (last column) to a 1 or 2; the canonical fstab | instructions suggest 2, but systemd treats 1 and 2 as the same. | | The first change means the EFI System partition will not be automatically | read-write mounted at boot time; it's a bad idea this is the default | because it puts the ESP at risk especially if ther are crashes, FAT has no | journal and will therefore always be marked dirty in such a case; no other | UEFI OS mounts the ESP by default. Second, if anything tries to access | /boot/efi (read or write), systemd will automatically mount it, and | because of the fs_passno 1 or 2, it will fsck it first and that fixes and | clears the dirty bit in case it's set. | | Right now without these changes, it's just a matter of having the right | number and bad timing of crashes to render the EFI System partition | corrupt. |

I'm continuing a very old thread of mine. Because of the age, I'm top-posting. Summary up until now: - system fails to reboot after power failure - problem seems to be firmware distaste for dirty bit on ESP (/boot/efi) - dirty bit always happens because /boot/efi is always mounted, even though it isn't needed 99% of the time - mounting-on-demand doesn't fix this because packagekit accesses the ESP almost immediately. This feels like a bug. - I disabled packagekit to prevent /boot/efi from being automounted. Well, it grew back, somehow. Updates, I guess. Now: So now I'll use a different approach. I'll get automount to also autodismount. Oh, the wonders and mysteries provided by systemd! 1. Modify the /etc/fstab entry for /boot/efi to include "x-systemd.idle-timeout=600" This asks for dismounting after 600 seconds of inactivity. 2. Get the relevant daemon(s) to pay attention: systemctl daemon-reload systemctl restart boot-efi.automount Note that boot-efi.automount's name is synthesized automatically from the mount point. So is the target script itself. 3. Hope this work See systemd.mount(5). You can ignore or be confused by systemd.automount(5). | From: D. Hugh Redelmeier <hugh@mimosa.com> | Date: Sun, 8 Aug 2021 09:47:42 -0400 (EDT) | Subject: Re: war story: power failure makes CentOS system unbootable | | I'm replying to a really old posting I made. | | [Mystery: I'm replying to the copy I sent. Formatting is screwed up when | I reply to the copy I got back from the list server.] | | [I'm top posting because I'm just including it whole, for reference. I'm | sure most of you have deleted it from you mailboxes after 4 years.] | | I followed the instructions on how to make /boot/esp mounted on demand, | but I still have problems with my CentOS box not rebooting after a | crash because the ESP | is marked as dirty. | | "journalctl -b" says that it is mounting /boot/efi because packagekitd | is requesting it | | Aug 07 10:38:35 reddoor.mimosa.com PackageKit[1855]: daemon start | Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Got automount request for /boot/efi, triggered by 1855 (packagekitd) | Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Starting File System Check on /dev/disk/by-uuid/A9B4-1C74... | Aug 07 10:38:35 reddoor.mimosa.com systemd-fsck[1866]: fsck.fat 3.0.20 (12 Jun 2013) | Aug 07 10:38:35 reddoor.mimosa.com systemd-fsck[1866]: /dev/sda1: 25 files, 2913/51145 clusters | Aug 07 10:38:35 reddoor.mimosa.com chronyd[724]: Selected source 207.34.48.31 | Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Started File System Check on /dev/disk/by-uuid/A9B4-1C74. | Aug 07 10:38:35 reddoor.mimosa.com systemd[1]: Mounting /boot/efi... | Aug 07 10:38:36 reddoor.mimosa.com spice-vdagent[1942]: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0 | Aug 07 10:38:36 reddoor.mimosa.com gnome-session-binary[1460]: WARNING: App 'spice-vdagent.desktop' exited with code 1 | Aug 07 10:38:36 reddoor.mimosa.com gnome-session-binary[1460]: Entering running state | Aug 07 10:38:36 reddoor.mimosa.com gnome-session[1460]: gnome-session-binary[1460]: WARNING: App 'spice-vdagent.desktop' exited with code 1 | Aug 07 10:38:36 reddoor.mimosa.com systemd[1]: Mounted /boot/efi. | | Fix: why the heck am I running packagekit? Stop that. | | $ sudo systemctl disable packagekit | $ sudo systemctl stop packagekit | $ sudo umount /boot/efi | | See https://bugzilla.redhat.com/show_bug.cgi?id=1991228 | | | From: D. Hugh Redelmeier <hugh@mimosa.com> | | To: Toronto Linux Users Group <talk@gtalug.org> | | Date: Thu, 6 Jul 2017 13:12:11 -0400 (EDT) | | Subject: war story: power failure makes CentOS system unbootable | | | | tl;dr / spoiler: change fstab so that /boot/efi is not automatically | | mounted. See the recommendation at the end of this message. | | | | My gateway computer is a little PC running CentOS. | | It does not come back after a power failure. | | The reason (as best I can tell) is interesting and I think that I have a | | fix. | | | | My system is UEFI. It boots from a UEFI system partition, known as | | /boot/efi to Linux. If this gets corrupted, it won't boot. It is a | | VFAT partition. | | | | On my gateway (a Zotac Zbox RI321, a cute little box with two ethernet | | ports), the UEFI firmware apparently won't boot if the dirty bit is | | set in the system partition. | | | | CentOS normally runs with /boot/efi mounted. So when the system isn't | | shut down cleanly, the dirty bit will be on. | | | | Consequence: the system is not going to boot after a power failure. | | | | Odd observation 1: if I put a live Fedora 25 USB stick in the system | | and try to boot in this otherwise-unbootable state, CentOS boots from | | the main disk. So this really looks like a firmware bug/quirk. | | | | Odd observation 2: fsck doesn't seem to automatically fix the system | | partition. Once CentOS 7 booted, I dismounted /boot/efi and | | did an fsck on it. | | | | $ sudo fsck /dev/sda1 | | fsck from util-linux 2.23.2 | | fsck.fat 3.0.20 (12 Jun 2013) | | 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. | | 1) Remove dirty bit | | 2) No action | | ? 1 | | Leaving filesystem unchanged. | | /dev/sda1: 16 files, 2420/51145 clusters | | | | Googling got me to <https://www.centos.org/forums/viewtopic.php?t=50917> | | In particular, this advice seemed quite good: | | | | ==== recommendation ==== | | | | Two other changes I recommend for UEFI systems, to each OS's /etc/fstab. | | | | - For the /boot/efi mountpoint, add the mount options | | x-systemd.automount,noauto | | | | - Change fs_passno (last column) to a 1 or 2; the canonical fstab | | instructions suggest 2, but systemd treats 1 and 2 as the same. | | | | The first change means the EFI System partition will not be automatically | | read-write mounted at boot time; it's a bad idea this is the default | | because it puts the ESP at risk especially if ther are crashes, FAT has no | | journal and will therefore always be marked dirty in such a case; no other | | UEFI OS mounts the ESP by default. Second, if anything tries to access | | /boot/efi (read or write), systemd will automatically mount it, and | | because of the fs_passno 1 or 2, it will fsck it first and that fixes and | | clears the dirty bit in case it's set. | | | | Right now without these changes, it's just a matter of having the right | | number and bad timing of crashes to render the EFI System partition | | corrupt. | | |

On Thu, Sep 29, 2022 at 12:22:08PM -0400, D. Hugh Redelmeier via talk wrote:
I'm continuing a very old thread of mine. Because of the age, I'm top-posting.
Summary up until now:
- system fails to reboot after power failure
- problem seems to be firmware distaste for dirty bit on ESP (/boot/efi)
- dirty bit always happens because /boot/efi is always mounted, even though it isn't needed 99% of the time
- mounting-on-demand doesn't fix this because packagekit accesses the ESP almost immediately. This feels like a bug.
- I disabled packagekit to prevent /boot/efi from being automounted. Well, it grew back, somehow. Updates, I guess.
Now:
So now I'll use a different approach. I'll get automount to also autodismount. Oh, the wonders and mysteries provided by systemd!
1. Modify the /etc/fstab entry for /boot/efi to include "x-systemd.idle-timeout=600" This asks for dismounting after 600 seconds of inactivity.
2. Get the relevant daemon(s) to pay attention: systemctl daemon-reload systemctl restart boot-efi.automount Note that boot-efi.automount's name is synthesized automatically from the mount point. So is the target script itself.
3. Hope this work
See systemd.mount(5). You can ignore or be confused by systemd.automount(5).
It seems to correct thing to do is get the kernel fixed. https://github.com/torvalds/linux/commit/b88a105802e9aeb6e234e8106659f5d1271... clearly states that windows only sets the dirty bit on write, yet for some reason it was chosen to make linux set it whenever you mount the vfat read-write. That is dumb. Of course the UEFI firmware that only works when the dirty bit is cleared is also dumb, but what can you do with firmware developers. Obviously these defective UEFI firmwares should ignore the bit but since they don't and windows doesn't break them (because it only sets it when writing to the filesystem the first time) it would make sense to fix it. Of course this means instead of being able to have all the logic do this at mount time, there has to be a state kept that is checked whenever writes happen to the filesystem and set the bit at that time instead. -- Len Sorensen

On Thu, Sep 29, 2022 at 06:11:29PM -0400, Lennart Sorensen via talk wrote:
It seems to correct thing to do is get the kernel fixed.
https://github.com/torvalds/linux/commit/b88a105802e9aeb6e234e8106659f5d1271... clearly states that windows only sets the dirty bit on write, yet for some reason it was chosen to make linux set it whenever you mount the vfat read-write. That is dumb. Of course the UEFI firmware that only works when the dirty bit is cleared is also dumb, but what can you do with firmware developers.
Obviously these defective UEFI firmwares should ignore the bit but since they don't and windows doesn't break them (because it only sets it when writing to the filesystem the first time) it would make sense to fix it. Of course this means instead of being able to have all the logic do this at mount time, there has to be a state kept that is checked whenever writes happen to the filesystem and set the bit at that time instead.
Hmm, it's more complicated than that. vfat supports atime updates, meaning just reading files from it causes writes. Lovely. So much for a simple fix. Can you somehow flag it to be mounted read-only and then manually remount it if you actually have a reason to update it? -- Len Sorensen

| From: Lennart Sorensen via talk <talk@gtalug.org> | Hmm, it's more complicated than that. vfat supports atime updates, | meaning just reading files from it causes writes. Lovely. So much for | a simple fix. Interesting. I never knew that. But the dirty bit could/should be cleared whenever the FS is in a consistent state. Like: after a sync AND no files are open. Maybe having a process with a working directory on the FS counts as having a file open; maybe not. | Can you somehow flag it to be mounted read-only and then manually remount | it if you actually have a reason to update it? I imagine that that would require cooperation of too many actors (each thinking they are the star of the show). I could experiment but how could I test all the cases> My initial position was: packagekit should not need to access the ESP except to apply updates. But packagekit accesses the ESP when it starts up. <https://bugzilla.redhat.com/show_bug.cgi?id=1991228> Notice that there has been no response.

On Thu, Sep 29, 2022 at 09:32:19PM -0400, D. Hugh Redelmeier wrote:
Interesting. I never knew that.
I didn't either. I thought atime was only one better filesystems.
But the dirty bit could/should be cleared whenever the FS is in a consistent state. Like: after a sync AND no files are open. Maybe having a process with a working directory on the FS counts as having a file open; maybe not.
Well even windows apparently only clears it when unmounting. I guess updating that flag everytime you are done writing gets to be a lot of writing. And of a file is open for writing I guess that means the filesystem is dirty until the file is closed and fully flushed to disk.
I imagine that that would require cooperation of too many actors (each thinking they are the star of the show). I could experiment but how could I test all the cases>
My initial position was: packagekit should not need to access the ESP except to apply updates. But packagekit accesses the ESP when it starts up. <https://bugzilla.redhat.com/show_bug.cgi?id=1991228> Notice that there has been no response.
I don't usually see too many responses on bugs filed with redhat. Is /boot/esp in /etc/fstab? Well as far as I can tell from the man page for systemd-gpt-auto-generator, it will actually automatically mount the ESP partition to /efi or /boot (not sure about /boot/efi which seems to be where most distributions put it as far as I know). It also mentions a flag you can set on the ESP partition to make it not auto mount it. -- Len Sorensen

| From: Lennart Sorensen via talk <talk@gtalug.org> | Is /boot/esp in /etc/fstab? It is called /boot/efi. Yes, on my systems, it is in /etc/fstab. On my desktop: UUID=3C5D-B772 /boot/efi vfat umask=0077,shortname=winnt 0 2 On the CentOS system that acts up: UUID=A9B4-1C74 /boot/efi vfat umask=0077,shortname=winnt,x-systemd.automount,noauto,x-systemd.idle-timeout=600 0 2 | Well as far as I can tell from the man page for | systemd-gpt-auto-generator, it will actually automatically mount the ESP | partition to /efi or /boot (not sure about /boot/efi which seems to be | where most distributions put it as far as I know). It also mentions a | flag you can set on the ESP partition to make it not auto mount it. I imagine that is something to streamline containers. Some distros might use it. None that I use, so far. The requirement that / be on the same device as the ESP would sometimes pinch me. Most of the time it is OK.
participants (3)
-
D. Hugh Redelmeier
-
Lennart Sorensen
-
William Park