war story: fixing a Fedora installation that got broken during update
Many people don't update their software. They claim that it only brings grief. I'm the opposite: I take any update offered in the hope it is an improvement. One of my computers hung during a Fedora update. I don't know why it happened because I wasn't there (I also don't watch paint drying). Symptom: booting would not get the the log-in screen (GDM?). Step 1: have booting log to the display Normally Fedora booting is quiet so as not to disturb the serenity of the user. In this case we want to see the log. It might tell us what's wrong and it might tell us how far it got before hanging. - at the GRUB boot menu, type "e" to be given the chance to edit the commands that GRUB will execute to boot Linux. - cursor to the end of the "linux" line - remove the last two parameters ("rhgb quiet") - type CTRL-x to get the modified commands executed Now lots of logging information will scroll by on the screen. In my particular case, it didn't tell me much but I could see that booting got a long way. Step 2: boot from a live Fedora USB and examine the system's HDD/SSD. I assume that you know how to boot from a live installation USB flash drive. It is easy from that desktop to examine the broken system. - SMART scanning the system's disk might by worthwhile, if only to eliminate that source of doubt. - file system checks are always a good idea In my particular case, I didn't see anything obvious. Complicated brain surgery can be done by starting up the broken system in a chroot environment. I started down that road but was interrupted before success or failure. Step 3: boot the system console mode, without a GUI. This had promise because it looked to me as if the failure was starting the graphical system Proceed like step 1, but after erasing "rhgb quiet", add " 3"; hit CTRL-X. Log in. Now you have a shell. Since the system failed after a software update, I decided to do more of them. You could instead back out of them. sudo dnf update Failure: There was a set of kernel packages that seemed to be half installed. I removed the half that was installed and asked again for an update. It worked. I could reboot the system and it would come up as expected. Success!
On Sun, Oct 20, 2024 at 18:10 D. Hugh Redelmeier via talk <talk@gtalug.org> wrote:
Many people don't update their software. They claim that it only brings grief. I'm the opposite: I take any update offered in the hope it is an improvement.
One of my computers hung during a Fedora update. I don't know why it happened because I wasn't there (I also don't watch paint drying).
Symptom: booting would not get the the log-in screen (GDM?).
Step 1: have booting log to the display
Normally Fedora booting is quiet so as not to disturb the serenity of the user. In this case we want to see the log. It might tell us what's wrong and it might tell us how far it got before hanging.
- at the GRUB boot menu, type "e" to be given the chance to edit the commands that GRUB will execute to boot Linux.
- cursor to the end of the "linux" line
- remove the last two parameters ("rhgb quiet")
- type CTRL-x to get the modified commands executed
Now lots of logging information will scroll by on the screen.
In my particular case, it didn't tell me much but I could see that booting got a long way.
Step 2: boot from a live Fedora USB and examine the system's HDD/SSD.
I assume that you know how to boot from a live installation USB flash drive.
It is easy from that desktop to examine the broken system.
- SMART scanning the system's disk might by worthwhile, if only to eliminate that source of doubt.
- file system checks are always a good idea
In my particular case, I didn't see anything obvious.
Complicated brain surgery can be done by starting up the broken system in a chroot environment. I started down that road but was interrupted before success or failure.
Step 3: boot the system console mode, without a GUI.
This had promise because it looked to me as if the failure was starting the graphical system
Proceed like step 1, but after erasing "rhgb quiet", add " 3"; hit CTRL-X.
Log in. Now you have a shell.
Since the system failed after a software update, I decided to do more of them. You could instead back out of them.
sudo dnf update
Failure: There was a set of kernel packages that seemed to be half installed.
I removed the half that was installed and asked again for an update. It worked.
I could reboot the system and it would come up as expected.
Success! --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
I love war stories In the future, add “verbose” and “systemd.debug-shell=1” to the grub line, the first is obvious, the second gives you unsecured root shell on tty9 (alt f9). -nick
On Sun, 20 Oct 2024 18:10:23 -0400 (EDT) "D. Hugh Redelmeier via talk" <talk@gtalug.org> wrote:
Many people don't update their software. They claim that it only brings grief. I'm the opposite: I take any update offered in the hope it is an improvement.
Hugh, For the last fifteen years, I have had a cron job that automatically updates my Fedora box on a regular basis. I forget when it actually runs. I have had no problems, I assume my security bugs are fixed. The machine just keeps working. Any problems I have had over the last fifteen years have been due to hardware. There are hard drive tests out there. Power supplies are my nemesis. These usually are the cause of erratic behaviour. -- Howard Gibson hgibson@eol.ca http://home.eol.ca/~hgibson
Since the system failed after a software update, I decided to do more of them. You could instead back out of them.
sudo dnf update
Failure: There was a set of kernel packages that seemed to be half installed.
I removed the half that was installed and asked again for an update. It worked.
This doesn't make sense. The kernel is the only package you can install multiples of. It will never only have one kernel "rpm" installed. Even if it failed to install part of it - you should have been able to boot to an older kernel. Did you try that? Thanks! Dhaval
From: Dhaval Giani via talk <talk@gtalug.org>
This doesn't make sense. The kernel is the only package you can install multiples of. It will never only have one kernel "rpm" installed. Even if it failed to install part of it - you should have been able to boot to an older kernel. Did you try that?
Yes. Same failure mode. And console mode worked, so the kernel was pretty functional. So: still a mystery.
participants (4)
-
D. Hugh Redelmeier -
Dhaval Giani -
Howard Gibson -
Nick Accad