Debian Live Linux -- Overlay Filesystem -- Where Allocated ??
Greetings To The GTALUG Community: I'm trying to discover where an overlay filesystem is mapped, for a Debian live boot from a USB stick. After perusing a dog's breakfast of output from various linux commands, I am appealing to GTALUG members for guidance. * * * * * * I boot live Debian 11 linux from a USB memory stick. This provides an overlay filesystem: ... user@debian:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 1.6M 1.6G 1% /run /dev/sdb1 3.5G 3.5G 0 100% /run/live/medium /dev/loop0 2.9G 2.9G 0 100% /run/live/rootfs/filesystem.squashfs tmpfs 7.8G 1.6G 6.3G 20% /run/live/overlay overlay 7.8G 1.6G 6.3G 20% / tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 7.8G 8.0K 7.8G 1% /tmp tmpfs 1.6G 1.7M 1.6G 1% /run/user/1000 /dev/sdc1 59G 1.7G 57G 3% /media/user/245B-74A8 /dev/sda2 1.8T 767G 958G 45% /media/user/32ec11e8-082c-4ca5-b751-dc2852f9d5e2 user@debian:~$ ... * * * * * * I believe that the overlay filesystem provides the root directory for linux. Files under GNOME shows under Other Locations > Computer a root directory: ... user@debian:/$ ls -aAl total 0 lrwxrwxrwx 1 root root 7 Dec 17 2022 bin -> usr/bin drwxr-xr-x 2 root root 144 Dec 17 2022 boot drwxr-xr-x 18 root root 3580 Mar 20 17:58 dev drwxr-xr-x 1 root root 640 Mar 20 14:32 etc drwxr-xr-x 1 root root 60 Mar 20 08:14 home lrwxrwxrwx 1 root root 31 Dec 17 2022 initrd.img -> boot/initrd.img-5.10.0-20-amd64 lrwxrwxrwx 1 root root 31 Dec 17 2022 initrd.img.old -> boot/initrd.img-5.10.0-20-amd64 lrwxrwxrwx 1 root root 7 Dec 17 2022 lib -> usr/lib lrwxrwxrwx 1 root root 9 Dec 17 2022 lib32 -> usr/lib32 lrwxrwxrwx 1 root root 9 Dec 17 2022 lib64 -> usr/lib64 lrwxrwxrwx 1 root root 10 Dec 17 2022 libx32 -> usr/libx32 drwxr-xr-x 1 root root 60 Mar 20 08:17 media drwxr-xr-x 2 root root 3 Dec 17 2022 mnt drwxr-xr-x 2 root root 3 Dec 17 2022 opt dr-xr-xr-x 266 root root 0 Mar 20 08:14 proc drwx------ 1 root root 60 Dec 17 2022 root drwxr-xr-x 31 root root 800 Mar 20 08:17 run lrwxrwxrwx 1 root root 8 Dec 17 2022 sbin -> usr/sbin drwxr-xr-x 2 root root 3 Dec 17 2022 srv dr-xr-xr-x 13 root root 0 Mar 20 08:14 sys drwxrwxrwt 17 root root 400 Mar 20 18:49 tmp drwxr-xr-x 1 root root 80 Dec 17 2022 usr drwxr-xr-x 1 root root 140 Dec 17 2022 var lrwxrwxrwx 1 root root 28 Dec 17 2022 vmlinuz -> boot/vmlinuz-5.10.0-20-amd64 lrwxrwxrwx 1 root root 28 Dec 17 2022 vmlinuz.old -> boot/vmlinuz-5.10.0-20-amd64 user@debian:/$ ... When I right-click on the word Computer in the title bar of the Files window, and select Properties, it shows: ... Name / Type Folder (inode/directory) Contents 297,003 items, totaling 10.9 GB (some contents unreadable) The space allocation graphic shows: 1.7 GB used 6.7 GB available Filesystem type overlay Open in Disks ... * * * * * * I have tried using many linux commands to find out where the overlay filesystem is mapped: ... df -h ls -l /dev/disk/by-id gdisk (/dev/sda) gdisk (/dev/sdb) gdisk (/dev/sdc) cfdisk /dev/sda cfdisk /dev/sda cfdisk /dev/sdc sfdisk -J /dev/sda sfdisk -J /dev/sdb sfdisk -J /dev/sdc user@debian:/$ cd /dev user@debian:/dev$ ls -aAl sfdisk -T --label (dos, gpt) blkid lsblk lsblk parted -l mount sgdisk ... Output from all these commands is in the attached text file: ASP -- commands - looking for overlay filesystem device - 20240320.txt * * * * * * Suggestions from GTALUG members would be greatly appreciated !! Steve Petrie Etobicoke (Toronto), Ontario, Canada apetrie@aspetrie.net 416-233-6116
On Wed, Mar 20, 2024 at 07:55:31PM -0400, Steve Petrie via talk wrote:
Greetings To The GTALUG Community:
I'm trying to discover where an overlay filesystem is mapped, for a Debian live boot from a USB stick.
After perusing a dog's breakfast of output from various linux commands, I am appealing to GTALUG members for guidance.
* * * * * *
I boot live Debian 11 linux from a USB memory stick.
This provides an overlay filesystem: ... user@debian:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 1.6M 1.6G 1% /run /dev/sdb1 3.5G 3.5G 0 100% /run/live/medium /dev/loop0 2.9G 2.9G 0 100% /run/live/rootfs/filesystem.squashfs tmpfs 7.8G 1.6G 6.3G 20% /run/live/overlay overlay 7.8G 1.6G 6.3G 20% / tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 7.8G 8.0K 7.8G 1% /tmp tmpfs 1.6G 1.7M 1.6G 1% /run/user/1000 /dev/sdc1 59G 1.7G 57G 3% /media/user/245B-74A8 /dev/sda2 1.8T 767G 958G 45% /media/user/32ec11e8-082c-4ca5-b751-dc2852f9d5e2 user@debian:~$
So it appears that it uses a squashfs file (compressed readonly filesystem) as root with a tmpfs (ramdisk) overlay. That way you can write to it, and the writes will go to the ramdisk, while any unchanged file comes from the compressed squashfs. overlayfs allows you to put one filesystem on top of another and the second one handles writes and any files it has replace (by hiding the original) existing files in the base filesystem. So with this setup, the system appears to have a writeable root filesystem but is completely non persistent. Another common use of overlay is docker containers. Docker images are built by taking an existing image as a starting point (well you can also start from SCRATCH which is empty) and then make changes on top of that. When you run a docker container, it mounts the base layer, then overlays the next layer, then the next layer, etc for any number of changes made to the docker image as it was built and finally a writeable layer on top to store any changes made by the container relative to the docker image it was started from. So it provides a convinient copy on write mechanism which saves space by allowing multiple images to share a base image and multiple containers to share an image on disk and only have to store the deltas they have made to it. -- Len Sorensen
Hello Lennart, Thank you for your speedy explanation that, probably the Debian live linux overlayfs mechanism is using a non-persistent tempfs ramdisk to store data written to the root filesystem. Your explanation could also explain why some Debian packages I install, disappear every time I boot my linux PC, which is often daily. So, I sometimes find myself occasionally re-installing a disappeared Debian package. * * * * * * I use a live Debian linux image booted from a USB memory stick, because the original Debian 9 linux installed on the 1.8 TB hard disk, was ruined when I ran a "fix broken packages" operation, on the advice of a web page, while I was trying to install some Debian package. This "fix" removed a zillion packages and left me with a minimal linux that would only boot to a command line. After struggling unsuccessfully to repair the broken Debian 9 on the 1.8 TB hard drive, I landed on the solution of booting Debian 11 live from a USB memory stick. (In an earlier post to GTALUG I described how I built the USB live boot stick from a Debian 11 CD image.) * * * * * * I would like to explore ways to change the writeable root ovwelay filesystem, from non-persistent tempfs ramdisk, to persistent hard disk storage. This would: (1) eliminate significant RAM consumption by the overlayfs, and (2) (presumably) eliminate the current nuisance of having Debian packages disappear, every time the linux PC is booted. According to the df -l report (below), it looks to me like the current tempfs ramdisk providing writeable root filesystem space, could occupy up to 7.8 GB of RAM. Right now the free command shows RAM in 1KB units: ... user@debian:~$ free total used free shared buff/cache available Mem: 16258232 1925004 10446368 1758144 3886860 12276760 Swap: 0 0 0 user@debian:~$ ... So the linux PC has a total of 16 GB of RAM. Assuming my interpretation of the free command output is correct, the tempfs ramdisk is currently using ("shared") approximately 1.7 GB of RAM (about 10 percent of total RAM), but could (presumably) grow to consume up to 7.8 GB of RAM (per df command output). At 7.8 GB this would mean the tempfs ramdisk would consume almost 50 % of total RAM.. My Debian live linux PC occasionally crashes, with the LED on the USB live stick flashing furiously, notably when I have many Firefox browser windows open. Maybe these crashes occur when the tempfs ramdisk gobbles up lots of RAM ?? * * * * * * According to the df -l report (below), the 1.8 TB hard drive still has 958 GB of free space available. So, moving the maximum 7.8 GB volatile tempfs overlay filesystem, from ramdisk to the 1.8 TB drive would consume only 0.81 percent (7.8 / 958) of the available space on the 1.8 TB drive. Scary to move the tempfs overlay filesystem to a new partition on the 1.8 TB drive ?? As I was trying to discover where the overlay filesystem is mapped, I read that some of the command line tools I was using, provide ways to RE-PARTITION disk drives. I am a VERY PARANOID ancient IT dude, who would NOT ENJOY doing some FINGERS CROSSED messing around with the partitioning on the 1.8 TB drive. Perhaps the stress-free way to move the overlay filesystem to (permanent) hard drive space, would be to add a second huge SATA hard drive to the linux PC ?? * * * * * * Before buying another hard drive, I would research technical details of how to tell the Debian 11 live linux, to map the overlay filesystem to a partition on the new hard drive ?? Naturally I will look into this question myself, but I would gratefully appreciate suggestions from GTALUG members :) Steve Petrie Etobicoke (Toronto), Ontario, Canada apetrie@aspetrie.net 416-233-6116 On Wed, Mar 20, 2024 at 07:55:31PM -0400, Steve Petrie via talk wrote:
Greetings To The GTALUG Community:
I'm trying to discover where an overlay filesystem is mapped, for a Debian live boot from a USB stick.
After perusing a dog's breakfast of output from various linux commands, I am appealing to GTALUG members for guidance.
* * * * * *
I boot live Debian 11 linux from a USB memory stick.
This provides an overlay filesystem: ... user@debian:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 1.6M 1.6G 1% /run /dev/sdb1 3.5G 3.5G 0 100% /run/live/medium /dev/loop0 2.9G 2.9G 0 100% /run/live/rootfs/filesystem.squashfs tmpfs 7.8G 1.6G 6.3G 20% /run/live/overlay overlay 7.8G 1.6G 6.3G 20% / tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 7.8G 8.0K 7.8G 1% /tmp tmpfs 1.6G 1.7M 1.6G 1% /run/user/1000 /dev/sdc1 59G 1.7G 57G 3% /media/user/245B-74A8 /dev/sda2 1.8T 767G 958G 45% /media/user/32ec11e8-082c-4ca5-b751-dc2852f9d5e2 user@debian:~$
So it appears that it uses a squashfs file (compressed readonly filesystem) as root with a tmpfs (ramdisk) overlay. That way you can write to it, and the writes will go to the ramdisk, while any unchanged file comes from the compressed squashfs. overlayfs allows you to put one filesystem on top of another and the second one handles writes and any files it has replace (by hiding the original) existing files in the base filesystem. So with this setup, the system appears to have a writeable root filesystem but is completely non persistent. Another common use of overlay is docker containers. Docker images are built by taking an existing image as a starting point (well you can also start from SCRATCH which is empty) and then make changes on top of that. When you run a docker container, it mounts the base layer, then overlays the next layer, then the next layer, etc for any number of changes made to the docker image as it was built and finally a writeable layer on top to store any changes made by the container relative to the docker image it was started from. So it provides a convinient copy on write mechanism which saves space by allowing multiple images to share a base image and multiple containers to share an image on disk and only have to store the deltas they have made to it. -- Len Sorensen
1. It is easy to install a Linux system onto a USB stick: just tell the Linux installer to use the (second) USB stick instead of the HDD. The result has ordinary filesystems that can be updated, just like the one on your hard drive. 2. (Less useful to you) Various distros are experimenting with an immutable core. So you could have the same problem on your disk as you now have on your stick. <https://fedoramagazine.org/what-is-silverblue/> I bought a USB SSD and installed Fedora on it (see 1 above). I boot from it for a bunch of maintenance tasks. For example: resizing partitions (gparted).
| From: D. Hugh Redelmeier via talk <talk@gtalug.org> | 2. (Less useful to you) Various distros are experimenting with an | immutable core. So you could have the same problem on your disk as you | now have on your stick. <https://fedoramagazine.org/what-is-silverblue/> There is a group of immutable desktop Fedora spins called Atomic Desktops: <https://fedoramagazine.org/introducing-fedora-atomic-desktops/> The writeup says they aren't exactly immutable.
Steve Petrie via talk wrote on 2024-03-21 08:19:
some Debian packages I install, *disappear every time I boot my linux PC, which is often daily*. So, I sometimes find myself occasionally re-installing a disappeared Debian package. It might be worth looking into Ventoy for creating a bootable USB stick with persistent storage.
I haven't tried it myself, but it gets a lot of attention and sounds like it might help your situation. rb
On Thu, Mar 21, 2024 at 11:19:34AM -0400, Steve Petrie via talk wrote:
Hello Lennart,
Thank you for your speedy explanation that, probably the Debian live linux overlayfs mechanism is using a non-persistent tempfs ramdisk to store data written to the root filesystem.
Your explanation could also explain why some Debian packages I install, disappear every time I boot my linux PC, which is often daily. So, I sometimes find myself occasionally re-installing a disappeared Debian package.
* * * * * *
I use a live Debian linux image booted from a USB memory stick, because the original Debian 9 linux installed on the 1.8 TB hard disk, was ruined when I ran a "fix broken packages" operation, on the advice of a web page, while I was trying to install some Debian package.
This "fix" removed a zillion packages and left me with a minimal linux that would only boot to a command line.
After struggling unsuccessfully to repair the broken Debian 9 on the 1.8 TB hard drive, I landed on the solution of booting Debian 11 live from a USB memory stick. (In an earlier post to GTALUG I described how I built the USB live boot stick from a Debian 11 CD image.)
* * * * * *
I would like to explore ways to change the writeable root ovwelay filesystem, from non-persistent tempfs ramdisk, to persistent hard disk storage.
This would: (1) eliminate significant RAM consumption by the overlayfs, and (2) (presumably) eliminate the current nuisance of having Debian packages disappear, every time the linux PC is booted.
According to the df -l report (below), it looks to me like the current tempfs ramdisk providing writeable root filesystem space, could occupy up to 7.8 GB of RAM.
Right now the free command shows RAM in 1KB units: ... user@debian:~$ free total used free shared buff/cache available Mem: 16258232 1925004 10446368 1758144 3886860 12276760 Swap: 0 0 0 user@debian:~$ ...
So the linux PC has a total of 16 GB of RAM. Assuming my interpretation of the free command output is correct, the tempfs ramdisk is currently using ("shared") approximately 1.7 GB of RAM (about 10 percent of total RAM), but could (presumably) grow to consume up to 7.8 GB of RAM (per df command output). At 7.8 GB this would mean the tempfs ramdisk would consume almost 50 % of total RAM..
My Debian live linux PC occasionally crashes, with the LED on the USB live stick flashing furiously, notably when I have many Firefox browser windows open. Maybe these crashes occur when the tempfs ramdisk gobbles up lots of RAM ??
* * * * * *
According to the df -l report (below), the 1.8 TB hard drive still has 958 GB of free space available. So, moving the maximum 7.8 GB volatile tempfs overlay filesystem, from ramdisk to the 1.8 TB drive would consume only 0.81 percent (7.8 / 958) of the available space on the 1.8 TB drive.
Scary to move the tempfs overlay filesystem to a new partition on the 1.8 TB drive ??
As I was trying to discover where the overlay filesystem is mapped, I read that some of the command line tools I was using, provide ways to RE-PARTITION disk drives.
I am a VERY PARANOID ancient IT dude, who would NOT ENJOY doing some FINGERS CROSSED messing around with the partitioning on the 1.8 TB drive.
Perhaps the stress-free way to move the overlay filesystem to (permanent) hard drive space, would be to add a second huge SATA hard drive to the linux PC ??
* * * * * *
Before buying another hard drive, I would research technical details of how to tell the Debian 11 live linux, to map the overlay filesystem to a partition on the new hard drive ??
Naturally I will look into this question myself, but I would gratefully appreciate suggestions from GTALUG members :)
Well my understanding is that the Debian Live is only intended as a trial system and optionally an installer. It is not meant to be persistent at all. Now having a quick look there is a page on the debian wiki with some short instructions on how to add persistence to DebianLive. https://live-team.pages.debian.net/live-manual/html/live-manual/customizing-... It seems to explain how one can make Debian live have persistent storage on a USB key. -- Len Sorensen
participants (4)
-
D. Hugh Redelmeier -
Lennart Sorensen -
Ron / BCLUG -
Steve Petrie