Hello Lennart,

Thank you for your speedy explanation that, probably the Debian live linux overlayfs mechanism is using a non-persistent tempfs ramdisk to store data written to the root filesystem.

Your explanation could also explain why some Debian packages I install, disappear every time I boot my linux PC, which is often daily. So, I sometimes find myself occasionally re-installing a disappeared Debian package.

* * *
* * *

I use a live Debian linux image booted from a USB memory stick, because the original Debian 9 linux installed on the 1.8 TB hard disk, was ruined when I ran a "fix broken packages" operation, on the advice of a web page, while I was trying to install some Debian package.

This "fix" removed a zillion packages and left me with a minimal linux that would only boot to a command line.

After struggling unsuccessfully to repair the broken Debian 9 on the 1.8 TB hard drive, I landed on the solution of booting Debian 11 live from a USB memory stick. (In an earlier post to GTALUG I described how I built the USB live boot stick from a Debian 11 CD image.)

* * *
* * *

I would like to explore ways to change the writeable root ovwelay filesystem, from non-persistent tempfs ramdisk, to persistent hard disk storage.

This would: (1) eliminate significant RAM consumption by the overlayfs, and (2) (presumably) eliminate the current nuisance of having Debian packages disappear, every time the linux PC is booted.

According to the df -l report (below), it looks to me like the current tempfs ramdisk providing writeable root filesystem space, could occupy up to 7.8 GB of RAM.

Right now the free command shows RAM in 1KB units:

...
user@debian:~$ free
total used free shared buff/cache available
Mem: 16258232 1925004 10446368 1758144 3886860 12276760
Swap: 0 0 0
user@debian:~$
...

So the linux PC has a total of 16 GB of RAM. Assuming my interpretation of the free command output is correct, the tempfs ramdisk is currently using ("shared") approximately 1.7 GB of RAM (about 10 percent of total RAM), but could (presumably) grow to consume up to 7.8 GB of RAM (per df command output). At 7.8 GB this would mean the tempfs ramdisk would consume almost 50 % of total RAM..

My Debian live linux PC occasionally crashes, with the LED on the USB live stick flashing furiously, notably when I have many Firefox browser windows open. Maybe these crashes occur when the tempfs ramdisk gobbles up lots of RAM ??

* * *
* * *

According to the df -l report (below), the 1.8 TB hard drive still has 958 GB of free space available. So, moving the maximum 7.8 GB volatile tempfs overlay filesystem, from ramdisk to the 1.8 TB drive would consume only 0.81 percent (7.8 / 958) of the available space on the 1.8 TB drive.

Scary to move the tempfs overlay filesystem to a new partition on the 1.8 TB drive ??

As I was trying to discover where the overlay filesystem is mapped, I read that some of the command line tools I was using, provide ways to RE-PARTITION disk drives.

I am a VERY PARANOID ancient IT dude, who would NOT ENJOY doing some FINGERS CROSSED messing around with the partitioning on the 1.8 TB drive.

Perhaps the stress-free way to move the overlay filesystem to (permanent) hard drive space, would be to add a second huge SATA hard drive to the linux PC ??

* * *

Before buying another hard drive, I would research technical details of how to tell the Debian 11 live linux, to map the overlay filesystem to a partition on the new hard drive ??

Naturally I will look into this question myself, but I would gratefully appreciate suggestions from GTALUG members :)

Steve Petrie

Etobicoke (Toronto), Ontario, Canada

apetrie@aspetrie.net

416-233-6116

On Wed, Mar 20, 2024 at 07:55:31PM -0400, Steve Petrie via talk wrote:

Greetings To The GTALUG Community:

I'm trying to discover where an overlay filesystem is mapped, for a Debian
live boot from a USB stick.

After perusing a dog's breakfast of output from various linux commands, I am
appealing to GTALUG members for guidance.

* * *
* * *

I boot live Debian 11 linux from a USB memory stick.

This provides an overlay filesystem:
...
user@debian:~$ df -h
Filesystem      Size Used Avail Use% Mounted on
udev            7.8G     0 7.8G   0% /dev
tmpfs           1.6G 1.6M 1.6G   1% /run
/dev/sdb1       3.5G 3.5G     0 100% /run/live/medium
/dev/loop0      2.9G 2.9G     0 100% /run/live/rootfs/filesystem.squashfs
tmpfs           7.8G 1.6G 6.3G 20% /run/live/overlay
overlay         7.8G 1.6G 6.3G 20% /
tmpfs           7.8G     0 7.8G   0% /dev/shm
tmpfs           5.0M 4.0K 5.0M   1% /run/lock
tmpfs           7.8G 8.0K 7.8G   1% /tmp
tmpfs           1.6G 1.7M 1.6G   1% /run/user/1000
/dev/sdc1        59G 1.7G   57G   3% /media/user/245B-74A8
/dev/sda2       1.8T 767G 958G 45% /media/user/32ec11e8-082c-4ca5-b751-dc2852f9d5e2
user@debian:~$

So it appears that it uses a squashfs file (compressed readonly
filesystem) as root with a tmpfs (ramdisk) overlay. That way you can
write to it, and the writes will go to the ramdisk, while any unchanged
file comes from the compressed squashfs. overlayfs allows you to put
one filesystem on top of another and the second one handles writes and
any files it has replace (by hiding the original) existing files in the
base filesystem.

So with this setup, the system appears to have a writeable root filesystem
but is completely non persistent.

Another common use of overlay is docker containers. Docker images are
built by taking an existing image as a starting point (well you can also
start from SCRATCH which is empty) and then make changes on top of that.
When you run a docker container, it mounts the base layer, then overlays
the next layer, then the next layer, etc for any number of changes made
to the docker image as it was built and finally a writeable layer on
top to store any changes made by the container relative to the docker
image it was started from. So it provides a convinient copy on write
mechanism which saves space by allowing multiple images to share a base
image and multiple containers to share an image on disk and only have
to store the deltas they have made to it.

--
Len Sorensen