On 20 November 2017 at 18:58, Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:

On Mon, Nov 20, 2017 at 05:25:16PM +0000, Giles Orr via talk wrote:
> I ran a backup today and noticed one file because it took so long to back
> up.  I'm using FC25 on this machine, and used the OS packages of Docker.  I
> seem to have a 100G file on a 12G partition:
>
> root@toshi7:/var/lib/docker/devicemapper/devicemapper# ls -lh
> total 34M
> -rw------- 1 root root 100G Oct 25 22:02 data
> -rw------- 1 root root 2.0G Oct 25 22:02 metadata

Try ls -lhs

For example:

~> dd if=/dev/zero bs=1M count=1 seek=1000 of=testfile
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00203937 s, 514 MB/s
~> ls -lhs testfile
1.0M -rw-r--r-- 1 lsorensen users 1001M Nov 20 13:56 testfile

So 1MB allocated out of 1001MB file size.  The first 1000MB are a 'hole'
in the file that isn't allocated yet.

The -s option to ls makes it show the allocated space in the first column.

Wow.  I did not know that, thank you.  And I see there's a specific switch to rsync for better handling of sparse files.

But ... then what exactly does straight-up 'ls' (without the '-s') report?  The man page says '-s' "print[s] the allocated size of each file, in blocks."  I was under the mistaken impression (for 23 years now) that that was more or less what 'ls' was already doing.

Here are some answers: https://en.wikipedia.org/wiki/Sparse_file ...  I get the utility of the idea, but it seems to come with some fairly significant hazards.

'ls' can be made to indicate directories (with '/') and links (with '@') and a couple other things with '-F'/'--classify': sparse files would seem to be staggeringly misleading and thus a good target for this kind of marking as well ...  Is that possible?

Where else am I likely to run into sparse files?  Sounds like mostly things that create file systems, like VirtualBox and friends, Docker (obviously) ... anywhere else?

Sorry to ask so many questions, but 'ls' seems like one of the most basic commands of Linux and I thought I knew what it did: I'm suddenly feeling like a newbie again and would like to get a handle on this ...