Re: [GTALUG] On the subject of backups.
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote:
Hi Alvin,
On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no?
now, for my own info/interest, can you tell me which fs is used for this ext3?
Hmm, sounds awful slow. Just for fun I ran find on one of my drives: # time find /data | wc -l 1825463 real 3m57s.208s That is with 5.3T used out of 6.0TB. Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. Another volume: # time find /mythdata | wc -l 54972 real 0m1.924s That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason). Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives. Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. -- Len Sorensen
On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote:
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote:
Hi Alvin,
On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no?
now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow.
Just for fun I ran find on one of my drives:
# time find /data | wc -l 1825463 real 3m57s.208s
That is with 5.3T used out of 6.0TB.
Running it a second time when it is cached takes 7.7s. Tree takes 14.7s.
Another volume: # time find /mythdata | wc -l 54972
real 0m1.924s
That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason).
Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives.
Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine.
I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote:
On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote:
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote:
Hi Alvin,
On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no?
now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow.
Just for fun I ran find on one of my drives:
# time find /data | wc -l 1825463 real 3m57s.208s
That is with 5.3T used out of 6.0TB.
Running it a second time when it is cached takes 7.7s. Tree takes 14.7s.
Another volume: # time find /mythdata | wc -l 54972
real 0m1.924s
That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason).
Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives.
Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine.
I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files.
The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out. Maybe that helps, Nick
On 5/4/20 7:28 PM, nick wrote:
On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote:
On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote:
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote:
Hi Alvin,
On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no?
now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow.
Just for fun I ran find on one of my drives:
# time find /data | wc -l 1825463 real 3m57s.208s
That is with 5.3T used out of 6.0TB.
Running it a second time when it is cached takes 7.7s. Tree takes 14.7s.
Another volume: # time find /mythdata | wc -l 54972
real 0m1.924s
That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason).
Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives.
Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files.
The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%. I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long.
The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out.
Maybe that helps,
Nick
-- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On 5/4/20 10:42 PM, Alvin Starr wrote:
On 5/4/20 7:28 PM, nick wrote:
On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote:
On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote:
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote:
Hi Alvin,
On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no?
now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow.
Just for fun I ran find on one of my drives:
# time find /data | wc -l 1825463 real 3m57s.208s
That is with 5.3T used out of 6.0TB.
Running it a second time when it is cached takes 7.7s. Tree takes 14.7s.
Another volume: # time find /mythdata | wc -l 54972
real 0m1.924s
That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason).
Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives.
Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files.
The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. Interesting, I recall CentOS 7 and RHEL 7 supporting it. You must be on a older system unless my memory is wrong :). The problem really is seek times. I've not sure if there is a way to tune the kernel from memory for this but that's your problem unfortunately.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%.
I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. See my above comments but the other two ideas would be to use some short of filesystem or LVM snapshots or try dd as a benchmark to see if a raw copy tool does better. Snapshots are probably the best way through or only changed blocked based tools as you logically came to the conclusion of.
Good luck through as lots of small files are always a pain in transferring and especially on rotating disks, Nick
The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out.
Maybe that helps,
Nick
-- Fundamentally an organism has conscious mental states if and only if there is something that it is like to be that organism--something it is like for the organism. - Thomas Nagel
On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote:
The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore.
ReiserFS was not reliable. I certainly stopped using it long before the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%.
I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long.
Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. -- Len Sorensen
On 5/5/20 11:27 PM, Lennart Sorensen wrote:
The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. ReiserFS was not reliable. I certainly stopped using it long before
On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%.
I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. I mentioned to check /proc/meminfo as well to look at cache in case you missed that.
Nick
-- Fundamentally an organism has conscious mental states if and only if there is something that it is like to be that organism--something it is like for the organism. - Thomas Nagel
On 5/6/20 12:37 AM, Nicholas Krause wrote: [snip]
Well, does the system have enough ram? That is something that often
isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. I mentioned to check /proc/meminfo as well to look at cache in case you missed that.
You did mention that and it is on my plate to research the amount of ram that is suggested for this kind of application. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On 5/5/20 11:27 PM, Lennart Sorensen wrote:
The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. ReiserFS was not reliable. I certainly stopped using it long before
On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup.
This is a nice example of YMMV. I moved my critical data from ext2/3 because they would trash my data and corrupt the whole filesystem under certain cases. I had much better luck with Reiser even in the face of a couple of system crashes.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%.
I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well.
The system has 32G and runs nothing but gluster. Not sure if that is enough ram and will require a bit of researching. It looks like the system has about 15G of cached data. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
ZFS is another option. And it handles delta-backups very easily. ../Dave On May 5, 2020, 11:27 PM -0400, Lennart Sorensen via talk <talk@gtalug.org>, wrote:
On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote:
The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore.
ReiserFS was not reliable. I certainly stopped using it long before the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup.
I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%.
I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long.
Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well.
-- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
On 5/6/20 7:25 AM, David Mason via talk wrote:
ZFS is another option. And it handles delta-backups very easily.
../Dave
I have been following ZFS on and off and it looks interesting but I am kind of stuck because it is not included in Centos/RH which is a requirement in this case. It would be interesting to test it at scale just to see how well is does work. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On Wed, May 06, 2020 at 07:25:29AM -0400, David Mason wrote:
ZFS is another option. And it handles delta-backups very easily.
It is however not the XFS that glusterfs says to use. If glusterfs is involved, XFS really seems to be the only option. -- Len Sorensen
On 5/6/20 10:18 AM, Lennart Sorensen via talk wrote:
On Wed, May 06, 2020 at 07:25:29AM -0400, David Mason wrote:
ZFS is another option. And it handles delta-backups very easily. It is however not the XFS that glusterfs says to use. If glusterfs is involved, XFS really seems to be the only option.
There is Gluster hosted documentation on how to use ZFS with Gluster but it not the preferred option. Putting Gluster on a thinlvm is the Gluster way for doing snapshots. I have found a package called wyng-backup(https://github.com/tasket/wyng-backup). This may solve my problem but it would require me to change all my volumes to thin provisioning. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On Wed, 6 May 2020 07:25:29 -0400 David Mason via talk <talk@gtalug.org> wrote:
ZFS is another option. And it handles delta-backups very easily.
David, How do you recover stuff from delta backups? You have to figure which backup the file or directory is in, right? My backup recoveries, admittedly here at home, consist of me recovering files I stupidly modified incorrectly, and some accidentally deleted directories. In the case of the directories, I noticed the problem several years after I did it. Since I back up everything, all I had to do was pull out the backup Blu-ray from shortly after I worked on the directory. -- Howard Gibson hgibson@eol.ca jhowardgibson@gmail.com http://home.eol.ca/~hgibson
On Wed, 2020/05/06 10:38:29AM -0400, Howard Gibson via talk <talk@gtalug.org> wrote: | > ZFS is another option. And it handles delta-backups very easily. | | How do you recover stuff from delta backups? You have to figure which backup the file or directory is in, right? Remember that snapshots, like RAID, are not actually backups, unless they are on a different machine, in a different place. ZFS makes it easy: You can browse through snapshots for /mypool/myfs by looking in /mypool/myfs/.zfs/snapshot and if your ZFS snapshots are named using dates, easy peasy to choose when. You can also brute force and find /mypool/myfs/.zfs/snapshot -name 'myfile.tex' -ls and find what's there. You can use "zfs rollback" to revert to a snapshot. You can use "zfs send ... | zfs recv ..." to copy a specific snapshot (or group of snapshots) to another pool, system, etc. And of course, when you create a snapshot, you could create your own index listing of what's there for easy grepping. ZFS is great. You can still (and likely should) continue to backup to Blu-ray, but ZFS will make sure your files don't rot in place unnoticed. Hope that helps, cheers John
participants (7)
-
Alvin Starr -
David Mason -
Howard Gibson -
John Sellens -
lsorense＠csclub.uwaterloo.ca -
Nicholas Krause -
nick