Which Distro is Best for Running a ZFS-on-Linux Fileserver.
Hello All, Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network. Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times. So, which Distro are the favourite for Running ZFS-on-Linux? Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool? Thank You in Advance for your Input. Regards, Amos
On Fri, Aug 24, 2018, 2:27 PM right.maple.nut via talk <talk@gtalug.org> wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
I think that if I were certain I wanted a ZFS machine on my network, I'd look into one of the Illumos variants, most likely OmniOS. https://omniosce.org/ That is a fork of Solaris, which is where ZFS is really native. All the other implementations of ZFS are ports, and likely a bit less satisfactory. That obviously isn't Linux. But it shouldn't be ridiculously unfamiliar.
On 2018-08-24 3:43 p.m., Christopher Browne via talk wrote:
On Fri, Aug 24, 2018, 2:27 PM right.maple.nut via talk <talk@gtalug.org <mailto:talk@gtalug.org>> wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
I think that if I were certain I wanted a ZFS machine on my network, I'd look into one of the Illumos variants, most likely OmniOS. https://omniosce.org/
That is a fork of Solaris, which is where ZFS is really native.
All the other implementations of ZFS are ports, and likely a bit less satisfactory.
That obviously isn't Linux. But it shouldn't be ridiculously unfamiliar.
I second Chris: most of the commands will be familiar, although sometimes the names will merely be reminiscent of one another: strace vs truss, for example. I used to be in the porting business, and Solaris <-> Linux was dead easy. --dave -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
| From: right.maple.nut via talk <talk@gtalug.org> | Hello All, Like the Subject Line says, I'm setting up a ZFS File Server | for my Home Network. Given that I will have to go to the trouble of | setting up the Distro and Migrating the Linux Install to ZFS Root, I | don't want to have to do this too many times. So, which Distro are the | favourite for Running ZFS-on-Linux? Also, is there such a thing as a | Linux Distro that is smart enough to give you a choice if you are | willing to use non-GPL'ed code in the Installer, so that I can just | Install Directly on a ZFS Pool? Thank You in Advance for your Input. | Regards, Amos Why not run a *BSD? Or something Solaris-related? Open Indiana???? - They more naturally run ZFS. - there is no clash-of-licenses drama. - the code is probably more tested. - there are even out of the box fileserver distros based on *BSD. - TrueOS, FreeNAS? Is there something that you don't like about UNIX (as opposed to Linux) for a file server?
I am interested in this question too. I currently am running on Debian 7.7 but I’m not sure I can upgrade to a more current version, which is frustrating because I want to install Java (to run a Minecraft server) and (when I tried, so while back) I couldn’t get it to install because I had such an old version of Debian. So I was thinking of FreeBSD, but OmniOS looks pretty interesting. I won’t be doing any upgrade for several months (because as far as I can see, I need to install a new ZFS system and copy it over. I can’t risk loosing this system). It would be good to have a backup, anyway. I could try this on a virtual system, but I don’t know long it would take to back-up over the network. ../Dave On Aug 24, 2018, 3:51 PM -0400, D. Hugh Redelmeier via talk <talk@gtalug.org>, wrote:
| From: right.maple.nut via talk <talk@gtalug.org>
| Hello All, Like the Subject Line says, I'm setting up a ZFS File Server | for my Home Network. Given that I will have to go to the trouble of | setting up the Distro and Migrating the Linux Install to ZFS Root, I | don't want to have to do this too many times. So, which Distro are the | favourite for Running ZFS-on-Linux? Also, is there such a thing as a | Linux Distro that is smart enough to give you a choice if you are | willing to use non-GPL'ed code in the Installer, so that I can just | Install Directly on a ZFS Pool? Thank You in Advance for your Input. | Regards, Amos
Why not run a *BSD?
Or something Solaris-related? Open Indiana????
- They more naturally run ZFS.
- there is no clash-of-licenses drama.
- the code is probably more tested.
- there are even out of the box fileserver distros based on *BSD.
- TrueOS, FreeNAS?
Is there something that you don't like about UNIX (as opposed to Linux) for a file server? --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
| From: David Mason via talk <talk@gtalug.org> | I am interested in this question too. I currently am running on Debian | 7.7 but I’m not sure I can upgrade to a more current version, which is | frustrating because I want to install Java (to run a Minecraft server) | and (when I tried, so while back) I couldn’t get it to install because I | had such an old version of Debian. Back in 2017 March 12 you mentioned a problem "Updating Wheezy to Jessie". Did you try Stewart's and Lennart's suggestions? This is exactly the kind of problem that a debian local users group might address. Recently I suggested that GTALUG could function as a debian LUG. I guess that here we have a test of this idea. | So I was thinking of FreeBSD, but OmniOS looks pretty interesting. I | won’t be doing any upgrade for several months (because as far as I can | see, I need to install a new ZFS system and copy it over. I can’t risk | loosing this system). It would be good to have a backup, anyway. I could | try this on a virtual system, but I don’t know long it would take to | back-up over the network. Even for the debian upgrade attempt it might be good to have a backup. If one falls behind on updates, there is a greater incentive to stand still and fall further behind. A vicious circle. I've had that happen. After enough time, the old hardware becomes obsolete too and it makes total system rebuild more sensible than update. In my most recent example, I jumped about 15 years ahead in hardware.
On Aug 25, 2018, 6:21 AM -0400, D. Hugh Redelmeier via talk <talk@gtalug.org>, wrote:
| From: David Mason via talk <talk@gtalug.org>
| I am interested in this question too. I currently am running on Debian | 7.7 but I’m not sure I can upgrade to a more current version, which is | frustrating because I want to install Java (to run a Minecraft server) | and (when I tried, so while back) I couldn’t get it to install because I | had such an old version of Debian.
Back in 2017 March 12 you mentioned a problem "Updating Wheezy to Jessie". Did you try Stewart's and Lennart's suggestions?
I put a little time into it, but didn’t get to success. Unfortunately, life intervened. It’s still important, but as you say, as you fall behind, it becomes more difficult to leap-frog into the present.
This is exactly the kind of problem that a debian local users group might address. Recently I suggested that GTALUG could function as a debian LUG. I guess that here we have a test of this idea.
I think it already does. What did you have in mind beyond the mailing list?
Even for the debian upgrade attempt it might be good to have a backup.
Yes, I know. :-)
After enough time, the old hardware becomes obsolete too and it makes total system rebuild more sensible than update. In my most recent example, I jumped about 15 years ahead in hardware.
This system is <5 years old, and at the time was kind-of leading edge. so I’m not worried about that. It’s a 4.4Tb raidz2 at 64% and has performed flawlessly. Unfortunately I don’t really have the time to do any serious digging right now, either. How do others backup their ZFS systems? Getting a 4T external drive doesn’t seem like the best plan, but maybe there isn’t any other choice. ../Dave
| From: David Mason via talk <talk@gtalug.org> | On Aug 25, 2018, 6:21 AM -0400, D. Hugh Redelmeier via talk <talk@gtalug.org>, wrote: | > | From: David Mason via talk <talk@gtalug.org> | > This is exactly the kind of problem that a debian local users group might | > address. Recently I suggested that GTALUG could function as a debian LUG. | > I guess that here we have a test of this idea. | | I think it already does. What did you have in mind beyond the mailing list? You asked a year ago, got replies, hit problems, but didn't post them to the list. Maybe you didn't expect the list to be of more help. This isn't an accusation, only an observation. | It’s a 4.4Tb raidz2 at 64% and has performed flawlessly. Unfortunately I | don’t really have the time to do any serious digging right now, either. I understand all too well. | How do others backup their ZFS systems? Getting a 4T external drive | doesn’t seem like the best plan, but maybe there isn’t any other choice. I'm not a ZFS user, but I'll throw out a guess: - take a snapshot (that may be the wrong terminology) - use a dump(8) command that matches zfs.
On Sat, 25 Aug 2018 at 14:21, David Mason via talk <talk@gtalug.org> wrote:
On Aug 25, 2018, 6:21 AM -0400, D. Hugh Redelmeier via talk < talk@gtalug.org>, wrote:
| From: David Mason via talk <talk@gtalug.org>
| I am interested in this question too. I currently am running on Debian | 7.7 but I’m not sure I can upgrade to a more current version, which is | frustrating because I want to install Java (to run a Minecraft server) | and (when I tried, so while back) I couldn’t get it to install because I | had such an old version of Debian.
Back in 2017 March 12 you mentioned a problem "Updating Wheezy to Jessie". Did you try Stewart's and Lennart's suggestions?
I put a little time into it, but didn’t get to success. Unfortunately, life intervened. It’s still important, but as you say, as you fall behind, it becomes more difficult to leap-frog into the present.
This is exactly the kind of problem that a debian local users group might address. Recently I suggested that GTALUG could function as a debian LUG. I guess that here we have a test of this idea.
I think it already does. What did you have in mind beyond the mailing list?
Even for the debian upgrade attempt it might be good to have a backup.
Yes, I know. :-)
After enough time, the old hardware becomes obsolete too and it makes total system rebuild more sensible than update. In my most recent example, I jumped about 15 years ahead in hardware.
This system is <5 years old, and at the time was kind-of leading edge. so I’m not worried about that. It’s a 4.4Tb raidz2 at 64% and has performed flawlessly. Unfortunately I don’t really have the time to do any serious digging right now, either.
How do others backup their ZFS systems? Getting a 4T external drive doesn’t seem like the best plan, but maybe there isn’t any other choice.
Actually, that sounds like a really good plan. In fact, buy two so you can do rotating backups. Think about your alternatives - about the only one that occurs to me is a tape drive. There used to be consumer-grade tape backups, but they don't exist anymore and I'd argue this is no longer a viable solution outside the data centre. Buying external hard drives is a really good idea: they're dirt cheap (at least compared to the alternative - failure of your primary). The first backup will take a day or so, but after that you use rsync (and/or rsnapshot or similar) and the backups are likely to run 30 minutes to 2 hours depending on the thrash on your drive. If you're going to use something that does differential backups, make sure the backup drive is larger than the source drive (rather than the default - we all assume "the same size" is okay). I use 2.5" spinning external 4TB drives as backups, one of which stays at my parent's place and gets exchanged approximately weekly. This may not be the right answer for you, but you seem to think the data on that big machine of yours is important: stop and think about how things would be if it failed. And then work out a backup system. -- Giles https://www.gilesorr.com/ gilesorr@gmail.com
On 2018-08-27 09:24 AM, Giles Orr via talk wrote:
On Sat, 25 Aug 2018 at 14:21, David Mason via talk <talk@gtalug.org <mailto:talk@gtalug.org>> wrote: This system is <5 years old, and at the time was kind-of leading edge. so I’m not worried about that. It’s a 4.4Tb raidz2 at 64% and has performed flawlessly. Unfortunately I don’t really have the time to do any serious digging right now, either.
How do others backup their ZFS systems? Getting a 4T external drive doesn’t seem like the best plan, but maybe there isn’t any other choice.
In my case I built a secondary NAS and disk array, and do regular 'zfs snapshots' and 'zfs sends'. In recent history I've started using zfs-snap-manager to automate that. https://github.com/khenderick/zfs-snap-manager It's a rather coarse tool... doesn't support automate snapshots more frequent then once a day, but will happily send over any you've made manually (via a cron job or alternative method). Currently the developer has only packaged it for Arch. But I've built an rpm spec file for it. Attached.
Actually, that sounds like a really good plan. In fact, buy two so you can do rotating backups. Think about your alternatives - about the only one that occurs to me is a tape drive. There used to be consumer-grade tape backups, but they don't exist anymore and I'd argue this is no longer a viable solution outside the data centre.
Buying external hard drives is a really good idea: they're dirt cheap (at least compared to the alternative - failure of your primary).
I agree with Giles. If you don't want to drop the coin on a second NAS, this is a very usable strategy. Get a 6 or even 8TB disk, format it as a ZFS pool and turn on zfs's block compression, and set copies to '2'. zfs set compression=lz4 <pool> zfs set copies=2 <pool> Setting a number of copies, is normally not useful for a multi-disk array, as the copies can end up on the same disk. But on a single disk, they are an insurance policy against bad sectors. Then you just zfs send your snapshots to it. I regularly use this as a local backup strategy with my work laptops. -- Scott Sullivan
OK, so I have an 8TB Seagate USB disk and have created a zpool on it called backup1. My main pool is called tank. I tried: : ~ ; sudo zfs snapshot -r tank@2018-08-31 : ~ ; sudo zfs list NAME USED AVAIL REFER MOUNTPOINT backup1 508K 7.14T 136K /backup1 tank 1.66T 916G 412K /tank tank/audio 12.1G 916G 12.1G /audio tank/cvs 32.7M 916G 32.7M /tank/cvs tank/etc 18.1M 916G 18.1M /tank/etc tank/home 531G 916G 531G /home : ~ ; sudo zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT tank@2018-08-31 0 - 412K - tank/audio@2018-08-31 0 - 12.1G - tank/cvs@2018-08-31 0 - 32.7M - tank/etc@2018-08-31 0 - 18.1M - tank/home@2018-08-31 0 - 531G - and now I try (after some research): : ~ ; sudo zfs send -R tank@2018-08-31 | sudo zfs recv -vd backup1 cannot receive new filesystem stream: destination 'backup1' exists must specify -F to overwrite it warning: cannot send 'tank@2018-08-31': Broken pipe Any quick help? Thanks ../Dave On Tue, 28 Aug 2018 at 17:33, Scott Sullivan via talk <talk@gtalug.org> wrote:
On 2018-08-27 09:24 AM, Giles Orr via talk wrote:
On Sat, 25 Aug 2018 at 14:21, David Mason via talk <talk@gtalug.org <mailto:talk@gtalug.org>> wrote: This system is <5 years old, and at the time was kind-of leading edge. so I’m not worried about that. It’s a 4.4Tb raidz2 at 64% and has performed flawlessly. Unfortunately I don’t really have the time to do any serious digging right now, either.
How do others backup their ZFS systems? Getting a 4T external drive doesn’t seem like the best plan, but maybe there isn’t any other choice.
In my case I built a secondary NAS and disk array, and do regular 'zfs snapshots' and 'zfs sends'. In recent history I've started using zfs-snap-manager to automate that.
https://github.com/khenderick/zfs-snap-manager
It's a rather coarse tool... doesn't support automate snapshots more frequent then once a day, but will happily send over any you've made manually (via a cron job or alternative method).
Currently the developer has only packaged it for Arch. But I've built an rpm spec file for it. Attached.
Actually, that sounds like a really good plan. In fact, buy two so you can do rotating backups. Think about your alternatives - about the only one that occurs to me is a tape drive. There used to be consumer-grade tape backups, but they don't exist anymore and I'd argue this is no longer a viable solution outside the data centre.
Buying external hard drives is a really good idea: they're dirt cheap (at least compared to the alternative - failure of your primary).
I agree with Giles. If you don't want to drop the coin on a second NAS, this is a very usable strategy. Get a 6 or even 8TB disk, format it as a ZFS pool and turn on zfs's block compression, and set copies to '2'.
zfs set compression=lz4 <pool> zfs set copies=2 <pool>
Setting a number of copies, is normally not useful for a multi-disk array, as the copies can end up on the same disk. But on a single disk, they are an insurance policy against bad sectors.
Then you just zfs send your snapshots to it. I regularly use this as a local backup strategy with my work laptops.
-- Scott Sullivan --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 2018-08-31 05:15 PM, David Mason wrote:
OK, so I have an 8TB Seagate USB disk and have created a zpool on it called backup1. My main pool is called tank. I tried:
: ~ ; sudo zfs snapshot -r tank@2018-08-31 : ~ ; sudo zfs list NAME USED AVAIL REFER MOUNTPOINT backup1 508K 7.14T 136K /backup1 tank 1.66T 916G 412K /tank tank/audio 12.1G 916G 12.1G /audio tank/cvs 32.7M 916G 32.7M /tank/cvs tank/etc 18.1M 916G 18.1M /tank/etc tank/home 531G 916G 531G /home
: ~ ; sudo zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT tank@2018-08-31 0 - 412K - tank/audio@2018-08-31 0 - 12.1G - tank/cvs@2018-08-31 0 - 32.7M - tank/etc@2018-08-31 0 - 18.1M - tank/home@2018-08-31 0 - 531G - and now I try (after some research):
: ~ ; sudo zfs send -R tank@2018-08-31 | sudo zfs recv -vd backup1 cannot receive new filesystem stream: destination 'backup1' exists must specify -F to overwrite it warning: cannot send 'tank@2018-08-31': Broken pipe
Any quick help?
Thanks ../Dave
That's expected Dave. Because backup1 is a new filesystem, it is inherently not a decedent of of your source zfs data set and is so a name collision. So -F to force is perfectly reasonable to remove the empty dataset and replicate your source into the pool. In following backups, you'd use the last common snapshot and most recent snapshot as arguments to 'zfs send -I'. An example from my own shell history: zfs send -I jarvis-charlie/backups/failfast.revident.ca@20180325_224731-0400 jarvis-charlie/backups/failfast.revident.ca@20180401_121435-0400 | ssh root@example.someplace.revident.ca "zfs receive -d jarvis-dr" -- Scott Sullivan
Hey Amos, Why not FreeBSD? This is one of the first projects to get ZFS ported, in my experience it's the closest thing to general-purpose linux distro in BSD land that you can get. All the GNU tools are available for FreeBSD and shell commands aren't that arcane. If you're only going to use the machine for hosting ZFS it might be worth it. Alex. On 2018-08-24 2:26 p.m., right.maple.nut via talk wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
Thank You in Advance for your Input.
Regards, Amos
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
Having read through the thread to date, I'm actually a little disappointed at the number of linux users pushing towards a Solaris or BSD for ZFS. My primary File servers (4 of them) are all using ZFS for their data partitions. Amos, ## Couple of Answers to your questions A) Disto? I regularly run ZFS on CentOS and Fedora on a mix of SSDs and HDDs of both the internal and external varieties. Fedora has some caveats, only in that sometimes the kernel releases get ahead of what the ZFS on linux team will support. And it's just a matter of waiting on a working kernel zfs combination a week or two for them to catch up. But frankly, just pick your favorite distro and follow the relevant getting started guide. https://zfsonlinux.org/ B) Distro with ZFS root support (at install time)? No distro install supports this yet as I've seen. Although the do it yourself ubuntu guide is lengthy, but very well detailed. https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS Arch also support ZFS root, but their installation is all largely manual to begin with. https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_ZFS ## Couple of my own Questions 1) Why root (/) on ZFS, what is your use case / risk your trying to mitigate? On 2018-08-24 02:26 PM, right.maple.nut via talk wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
Thank You in Advance for your Input.
Regards, Amos
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Scott Sullivan
At the risk of poking the bear. Why use ZFS at all? ext4,XFS+LVM will do most all of the same things and from what I have read ZFS is slower than ext4,XFS and BTRFS. On 08/28/2018 05:18 PM, Scott Sullivan via talk wrote:
Having read through the thread to date, I'm actually a little disappointed at the number of linux users pushing towards a Solaris or BSD for ZFS.
My primary File servers (4 of them) are all using ZFS for their data partitions.
Amos,
## Couple of Answers to your questions
A) Disto?
I regularly run ZFS on CentOS and Fedora on a mix of SSDs and HDDs of both the internal and external varieties. Fedora has some caveats, only in that sometimes the kernel releases get ahead of what the ZFS on linux team will support. And it's just a matter of waiting on a working kernel zfs combination a week or two for them to catch up.
But frankly, just pick your favorite distro and follow the relevant getting started guide.
B) Distro with ZFS root support (at install time)?
No distro install supports this yet as I've seen. Although the do it yourself ubuntu guide is lengthy, but very well detailed.
https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS
Arch also support ZFS root, but their installation is all largely manual to begin with.
https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_ZFS
## Couple of my own Questions
1) Why root (/) on ZFS, what is your use case / risk your trying to mitigate?
On 2018-08-24 02:26 PM, right.maple.nut via talk wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
Thank You in Advance for your Input.
Regards, Amos
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
I haven’t done a comparison lately, but used LVM for years, and liked it, but ZFS is so much more flexible: resize filesystems, snapshots, raidz2, sending to another ZFS system (for seamless backups and redundancy), auto-silvering, plug-and-play. ../Dave On Aug 28, 2018, 9:18 PM -0400, Alvin Starr via talk <talk@gtalug.org>, wrote:
At the risk of poking the bear.
Why use ZFS at all?
ext4,XFS+LVM will do most all of the same things and from what I have read ZFS is slower than ext4,XFS and BTRFS.
On 08/28/2018 05:18 PM, Scott Sullivan via talk wrote:
Having read through the thread to date, I'm actually a little disappointed at the number of linux users pushing towards a Solaris or BSD for ZFS.
My primary File servers (4 of them) are all using ZFS for their data partitions.
Amos,
## Couple of Answers to your questions
A) Disto?
I regularly run ZFS on CentOS and Fedora on a mix of SSDs and HDDs of both the internal and external varieties. Fedora has some caveats, only in that sometimes the kernel releases get ahead of what the ZFS on linux team will support. And it's just a matter of waiting on a working kernel zfs combination a week or two for them to catch up.
But frankly, just pick your favorite distro and follow the relevant getting started guide.
B) Distro with ZFS root support (at install time)?
No distro install supports this yet as I've seen. Although the do it yourself ubuntu guide is lengthy, but very well detailed.
https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS
Arch also support ZFS root, but their installation is all largely manual to begin with.
https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_ZFS
## Couple of my own Questions
1) Why root (/) on ZFS, what is your use case / risk your trying to mitigate?
On 2018-08-24 02:26 PM, right.maple.nut via talk wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
Thank You in Advance for your Input.
Regards, Amos
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
Hello All, Thank you for your Feedback and Discussion. FreeBSD is a good suggestion but I don't want to wander away from what I know and as Scott pointed out, this is a Linux User Group ... Also, once I get More RAM, I will want to Consolidate Services running on other machines on to the NAS. Currently, I have 8 GB of RAM, which my reading says should be enough to support a 4 TB ZFS Pool and the Necessary Samba configuration. At the moment, my Network copy speed appears to be limited to about 25 MB/s but I suspect that this is because my Shares are on a Windows Machine with a PCI SATA Card. So, if the NAS can do better, I'll declare Victory ... Scott, My reasoning for / on ZFS is pretty Simple ... the machine that is becoming my first NAS only has 4 SATA Ports, so I can't afford to Waste one on a boot drive. For Distro, I think I'll go with Fedora, as long as the / on ZFS guide is sufficiently detailed. Thank You All, Amos Sent from my android device. -----Original Message----- From: Scott Sullivan via talk <talk@gtalug.org> To: talk@gtalug.org Sent: Tue, 28 Aug 2018 5:19 PM Subject: Re: [GTALUG] Which Distro is Best for Running a ZFS-on-Linux Fileserver. Having read through the thread to date, I'm actually a little disappointed at the number of linux users pushing towards a Solaris or BSD for ZFS. My primary File servers (4 of them) are all using ZFS for their data partitions. Amos, ## Couple of Answers to your questions A) Disto? I regularly run ZFS on CentOS and Fedora on a mix of SSDs and HDDs of both the internal and external varieties. Fedora has some caveats, only in that sometimes the kernel releases get ahead of what the ZFS on linux team will support. And it's just a matter of waiting on a working kernel zfs combination a week or two for them to catch up. But frankly, just pick your favorite distro and follow the relevant getting started guide. https://zfsonlinux.org/ B) Distro with ZFS root support (at install time)? No distro install supports this yet as I've seen. Although the do it yourself ubuntu guide is lengthy, but very well detailed. https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS Arch also support ZFS root, but their installation is all largely manual to begin with. https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_ZFS ## Couple of my own Questions 1) Why root (/) on ZFS, what is your use case / risk your trying to mitigate? On 2018-08-24 02:26 PM, right.maple.nut via talk wrote:
Hello All,
Like the Subject Line says, I'm setting up a ZFS File Server for my Home Network.
Given that I will have to go to the trouble of setting up the Distro and Migrating the Linux Install to ZFS Root, I don't want to have to do this too many times.
So, which Distro are the favourite for Running ZFS-on-Linux?
Also, is there such a thing as a Linux Distro that is smart enough to give you a choice if you are willing to use non-GPL'ed code in the Installer, so that I can just Install Directly on a ZFS Pool?
Thank You in Advance for your Input.
Regards, Amos
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Scott Sullivan --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 2018-08-29 11:43 PM, Amos H. Weatherill wrote:
Scott,
My reasoning for / on ZFS is pretty Simple ... the machine that is becoming my first NAS only has 4 SATA Ports, so I can't afford to Waste one on a boot drive.
Recommended best Practice is to use ZFS with whole disks. That said, most of the arguments for that are 'because the manual says so', 'because zfs datasets are far more flexible then partitions' and references to Solairs taking advantage of disk caches. I throw that all out the windows in favor of doing at rest encryption, with whole luks partitions(*). My more practical argument is choice of MBR vs GUID partitioning. The latter is just cleaner (and the default when ZFS manages the disk), and works well with large disks (>2TB). But if your booting from that disk, you either need to be: "BIOS / CSM" + MBR + /boot or UEFI + GUID + "biosboot (partition)" + /boot Either of those makes for some lopsided partitioning, compared to the remainder of your data disks. A work around is to use a USB drive for your /boot. But in general your creating a more complex setup to maintain either way. Not knowing what hardware your using, if you have PCIe slots additional sata ports can be had for a low a $10/port. I've been using the Syba / IOCrest cards for a variety of needs, including ZFS arrays without issue. https://www.newegg.ca/Product/Product.aspx?Item=N82E16816124064
For Distro, I think I'll go with Fedora, as long as the / on ZFS guide is sufficiently detailed.
Fedora was not one of the ones I listed as having a guide to do rootfs on ZFS. If you found one, can you post the link? I'd also not recommend fedora in general for a NAS. CentOS would be a more dependable choice. LTS Ubuntu would be more reasonable as they ship(**) ZFS and support rootfs on it. === * Native encryption in ZFS was added after the OpenZFS split from Sun/Oracle. So work to re-added it has been happening for a while. We're likely to see a stable version in the v0.8.x series. ** This is due to their adoption of a minority legal opinion about compatibility of the CDDL and GPL licenses that has not been tested in court. https://blog.ubuntu.com/2016/02/18/zfs-licensing-and-linux https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ -- Scott Sullivan
If you’re building 4TB, I’t recommend raidz2 which doesn’t give you much storage (4 x 1TB disks would give you <2TB of user storage). I have 5 x 1TB raidz2 which gives me 2.82TB of user storage. This may have changed, but when I went to ZFS you couldn’t add more disks and change the structure, you could only enlarge the disks - so if you went for a 5-disk raidz2, you were stuck with a 5-disk raidz2. I really hope this has changed, but I haven’t researched it. The reason to go with raidz2 as the pool gets larger is that with raidz if a disk dies, while it rebuilds on the replacement disk, you have no redundancy, so any error on one of the other disks will be replicated. So I’d recommend you get an additional SATA card, and then you don’t need to do ZFS on /, and you could add another drive to get better cost-effectiveness from your ZFS. I have / on a moderate-sized SSD and 5 hard drives in a ZFS pool. One reason to use not-whole-disks for ZFS is that you can migrate to larger disks as time goes on. I started out with 500GB (if I remember correctly) and was able to move to 1TB and copy the partitions. Scott knows much more about this than I, but it may (now) be that (with a raidz2) you can enlarge by simply removing a xGB disk and replacing it with a 2xGB drive, let it rebuild, then replace the next one. The problem with that is you are replacing the disks, so it’s not very incremental. So if you can enlarge by adding an additional drive, that is substantially better. ../Dave On Aug 30, 2018, 9:34 AM -0400, Scott Sullivan via talk <talk@gtalug.org>, wrote:
On 2018-08-29 11:43 PM, Amos H. Weatherill wrote:
Scott,
My reasoning for / on ZFS is pretty Simple ... the machine that is becoming my first NAS only has 4 SATA Ports, so I can't afford to Waste one on a boot drive.
Recommended best Practice is to use ZFS with whole disks. That said, most of the arguments for that are 'because the manual says so', 'because zfs datasets are far more flexible then partitions' and references to Solairs taking advantage of disk caches. I throw that all out the windows in favor of doing at rest encryption, with whole luks partitions(*).
My more practical argument is choice of MBR vs GUID partitioning. The latter is just cleaner (and the default when ZFS manages the disk), and works well with large disks (>2TB).
But if your booting from that disk, you either need to be: "BIOS / CSM" + MBR + /boot or UEFI + GUID + "biosboot (partition)" + /boot
Either of those makes for some lopsided partitioning, compared to the remainder of your data disks. A work around is to use a USB drive for your /boot. But in general your creating a more complex setup to maintain either way.
Not knowing what hardware your using, if you have PCIe slots additional sata ports can be had for a low a $10/port.
I've been using the Syba / IOCrest cards for a variety of needs, including ZFS arrays without issue.
https://www.newegg.ca/Product/Product.aspx?Item=N82E16816124064
For Distro, I think I'll go with Fedora, as long as the / on ZFS guide is sufficiently detailed.
Fedora was not one of the ones I listed as having a guide to do rootfs on ZFS. If you found one, can you post the link?
I'd also not recommend fedora in general for a NAS. CentOS would be a more dependable choice. LTS Ubuntu would be more reasonable as they ship(**) ZFS and support rootfs on it.
=== * Native encryption in ZFS was added after the OpenZFS split from Sun/Oracle. So work to re-added it has been happening for a while. We're likely to see a stable version in the v0.8.x series.
** This is due to their adoption of a minority legal opinion about compatibility of the CDDL and GPL licenses that has not been tested in court. https://blog.ubuntu.com/2016/02/18/zfs-licensing-and-linux https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/
-- Scott Sullivan --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
participants (10)
-
Alex Volkov -
Alvin Starr -
Amos H. Weatherill -
Christopher Browne -
D. Hugh Redelmeier -
David Collier-Brown -
David Mason -
Giles Orr -
right.maple.nut
-
Scott Sullivan