I have a decade-old server that gets no love, and is running a very old Debian and ZFS (Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux) on a bunch of spinning rust. zpool is producing strange status (i.e. resilvering on multiple drives, making no progress, etc.). So I want to move things off to something more stable. So I want something with at least 5 SST drives for ZFS and a bootable drive not on ZFS (ideally). PCIe would be nice (GBs/sec) but 6 or 8 SATA drives would still be a lot faster than the existing. I'd like it to use DDR5 ECC. I don't need much performance, and I'd like it to be relatively low power. At least a GB Ethernet. I don't need, nor care about, GPUs - at least not at this point. There are a truly mindboggling number of options, but most of the ones that meet my I/O requirements are very high-performance and power-hungry. And almost nothing supports ECC properly On the Intel side there are LGA1851 socket systems. A reasonably-priced one is https://www.canadacomputers.com/en/intel-motherboards/264441/gigabyte-z890-a... but it doesn't do ECC properly. On the AMD side there are sTR5 socket systems. A less reasonably priced one is https://www.newegg.ca/gigabyte-trx50-aero-d-extended-atx-amd-trx50-am5/p/N82... but it at least does ECC. Then we get to the CPUs - and even more expensive... I'm not afraid of spending money for a system that will hopefully support our needs for more than another decade (8 2Tb drives is about $1600, but I don't really want to spend another $3000+ for MB+CPU). If I do have to go that high-performance, I'm thinking about Proxmox, but not completely sure - any thoughts on that, too, would be appreciated. Thanks, ../Dave
From: David Mason via talk <talk@gtalug.org>
So I want something with at least 5 SST drives for ZFS and a bootable drive not on ZFS (ideally).
SST? Do you mean SSD?
PCIe would be nice (GBs/sec) but 6 or 8 SATA drives would still be a lot faster than the existing.
I'd like it to use DDR5 ECC. I don't need much performance, and I'd like it to be relatively low power. At least a GB Ethernet. I don't need, nor care about, GPUs - at least not at this point.
There are a truly mindboggling number of options, but most of the ones that meet my I/O requirements are very high-performance and power-hungry. And almost nothing supports ECC properly
On the Intel side there are LGA1851 socket systems. A reasonably-priced one is https://www.canadacomputers.com/en/intel-motherboards/264441/gigabyte-z890-a... but it doesn't do ECC properly.
What do you mean by "properly"? At one point I believed that IBM's phrase "chip kill" was relevant. ECC normally detects single-bit errors in a word but in many configurations, one dead chip will take out more than one bit in a word.
On the AMD side there are sTR5 socket systems. A less reasonably priced one is https://www.newegg.ca/gigabyte-trx50-aero-d-extended-atx-amd-trx50-am5/p/N82... but it at least does ECC.
Thread Ripper seems like the opposite of what you want for a cool and quiet home server. I think that (some?) ordinary Ryzen's support ECC but that not many firmwares do. And perhaps not many chipsets do. Here's a kind of old link to Asus: <https://www.asus.com/support/faq/1045186/> There are some weird old Atom variants that support ECC and a lot of SATA. The "C Series" was meant for servers. Here's one chip: <https://ark.intel.com/content/www/us/en/ark/products/97929/intel-atom-processor-c3538-8m-cache-up-to-2-10-ghz.html> - ECC - 12 SATA! I wonder if any refurbished boxes are available cheap. Maybe servethehome.com folks would be knowledgable. --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
D. Hugh Redelmeier via talk wrote on 2024-11-05 15:29:
- 12 SATA!
Can't one get a separate PCIe card that supports extra SATA connections, off-loading some traffic to a second DMA channel (if that is the right terminology)? Just a thought (and a question of when would that be worthwhile?)...
Thanks for the reply, Hugh. On Tue, 5 Nov 2024 at 18:29, D. Hugh Redelmeier <hugh@mimosa.com> wrote:
From: David Mason via talk <talk@gtalug.org>
So I want something with at least 5 SST drives for ZFS and a bootable drive not on ZFS (ideally).
SST? Do you mean SSD?
Yes, SSD.
On the Intel side there are LGA1851 socket systems. A reasonably-priced one
is
https://www.canadacomputers.com/en/intel-motherboards/264441/gigabyte-z890-a...
but it doesn't do ECC properly.
What do you mean by "properly"?
As I understand it, all DR5 does single-bit error correction, but it is up to the board/CPU to trap on multi-bit errors. I don't know if DDR5 writes bback the corrected bit, or if that's up to the other components.
At one point I believed that IBM's phrase "chip kill" was relevant. ECC normally detects single-bit errors in a word but in many configurations, one dead chip will take out more than one bit in a word.
All I know is what I said above.
On the AMD side there are sTR5 socket systems. A less reasonably priced one is
https://www.newegg.ca/gigabyte-trx50-aero-d-extended-atx-amd-trx50-am5/p/N82...
but it at least does ECC.
Thread Ripper seems like the opposite of what you want for a cool and quiet home server.
Yes, but finding ECC-aware systems is challenging.
I think that (some?) ordinary Ryzen's support ECC but that not many firmwares do. And perhaps not many chipsets do. Here's a kind of old link to Asus: <https://www.asus.com/support/faq/1045186/>
That was quite helpful. This seems to be closer to what I'm looking for: https://www.newegg.ca/asrock-rack-x570d4u-supports-3rd-gen-amd-ryzen-process... Thanks, ../Dave
From: David Mason via talk <talk@gtalug.org>
I'm interested in this topic but have no time to research it. And the information is kind of hard to find. But google is powerful. So I spent an hour or two. Please tell us what you find and conclude.
As I understand it, all DR5 does single-bit error correction, but it is up to the board/CPU to trap on multi-bit errors. I don't know if DDR5 writes bback the corrected bit, or if that's up to the other components.
I did not know that. I've now seen it stated elsewhere. I do know that single-bit correction, with no reporting for correction and no reporting for detected multi-bit errors is not of any use to the user. When I first saw a machine with ECC RAM (an IBM 370/145), the purpose of ECC was to allow the use of unreliable RAM chips. It did not actually help the user (except in price). I was so disappointed.
At one point I believed that IBM's phrase "chip kill" was relevant. ECC normally detects single-bit errors in a word but in many configurations, one dead chip will take out more than one bit in a word.
I don't know about the newest RAM technology. Typically, memory modules for ECC were 72-bits wide: 64 bits of data and 8 check bits. That allows Hamming code to be used. Originally that extra 8 bits was meant to be used as a parity bit for each byte. It just happened to allow ECC, as long as the RAM was treated as 8-byte words. (My Sun3/60 used parity on such RAM.)
I think that (some?) ordinary Ryzen's support ECC but that not many firmwares do. And perhaps not many chipsets do. Here's a kind of old link to Asus: <https://www.asus.com/support/faq/1045186/>
That was quite helpful. This seems to be closer to what I'm looking for: https://www.newegg.ca/asrock-rack-x570d4u-supports-3rd-gen-amd-ryzen-process...
Maybe I'm too penny-wise but that seems pretty expensive. There are hidden gems to be found. - My ThinkServer TS-100 has a Haswell i3 and (surprise!) supports ECC. That's only partly a function of the chipset. i5 or i7 chips would not support ECC. VERY ODD. Note: this box cost me about $400 new, a decade ago, so ECC does not need to be expensive. - <https://forums.servethehome.com/index.php?threads/lenovo-thinkcentre-thinkstation-tiny-project-tinyminimicro-reference-thread.34925/> notes that Lenovo ThinkCentre "tiny" computers with certain AMD processors and chipsets support ECC. - I asked Bee-Link if my SER7 mini-PC with AMD Ryzen 7 7840HS supported ECC; they said no. That's not surprising since AMD says that it doesn't. Perhaps that is true of all laptop processors. <https://www.amd.com/en/products/processors/laptop/ryzen/7000-series/amd-ryzen-7-7840hs.html> Unresearched threads: Many Xeon chips support ECC. Some of those may be among the less expensive ones (i.e. aimed at workstations rather than servers). (Almost) no Intel Core processors support ECC. I infer that this is a matter of market segmentation. Lots of AMD processors support ECC but not all. The chipset and the firmware can veto this capability. I infer that the desktop processors with decent iGPUs ("G" suffix) don't support ECC. Note: recent desktop processors without G suffix have GPUs that are low performance but work -- perfect for servers where you might want a head. Why the divergence? Market segmentation is best for a manufacturer of an expensive thing that competes with its own cheaper thing. Since AMD's main competitor as Intel, not itself, they threw in anything useful that didn't cost much to add. Here's a random servethehome review that might be interesting. <https://www.servethehome.com/amd-ryzen-server-the-asrock-rack-1u4lw-b650-2l2t-review/> Do look for more. This thread has interesting and sometimes conflicting claims: <https://forum.level1techs.com/t/fun-intel-anecdote-qs-amd-ecc-mobos/216354/14> This statement looked intriguing (talk about moving the goal posts!) DDR5 RDIMMs can and often do support 80-bit wide ECC (EC8).
Thanks, Hugh. I have spent many hours trying to make sense of this and come up to speed. Thanks for the links, and I'll definitely follow up with what I conclude. On Fri, 8 Nov 2024 at 00:31, D. Hugh Redelmeier via talk <talk@gtalug.org> wrote:
From: David Mason via talk <talk@gtalug.org>
I'm interested in this topic but have no time to research it. And the information is kind of hard to find. But google is powerful. So I spent an hour or two.
Please tell us what you find and conclude.
As I understand it, all DR5 does single-bit error correction, but it is up to the board/CPU to trap on multi-bit errors. I don't know if DDR5 writes bback the corrected bit, or if that's up to the other components.
I did not know that. I've now seen it stated elsewhere.
I do know that single-bit correction, with no reporting for correction and no reporting for detected multi-bit errors is not of any use to the user.
When I first saw a machine with ECC RAM (an IBM 370/145), the purpose of ECC was to allow the use of unreliable RAM chips. It did not actually help the user (except in price). I was so disappointed.
At one point I believed that IBM's phrase "chip kill" was relevant. ECC normally detects single-bit errors in a word but in many configurations, one dead chip will take out more than one bit in a word.
I don't know about the newest RAM technology.
Typically, memory modules for ECC were 72-bits wide: 64 bits of data and 8 check bits. That allows Hamming code to be used. Originally that extra 8 bits was meant to be used as a parity bit for each byte. It just happened to allow ECC, as long as the RAM was treated as 8-byte words.
(My Sun3/60 used parity on such RAM.)
I think that (some?) ordinary Ryzen's support ECC but that not many firmwares do. And perhaps not many chipsets do. Here's a kind of old link to Asus: <https://www.asus.com/support/faq/1045186/>
That was quite helpful. This seems to be closer to what I'm looking for:
https://www.newegg.ca/asrock-rack-x570d4u-supports-3rd-gen-amd-ryzen-process...
Maybe I'm too penny-wise but that seems pretty expensive.
There are hidden gems to be found.
- My ThinkServer TS-100 has a Haswell i3 and (surprise!) supports ECC. That's only partly a function of the chipset. i5 or i7 chips would not support ECC. VERY ODD. Note: this box cost me about $400 new, a decade ago, so ECC does not need to be expensive.
- < https://forums.servethehome.com/index.php?threads/lenovo-thinkcentre-thinkst...
notes that Lenovo ThinkCentre "tiny" computers with certain AMD processors and chipsets support ECC.
- I asked Bee-Link if my SER7 mini-PC with AMD Ryzen 7 7840HS supported ECC; they said no. That's not surprising since AMD says that it doesn't. Perhaps that is true of all laptop processors. < https://www.amd.com/en/products/processors/laptop/ryzen/7000-series/amd-ryze...
Unresearched threads:
Many Xeon chips support ECC. Some of those may be among the less expensive ones (i.e. aimed at workstations rather than servers).
(Almost) no Intel Core processors support ECC. I infer that this is a matter of market segmentation.
Lots of AMD processors support ECC but not all. The chipset and the firmware can veto this capability. I infer that the desktop processors with decent iGPUs ("G" suffix) don't support ECC. Note: recent desktop processors without G suffix have GPUs that are low performance but work -- perfect for servers where you might want a head.
Why the divergence? Market segmentation is best for a manufacturer of an expensive thing that competes with its own cheaper thing. Since AMD's main competitor as Intel, not itself, they threw in anything useful that didn't cost much to add.
Here's a random servethehome review that might be interesting. < https://www.servethehome.com/amd-ryzen-server-the-asrock-rack-1u4lw-b650-2l2...
Do look for more.
This thread has interesting and sometimes conflicting claims: < https://forum.level1techs.com/t/fun-intel-anecdote-qs-amd-ecc-mobos/216354/1...
This statement looked intriguing (talk about moving the goal posts!) DDR5 RDIMMs can and often do support 80-bit wide ECC (EC8). --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
On Fri, 8 Nov 2024 at 09:06, David Mason <dmason@torontomu.ca> wrote:
Thanks, Hugh.
I have spent many hours trying to make sense of this and come up to speed. Thanks for the links, and I'll definitely follow up with what I conclude.
I have spent more than 24 hours researching this... many rabbit holes! As Hugh noted, on the Intel side, ECC awareness is mostly a Xeon thing. AMD is much more supportive, but only with some chipsets/motherboards. ECC memory also comes in 2 flavours: 1) Registered (buffered) 80-bit wide, like https://www.kingston.com/datasheets/KF556R36RB-32.pdf 2) Unregistered (unbuffered) 72-bit wide, like: https://www.kingston.com/datasheets/KSM56E46BD8KM-32HA.pdf and, as far as I can tell, they are not inter-compatible if you want ECC. I wanted to take advantage of the NVME SSDs, as they are an order of magnitude faster than SATA, but as I mentioned in another thread, I want RAIDZ3, so I want at least 8 SSDs. So I came across https://www.amazon.ca/Highpoint-SSD7540-PCIe-8-Port-Controller/dp/B08LP2HTX3... which is supports 8 M.2 SSDs in a single PCIe-x16 socket.... the only drawback is $1660. So I also found https://www.amazon.ca/gp/product/B0863KK2BP/ (and many cheaper ones without fan/heatsink) which supports 4 M.2 SSDs (but requires turning on bifurcation mode) but is just over $100. Great, I thought, I throw 2 of those in, and I'm good to go.... so I went off and looked for ECC-supporting motherboards with 2 PCIe-x18 slots. But when I looked at that adapter closely, I discovered I needed to worry about PCI Express lanes! https://linustechtips.com/topic/1497718-basics-about-pci-lanes-chipset-lanes... has a good explanation about these, but the bottom line is that these are how the PCIe devices get access to memory via the CPU. Intel Core chips have 20, Ryzen have 24 usable (Xeon and ThreadRipper have 100+). So I went off and looked at Xeon and ThreadRipper chips and motherboards for a while.... but my budget didn't extend that far, and this *is* supposed to be mainly a file server. So I could only have 1 of those 4x M.2 interface boards. So I ended up looking at boards with 1 PCIe-x16, and 4 M.2-x4 slots. Because of the 4-lanes fewer connections of Intel, this basically brought me to Ryzen. So I ended up with: 1x ASUS Prime X670E-PRO WiFi <https://www.amazon.ca/gp/product/B0BF6VKQP4/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1> motherboard (considered Asrock X670E Taichi <https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp> but couldn't source it) 1x AMD Ryzen™ 7 7700 8-Core <https://www.amazon.ca/gp/product/B0BMQHSCVF/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1> CPU - 35 watts, includes cooler 2x Kingston Server Premier 32GB 5600MT/s DDR5 ECC <https://www.amazon.ca/gp/product/B0C7W4GK6R/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1> memory 1x ASUS Hyper M.2 x16 Gen 4 <https://www.amazon.ca/gp/product/B0863KK2BP/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1> PCIe adapter 1x TEAMGROUP T-Force Z540 2TB <https://www.amazon.ca/gp/product/B0CGR7RNCD/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1> SSD 7x TEAMGROUP MP44 2TB <https://www.amazon.ca/gp/product/B0C3VCD5Z8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1> SSDs 1x SAMA 850W ATX3.0 Fully Modular 80 Plus Gold <https://www.newegg.ca/sama-xf-series-xf850w-850-w/p/1HU-02S6-00030?Item=9SIB41PJT48552> power supply - even though 350W woul do 1x Corsair 4000D Airflow CC-9011201-WW White <https://www.newegg.ca/white-corsair-4000d-airflow-atx-mid-tower/p/N82E16811139157?Item=N82E16811139157> mid-tower case Total with taxes, just over $3000. Note that this will run mostly headless, apart from maintenance, so I'm more than happy with the built-in Radeon GPU. I have one high performance SSD which will have small boot and swap partitions. The rest are the best cheap SSDs I could find. All 8 will be mostly a large partition for ZFS for a total of about 10TB of RAIDZ3. I could have doubled that for an additional $1250. I found https://ca.pcpartpicker.com/list/ very useful, except it doesn't list ECC memory, and doesn't understand PCIe adapters, so it thought I couldn't hook up all my M.2s, and it didn't list the SAMA PSU. As this has turned out to be a pretty beefy box, I will likely be running Proxmos on top of Debian with ZFS. ../Dave
On Mon, Nov 25, 2024 at 11:08:13AM -0500, David Mason via talk wrote:
I have spent more than 24 hours researching this... many rabbit holes!
As Hugh noted, on the Intel side, ECC awareness is mostly a Xeon thing. AMD is much more supportive, but only with some chipsets/motherboards. ECC memory also comes in 2 flavours: 1) Registered (buffered) 80-bit wide, like https://www.kingston.com/datasheets/KF556R36RB-32.pdf 2) Unregistered (unbuffered) 72-bit wide, like: https://www.kingston.com/datasheets/KSM56E46BD8KM-32HA.pdf and, as far as I can tell, they are not inter-compatible if you want ECC.
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way. You can not mix buffered and unbuffered as far as I remember. Servers with a lot of slots need buffered memory, while systems with only a couple of slots per channel don't.
I wanted to take advantage of the NVME SSDs, as they are an order of magnitude faster than SATA, but as I mentioned in another thread, I want RAIDZ3, so I want at least 8 SSDs. So I came across https://www.amazon.ca/Highpoint-SSD7540-PCIe-8-Port-Controller/dp/B08LP2HTX3... which is supports 8 M.2 SSDs in a single PCIe-x16 socket.... the only drawback is $1660. So I also found https://www.amazon.ca/gp/product/B0863KK2BP/ (and many cheaper ones without fan/heatsink) which supports 4 M.2 SSDs (but requires turning on bifurcation mode) but is just over $100. Great, I thought, I throw 2 of those in, and I'm good to go.... so I went off and looked for ECC-supporting motherboards with 2 PCIe-x18 slots. But when I looked at that adapter closely, I discovered I needed to worry about PCI Express lanes!
https://linustechtips.com/topic/1497718-basics-about-pci-lanes-chipset-lanes... has a good explanation about these, but the bottom line is that these are how the PCIe devices get access to memory via the CPU. Intel Core chips have 20, Ryzen have 24 usable (Xeon and ThreadRipper have 100+). So I went off and looked at Xeon and ThreadRipper chips and motherboards for a while.... but my budget didn't extend that far, and this *is* supposed to be mainly a file server. So I could only have 1 of those 4x M.2 interface boards. So I ended up looking at boards with 1 PCIe-x16, and 4 M.2-x4 slots. Because of the 4-lanes fewer connections of Intel, this basically brought me to Ryzen.
So I ended up with: 1x ASUS Prime X670E-PRO WiFi <https://www.amazon.ca/gp/product/B0BF6VKQP4/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1> motherboard (considered Asrock X670E Taichi <https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp> but couldn't source it) 1x AMD Ryzen™ 7 7700 8-Core <https://www.amazon.ca/gp/product/B0BMQHSCVF/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1> CPU - 35 watts, includes cooler 2x Kingston Server Premier 32GB 5600MT/s DDR5 ECC <https://www.amazon.ca/gp/product/B0C7W4GK6R/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1> memory 1x ASUS Hyper M.2 x16 Gen 4 <https://www.amazon.ca/gp/product/B0863KK2BP/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1> PCIe adapter 1x TEAMGROUP T-Force Z540 2TB <https://www.amazon.ca/gp/product/B0CGR7RNCD/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1> SSD 7x TEAMGROUP MP44 2TB <https://www.amazon.ca/gp/product/B0C3VCD5Z8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1> SSDs 1x SAMA 850W ATX3.0 Fully Modular 80 Plus Gold <https://www.newegg.ca/sama-xf-series-xf850w-850-w/p/1HU-02S6-00030?Item=9SIB41PJT48552> power supply - even though 350W woul do 1x Corsair 4000D Airflow CC-9011201-WW White <https://www.newegg.ca/white-corsair-4000d-airflow-atx-mid-tower/p/N82E16811139157?Item=N82E16811139157> mid-tower case
Total with taxes, just over $3000.
Note that this will run mostly headless, apart from maintenance, so I'm more than happy with the built-in Radeon GPU. I have one high performance SSD which will have small boot and swap partitions. The rest are the best cheap SSDs I could find. All 8 will be mostly a large partition for ZFS for a total of about 10TB of RAIDZ3. I could have doubled that for an additional $1250. I found https://ca.pcpartpicker.com/list/ very useful, except it doesn't list ECC memory, and doesn't understand PCIe adapters, so it thought I couldn't hook up all my M.2s, and it didn't list the SAMA PSU.
As this has turned out to be a pretty beefy box, I will likely be running Proxmos on top of Debian with ZFS.
So 4 NVMe drives each with 4 PCIe lanes then in a single x16 PCIe slot? Should give decent bandwidth, although how much of that can your network connection even take advantage of? After all what good is 256Gbps NVMe if your ethernet is 1Gbps or even 10Gbps? -- Len Sorensen
My daily driver is a repurposed server CPU with a repurposed server motherboard. It has a Chinese motherboard (Machinist MR9A) paired with an old Xeon E5-2667 v3. Have a lot of SATA connections (6 IIRC), 2 NVMe slots, support quad-channel memory (up to 4x32GB), and ECC memory. Paid 50 bucks for the motherboard and less than that on the CPU on AliExpress. It will need a beefy power supply (mine is 800W) and will heat your house if abused (that can even be an upside, depending on the weather). It does not have video nor WIFI on the board, so you will need an adapter. It has 1Gbps Ethernet onboard (Realtek PCIe GbE Family Controller). Mauro https://www.maurosouza.com - registered Linux User: 294521 Scripture is both history, and a love letter from God. On Mon, Nov 25, 2024 at 5:38 PM Lennart Sorensen via talk <talk@gtalug.org> wrote:
On Mon, Nov 25, 2024 at 11:08:13AM -0500, David Mason via talk wrote:
I have spent more than 24 hours researching this... many rabbit holes!
As Hugh noted, on the Intel side, ECC awareness is mostly a Xeon thing. AMD is much more supportive, but only with some chipsets/motherboards. ECC memory also comes in 2 flavours: 1) Registered (buffered) 80-bit wide, like https://www.kingston.com/datasheets/KF556R36RB-32.pdf 2) Unregistered (unbuffered) 72-bit wide, like: https://www.kingston.com/datasheets/KSM56E46BD8KM-32HA.pdf and, as far as I can tell, they are not inter-compatible if you want ECC.
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way.
You can not mix buffered and unbuffered as far as I remember. Servers with a lot of slots need buffered memory, while systems with only a couple of slots per channel don't.
I wanted to take advantage of the NVME SSDs, as they are an order of magnitude faster than SATA, but as I mentioned in another thread, I want RAIDZ3, so I want at least 8 SSDs. So I came across
https://www.amazon.ca/Highpoint-SSD7540-PCIe-8-Port-Controller/dp/B08LP2HTX3...
which is supports 8 M.2 SSDs in a single PCIe-x16 socket.... the only drawback is $1660. So I also found https://www.amazon.ca/gp/product/B0863KK2BP/ (and many cheaper ones without fan/heatsink) which supports 4 M.2 SSDs (but requires turning on bifurcation mode) but is just over $100. Great, I thought, I throw 2 of those in, and I'm good to go.... so I went off and looked for ECC-supporting motherboards with 2 PCIe-x18 slots. But when I looked at that adapter closely, I discovered I needed to worry about PCI Express lanes!
https://linustechtips.com/topic/1497718-basics-about-pci-lanes-chipset-lanes...
has a good explanation about these, but the bottom line is that these are how the PCIe devices get access to memory via the CPU. Intel Core chips have 20, Ryzen have 24 usable (Xeon and ThreadRipper have 100+). So I went off and looked at Xeon and ThreadRipper chips and motherboards for a while.... but my budget didn't extend that far, and this *is* supposed to be mainly a file server. So I could only have 1 of those 4x M.2 interface boards. So I ended up looking at boards with 1 PCIe-x16, and 4 M.2-x4 slots. Because of the 4-lanes fewer connections of Intel, this basically brought me to Ryzen.
So I ended up with: 1x ASUS Prime X670E-PRO WiFi < https://www.amazon.ca/gp/product/B0BF6VKQP4/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1
motherboard (considered Asrock X670E Taichi <https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp> but couldn't source it) 1x AMD Ryzen™ 7 7700 8-Core < https://www.amazon.ca/gp/product/B0BMQHSCVF/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1
CPU - 35 watts, includes cooler 2x Kingston Server Premier 32GB 5600MT/s DDR5 ECC < https://www.amazon.ca/gp/product/B0C7W4GK6R/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
memory 1x ASUS Hyper M.2 x16 Gen 4 < https://www.amazon.ca/gp/product/B0863KK2BP/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1
PCIe adapter 1x TEAMGROUP T-Force Z540 2TB < https://www.amazon.ca/gp/product/B0CGR7RNCD/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
SSD 7x TEAMGROUP MP44 2TB < https://www.amazon.ca/gp/product/B0C3VCD5Z8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1
SSDs 1x SAMA 850W ATX3.0 Fully Modular 80 Plus Gold < https://www.newegg.ca/sama-xf-series-xf850w-850-w/p/1HU-02S6-00030?Item=9SIB...
power supply - even though 350W woul do 1x Corsair 4000D Airflow CC-9011201-WW White < https://www.newegg.ca/white-corsair-4000d-airflow-atx-mid-tower/p/N82E168111...
mid-tower case
Total with taxes, just over $3000.
Note that this will run mostly headless, apart from maintenance, so I'm more than happy with the built-in Radeon GPU. I have one high performance SSD which will have small boot and swap partitions. The rest are the best cheap SSDs I could find. All 8 will be mostly a large partition for ZFS for a total of about 10TB of RAIDZ3. I could have doubled that for an additional $1250. I found https://ca.pcpartpicker.com/list/ very useful, except it doesn't list ECC memory, and doesn't understand PCIe adapters, so it thought I couldn't hook up all my M.2s, and it didn't list the SAMA PSU.
As this has turned out to be a pretty beefy box, I will likely be running Proxmos on top of Debian with ZFS.
So 4 NVMe drives each with 4 PCIe lanes then in a single x16 PCIe slot? Should give decent bandwidth, although how much of that can your network connection even take advantage of?
After all what good is 256Gbps NVMe if your ethernet is 1Gbps or even 10Gbps?
-- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
On Mon, 25 Nov 2024 at 16:13, Mauro Souza via talk <talk@gtalug.org> wrote:
My daily driver is a repurposed server CPU with a repurposed server motherboard. It has a Chinese motherboard (Machinist MR9A) paired with an old Xeon E5-2667 v3. Have a lot of SATA connections (6 IIRC), 2 NVMe slots, support quad-channel memory (up to 4x32GB), and ECC memory. Paid 50 bucks for the motherboard and less than that on the CPU on AliExpress. It will need a beefy power supply (mine is 800W) and will heat your house if abused (that can even be an upside, depending on the weather).
Yup, there is no question you can do this quite inexpensively. (apart from the SSD), and I've had good luck with EBay in the past, but I just wasn't willing to do that this time. ../Dave
On 11/25/24 15:32, Lennart Sorensen via talk wrote:
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way.
Oh my goodness sake (;-)) That /really/ takes me back: I learned B on a DPS-8, which used 72-bit double-words. --dave -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
From: David Collier-Brown via talk <talk@gtalug.org>
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way.
Oh my goodness sake (;-))
That /really/ takes me back: I learned B on a DPS-8, which used 72-bit double-words.
The DPS-8 surely had parity or ECC on top of the 72 data bits. The DPS-8 was a 36-bit word system. A logical descendant of the IBM 701 (which had 72 Williams Tubes for RAM). Honeywell and UNIVAX hung on to 36-bits much longer than IBM. (The first 36-bit word machine I used was an IBM 7040/44 but I only used FORTRAN on it.)
On 11/26/24 09:16, D. Hugh Redelmeier via talk wrote:
From: David Collier-Brown via talk <talk@gtalug.org>
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way. Oh my goodness sake (;-))
That /really/ takes me back: I learned B on a DPS-8, which used 72-bit double-words. The DPS-8 surely had parity or ECC on top of the 72 data bits.
Yes, I vaguely remember that the raw data size was 80 bits, for ecc. It used to silently misbehave on misaligned fetches (:-(), but loudly complain about memory problems (:-)).
The DPS-8 was a 36-bit word system. A logical descendant of the IBM 701 (which had 72 Williams Tubes for RAM). Honeywell and UNIVAX hung on to 36-bits much longer than IBM.
(The first 36-bit word machine I used was an IBM 7040/44 but I only used FORTRAN on it.) --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
-- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
participants (6)
-
D. Hugh Redelmeier -
David Collier-Brown -
David Mason -
Lennart Sorensen -
Mauro Souza -
Ron / BCLUG