My daily driver is a repurposed server CPU with a repurposed server motherboard. It has a Chinese motherboard (Machinist MR9A) paired with an old Xeon E5-2667 v3. Have a lot of SATA connections (6 IIRC), 2 NVMe slots, support quad-channel memory (up to 4x32GB), and ECC memory. Paid 50 bucks for the motherboard and less than that on the CPU on AliExpress. It will need a beefy power supply (mine is 800W) and will heat your house if abused (that can even be an upside, depending on the weather). It does not have video nor WIFI on the board, so you will need an adapter. It has 1Gbps Ethernet onboard (Realtek PCIe GbE Family Controller). Mauro https://www.maurosouza.com - registered Linux User: 294521 Scripture is both history, and a love letter from God. On Mon, Nov 25, 2024 at 5:38 PM Lennart Sorensen via talk <talk@gtalug.org> wrote:
On Mon, Nov 25, 2024 at 11:08:13AM -0500, David Mason via talk wrote:
I have spent more than 24 hours researching this... many rabbit holes!
As Hugh noted, on the Intel side, ECC awareness is mostly a Xeon thing. AMD is much more supportive, but only with some chipsets/motherboards. ECC memory also comes in 2 flavours: 1) Registered (buffered) 80-bit wide, like https://www.kingston.com/datasheets/KF556R36RB-32.pdf 2) Unregistered (unbuffered) 72-bit wide, like: https://www.kingston.com/datasheets/KSM56E46BD8KM-32HA.pdf and, as far as I can tell, they are not inter-compatible if you want ECC.
Apparently at the moment, most RDIMM DDR5 is EC8 ECC which is 80bit. And most UDIMM is EC4 ECC which is 72 bit. I don't believe there is any requirement for it to be that way.
You can not mix buffered and unbuffered as far as I remember. Servers with a lot of slots need buffered memory, while systems with only a couple of slots per channel don't.
I wanted to take advantage of the NVME SSDs, as they are an order of magnitude faster than SATA, but as I mentioned in another thread, I want RAIDZ3, so I want at least 8 SSDs. So I came across
https://www.amazon.ca/Highpoint-SSD7540-PCIe-8-Port-Controller/dp/B08LP2HTX3...
which is supports 8 M.2 SSDs in a single PCIe-x16 socket.... the only drawback is $1660. So I also found https://www.amazon.ca/gp/product/B0863KK2BP/ (and many cheaper ones without fan/heatsink) which supports 4 M.2 SSDs (but requires turning on bifurcation mode) but is just over $100. Great, I thought, I throw 2 of those in, and I'm good to go.... so I went off and looked for ECC-supporting motherboards with 2 PCIe-x18 slots. But when I looked at that adapter closely, I discovered I needed to worry about PCI Express lanes!
https://linustechtips.com/topic/1497718-basics-about-pci-lanes-chipset-lanes...
has a good explanation about these, but the bottom line is that these are how the PCIe devices get access to memory via the CPU. Intel Core chips have 20, Ryzen have 24 usable (Xeon and ThreadRipper have 100+). So I went off and looked at Xeon and ThreadRipper chips and motherboards for a while.... but my budget didn't extend that far, and this *is* supposed to be mainly a file server. So I could only have 1 of those 4x M.2 interface boards. So I ended up looking at boards with 1 PCIe-x16, and 4 M.2-x4 slots. Because of the 4-lanes fewer connections of Intel, this basically brought me to Ryzen.
So I ended up with: 1x ASUS Prime X670E-PRO WiFi < https://www.amazon.ca/gp/product/B0BF6VKQP4/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1
motherboard (considered Asrock X670E Taichi <https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp> but couldn't source it) 1x AMD Ryzen™ 7 7700 8-Core < https://www.amazon.ca/gp/product/B0BMQHSCVF/ref=ppx_yo_dt_b_asin_title_o00_s02?ie=UTF8&psc=1
CPU - 35 watts, includes cooler 2x Kingston Server Premier 32GB 5600MT/s DDR5 ECC < https://www.amazon.ca/gp/product/B0C7W4GK6R/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
memory 1x ASUS Hyper M.2 x16 Gen 4 < https://www.amazon.ca/gp/product/B0863KK2BP/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1
PCIe adapter 1x TEAMGROUP T-Force Z540 2TB < https://www.amazon.ca/gp/product/B0CGR7RNCD/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
SSD 7x TEAMGROUP MP44 2TB < https://www.amazon.ca/gp/product/B0C3VCD5Z8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1
SSDs 1x SAMA 850W ATX3.0 Fully Modular 80 Plus Gold < https://www.newegg.ca/sama-xf-series-xf850w-850-w/p/1HU-02S6-00030?Item=9SIB...
power supply - even though 350W woul do 1x Corsair 4000D Airflow CC-9011201-WW White < https://www.newegg.ca/white-corsair-4000d-airflow-atx-mid-tower/p/N82E168111...
mid-tower case
Total with taxes, just over $3000.
Note that this will run mostly headless, apart from maintenance, so I'm more than happy with the built-in Radeon GPU. I have one high performance SSD which will have small boot and swap partitions. The rest are the best cheap SSDs I could find. All 8 will be mostly a large partition for ZFS for a total of about 10TB of RAIDZ3. I could have doubled that for an additional $1250. I found https://ca.pcpartpicker.com/list/ very useful, except it doesn't list ECC memory, and doesn't understand PCIe adapters, so it thought I couldn't hook up all my M.2s, and it didn't list the SAMA PSU.
As this has turned out to be a pretty beefy box, I will likely be running Proxmos on top of Debian with ZFS.
So 4 NVMe drives each with 4 PCIe lanes then in a single x16 PCIe slot? Should give decent bandwidth, although how much of that can your network connection even take advantage of?
After all what good is 256Gbps NVMe if your ethernet is 1Gbps or even 10Gbps?
-- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk