From: Evan Leibovitch via Talk <talk@lists.gtalug.org>
Thanks for sharing your experience and insights. We really do learn from each others different experiences.
The AI world is moving faster than even Fedora, so that's less of a factor here.
I guess it depends on who is supplying what parts of the stack. Every supplier wants the stuff below them to stand still so they don't need to support many variants.
Well, it's a choice, but I believe a safe one for the longer term though a pain in the short term. I intend to keep this box for a while, and what's bleeding edge today will be legacy before I can blink.
Yeah. Ordinary computer hardware has been slow to change for the last decade or so. That reduced stresses on us. Hardware for AI is innovating like crazy. We're not used to that. It stresses all parts of the stack, including the OS.
FWIW, it's based on the AMD Ryzen AI MAX+ 395. Dumb name but quite the capability. I can assign 96GB to GPU memory which should let me run some interesting LLMs locally. It's probably a little slower than an RTX 5090, but that maxes out at 32GB and the GPU card costs more than my entire system.
Right. My (unearned) intuition that what matters is memory bandwidth and GPU's get amazing bandwidth via very-wide memory paths. You cannot do that with ordinary RAM on a PC. (The Mac gets partway there.) You are not trying to train models (the really compute-intensive stuff) so maybe the demands are not beyond what PC memory can handle.
It's new enough that AMD is about to roll out ROCm for this architecture Real Soon Now. I'm confident they will since obviously they have a big stake in maximizing this chip's performance.
I don't understand AMD's behaviour. They certainly have been slow and sporadic in their ROCm support for various cards. There were indications at some points that they only cared about the industrial AI cards.
And, yes, in a sense, I *did* want to send a message through the marketplace. AMD (for which I still have some sentiment because of ATI) should be encouraged to produce systems like this,
I've been encouraging AMD for decades. The results have been frustratingly mediocre.
Or, as you did for a while, run Windows on your desktop with an enclave for Linux when useful.
Indeed I did. But I've been researching and found that Windows Subsystem for Linux is sub-optimal for AI tasks -- especially on AMD.
Sometimes it makes sense to have different boxes for different universes. Mixing your everyday desktop and your compute monster might not be the right choice. Or it might be. - I want my desktop on all the time so I'd like its power requirements and noise to be minimal. Those might be limiting for AI (or might not). - I want a certain kind of stability on my desktop. Probably different from an AI box.