On 2025-02-14 00:14, Evan Leibovitch wrote:
Most NUC-sized systems rely on the on-board GPU found on many Intel and some Ryzen CPUs. They're going to have neither the onboard memory nor the horsepower to host an R1 model, the requirements page I use <https://apxml.com/posts/gpu-requirements-deepseek-r1> says that an RTX3060 with 12GB of GPU RAM is the minimum for even the smallest R1 model.
For sure, those are the recommended specs. But it runs just fine on my SER6 system, for what the model can actually do.
Obviously there are tradeoffs, in both cost -- maybe 5x the price of your SER6 -- and how many watts can be pumped out of a NUC-factor power supply. A small desktop-sized PC might better cool and power such a rig and may even be cheaper, though you might want to hold off on the current rev of the 5080/5090 <https://www.tweaktown.com/news/103255/its- not-just-the-geforce-rtx-5090-weve-now-got-melting-connectors-on-5080/ index.html>.
I'd rather hold off until the models can safely do what the hosted models can. That's agentic AI. Right now you seemingly need to rely on hosted services, and I'm just not okay with that. https://angiejones.tech/system-access-for-ai-agents/
Consider the expectations being set. Nobody would react if you complained your Beelink wasn't very good at running /God of War/.
It runs the model and God of War about the same -- Thanks to WINE & GabeN (Valve) -- adequate, with reasonable expectations. It's just that there is so much hype around Reasoning, especially in how it adds to Agentic AI. But the open source models don't seem to be there yet. I tried Cohere 7B today as well... it's still really only useful for analysis generation.
Yes, a reasonable entry-level AI rig for serious work will cost about upwards of $4K, but that's not far from the cost of a good-level gaming, video-production setup, or a Macbook M4. And ... especially ... think of what was just announced at CES <https://arstechnica.com/ai/2025/01/ nvidias-first-desktop-pc-can-run-local-ai-models-for-3000/>.
For serious work, no doubt about that. Renting a H100 VPS from Digitalocean at TOR1 costs $30k/yr , so that's quite a bargain. I think my $dayjob would be happy to pay $4k + as well. However, I'm not interested to see what comes of the continuation of advancements. Such as the paper I found in the Level1techs drop from yesterday: https://arxiv.org/abs/2308.15030
- Evan
Thanks, Evan! -- Mark Prosser // E: mark@zealnetworks.ca // W: https://zealnetworks.ca