On Thu, Feb 13, 2025 at 10:32 PM Mark Prosser via talk <talk@gtalug.org> wrote:

I was thrilled to find out just how easy it was to self-host these models, and that it was feasible to do it on a mini PC, like my Beelink SER6 (small NUC-like mobile ryzen platform). I downloaded the model last ight, and to my surprise, the performance was more than adequate. The model was able to reason & give seemingly good results, despite my box being at the very low end of "AI performance".

The bad news is that, all this hype about "Agentic AI" and all the other cool "tool models" isn't really fully available yet, in a format that can be hosted well on my box -- let alone a box I can afford.

Most NUC-sized systems rely on the on-board GPU found on many Intel and some Ryzen CPUs. They're going to have neither the onboard memory nor the horsepower to host an R1 model, the requirements page I use says that an RTX3060 with 12GB of GPU RAM is the minimum for even the smallest R1 model.

Having said that, there are NUCs and there are NUCs. Asus has been able to do some interesting things with the form factor ever since taking over Intel's NUC business, especially on its ROG gamer's side. Consider the 2025 ROG NUC which can sport an RTX5080.

Obviously there are tradeoffs, in both cost -- maybe 5x the price of your SER6 -- and how many watts can be pumped out of a NUC-factor power supply. A small desktop-sized PC might better cool and power such a rig and may even be cheaper, though you might want to hold off on the current rev of the 5080/5090.

Anyway, it's neat that I can have this little chat bot or coding assistant hosted on a little cube on my desk; however, it seems that it's still quite far away that I can have little AI interns working on generating insightful signals for my automation workflows.

Consider the expectations being set. Nobody would react if you complained your Beelink wasn't very good at running God of War.

Yes, a reasonable entry-level AI rig for serious work will cost about upwards of $4K, but that's not far from the cost of a good-level gaming, video-production setup, or a Macbook M4. And ... especially ... think of what was just announced at CES.

- Evan