But this isn’t like withholding information about the core counts of the chips inside your machine, or the TGP of the mobile GPU at its heart for fear of confusing some fictitious customer. There are people who care about the hardware inside these devices, but it’s becoming clear there are precious few who care about the AI components or theoretical capabilities of those machines.
Oh, there are. I’m one of them.
There are dozens of us!
…Problem is, NPUs are junk.
I could ramble on, but basically they’ve fallen into the hole of “obtuse proprietary APIs for esoteric hardware” that FPGAs did, so no one wrote anything useful for them outside of business niches, like (say) face recognition to login to Windows or embedded vision stuff for industrial robots. I can’t do anything useful with an NPU, even being familiar with the software stacks/APIs.
To be more concrete, if I had a shiny new laptop and wanted to use my NPU for an LLM, my only option is basically proprietary weights of llama 8B. A tiny, obsolete model, with obsolete quantization, with obsolete sampling and features and API.
Vision? Audio? Forget it. Same with newer models; no one is working on it. Going outside the tiny NPU memory pool for offloading? Batching? Laughs.
And you couldn’t even run old models until ~2025! It wasn’t even developed. Best one can do right now is the AMD Lemonade or a similar Intel docker server because it’s otherwise such a nightmare to install/develop. How many laptop buyers do you think use docker for an obscure piece of software?
And why the heck would I even bother with that when I can run GLM 4.6V 120B quickly on a CPU and tiny GPU? And, more importantly, it fucking works.
The only functional “AI” product in the western market is Strix Halo (branded as the AMD AI Max series), which is so expensive it’s not worth it over used stuff. Until now, I guess.
The Chinese market is a bit different with homegrown server NPUs, but that’s a whole other tangent.
TL;DR:
Brands don’t know what the fuck they’re talking about, as their products don’t actually work for self hosting/local inference. It’s all bullshit!
Oh, there are. I’m one of them.
There are dozens of us!
…Problem is, NPUs are junk.
I could ramble on, but basically they’ve fallen into the hole of “obtuse proprietary APIs for esoteric hardware” that FPGAs did, so no one wrote anything useful for them outside of business niches, like (say) face recognition to login to Windows or embedded vision stuff for industrial robots. I can’t do anything useful with an NPU, even being familiar with the software stacks/APIs.
To be more concrete, if I had a shiny new laptop and wanted to use my NPU for an LLM, my only option is basically proprietary weights of llama 8B. A tiny, obsolete model, with obsolete quantization, with obsolete sampling and features and API.
Vision? Audio? Forget it. Same with newer models; no one is working on it. Going outside the tiny NPU memory pool for offloading? Batching? Laughs.
And you couldn’t even run old models until ~2025! It wasn’t even developed. Best one can do right now is the AMD Lemonade or a similar Intel docker server because it’s otherwise such a nightmare to install/develop. How many laptop buyers do you think use docker for an obscure piece of software?
And why the heck would I even bother with that when I can run GLM 4.6V 120B quickly on a CPU and tiny GPU? And, more importantly, it fucking works.
The only functional “AI” product in the western market is Strix Halo (branded as the AMD AI Max series), which is so expensive it’s not worth it over used stuff. Until now, I guess.
The Chinese market is a bit different with homegrown server NPUs, but that’s a whole other tangent.
TL;DR:
Brands don’t know what the fuck they’re talking about, as their products don’t actually work for self hosting/local inference. It’s all bullshit!
Dell finally figured that out. Good on them.