I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
The answer to your question is, it depends on your expectations. The tradeoff of a typical no VRAM machine is latency and quality.
My progression:
- I decided to try ollama and ran a small model on 8G RAM on a VPS. Slow but it worked. Ran an 3B parameter models – slowly. Struggled with 8B models. Problems with hallucinations from these small models. This was my induction into self hosting llms.
- Installed similar setup on a SFF desktop with 16G RAM. Ran 8B models slowly. More capability, but still below the threshold I was looking for. Open WebUI + ollama worked but still too slow.
- Acquired a 16G VRAM card and installed in an old desktop with 16G RAM. This runs 12B models and crosses the threshold, in my opinion, of usability as an assistant. This setup runs Open WebUI, again reinforcing usability.
- Am I satisfied? No. I’d like bigger, faster meaning more VRAM, on a newer platform, but content enough now to continue experimenting with what I have.
In sum: Don’t expect low laitency multi-step reasoning/planning with a low resource / no VRAM machine. Anything close to what you can get online at reasonable speeds requires ample VRAM.
For a reality check … I did a quick run on those three machines as they are still active. What follows has been synthesized through a cloud based model from real data:
same prompt, three machines
Prompt: “Explain how a hash table works, including collisions, in about 250 words.”
Run once on each machine withollama run <model> --verbose, all Q4_K_M quantization.Machine Model Processor Speed (tokens/s) Total time Quality (/20) VPS, 8 GB RAM gemma3n:latest (6.9B) CPU only 5.3 1m 11s 16 SFF desktop, 16 GB RAM gemma3:4b (3.9B) CPU only 6.8 51s 13 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking off GPU 31.2 11.5s 18 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking on GPU 30.9 38.5s 18 What stood out
- The GPU box generated about 4-5x faster than the SFF while running a model roughly 3x bigger.
- Both CPU-only machines landed at 5-7 t/s. Usable, but slow for interactive chat.
- Turning thinking off on the 12B cut total time from 38s to 11.5s with no change in speed (~31 t/s) and no drop in answer quality on this prompt. Thinking would likely matter more for math, code, or multi-step problems, which this test doesn’t cover.
- The 12B had no factual slips. The smaller models made small conceptual errors (blurring hash code vs. index, calling the whole array a “bucket”) and one skipped lookup entirely.
- All of them left out worst-case O(n) behavior and resizing.
Caveats
- Different models on each machine, so this shows what each box can run well, not identical hardware running identical software.
- The thinking-on run generated about 1,180 tokens versus about 350 with thinking off, so compare tokens/sec rather than total time across those two rows.
- Quality scores are my rough 1-5 ratings on accuracy, length, completeness and cleanliness, with an LLM’s help, from one run each at temperature 1. Treat them as anecdotal, not a benchmark.
- gemma3n is listed by Ollama as 6.9B, but it uses an “effective parameter” design (Google calls it E4B, roughly 4B in practice), so it’s lighter than a dense 7B. That’s why it ran on the 8 GB VPS where a standard 8B struggled.
The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.
Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.
Most of the world’s food comes from small local farms. Maybe join a CSA if you live in an industrialized capitalist country without a proper farmer’s market nearby
There are 3 or 4 billionaires in my country. Founders of Wise and Bolt. None of them own farms or grocery stores.
Matter of fact, the most popular grocery store chain is coop so most of the “owners” (members) of your local cooperative aren’t even millionaires.
I donate my compute to https://aihorde.net/ once in a while. got to play around with some cool AI bullshit there. Its fun to play around with.
Its all volunteer run. I got my steam deck pushing small updates to peoples like dnd rollplay kinda stuff. koboldcpp makes it easy to set up and you can use hugging face models or your own.
Theres some really excellent open models out there if you care to look! Most are hyper specialized better than say GPT and such.
Download Ollama. Download one of the smaller Qwen models. Have at it.
Try qwen3.5:9b (a 7.6Gb model) to start.
Friends don’t let friends use Ollama.
Somethin’ happening with them?
I think from a performance standpoint llamacpp is better, and requires approximately the same amount of technical knowledge to operate.
Use open weight models either small enough to run yourself or via your diverse online host of choice.
without supporting the data centers
online host of choice
I believe there are options to host open models on hardware you don’t own without having to give money to the “evil” data centers. Getting away from data centers entirely is impossible as that is how the internet has worked for a very long time now.
Not to mention that it’s training and not inference where most of the compute is going anyway.
I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
trying to code
copy pasting to and from duck
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.
OP said they didn’t want to support billionaires and data centers
The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten
Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM
Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B
If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.
I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.
Sounds like golem. Interesting
99% of AI models available are based on stolen work, so even running it locally is immoral…
“Copyright” is immoral and exists to protect capitalists. Intellectual property as a concept needs to die, and knowledge needs to be accessable to everybody.
This is what a person who has never developed anything would say.
Like, Stallman?
I’ve developed and written loads. Everything licensed CC BY-SA.
I will now reuse your work without attribution and claim it is the AI’s fault. Easy.
Everything you’ve made is now CC BY-SA too, awesome. Thanks for contributing.
No problem, you’re welcome. I also paid a small car’s worth of money to a fascist oligarch while borrowing the fruits of your work. I, too, am very happy with this.
But look at the bright side, you too can now pay a small car’s worth of money to a fascist oligarch and then get some source code that may or may not be close to my proprietary stuff. You know, the one made from your stuff without attributing. This is fine.
I think you may have lost the plot there.
I like deepseek v4flash, I use opencode go and commandcode, somehow feels not as bad since its cheaper and technically can be ran at home if you can afford it. I don’t care for regular ai chat, feels useless after using agents. Agents are what you expect when you think of ai at the computer, hermes is a solid free one. Those two subs rotating as one runs out should last you a while at 20$ a month. These smaller companies/datacenters aren’t the issue imo, the larger ones like Anthropic, google, openai, etc. are.
It just functions so much better to use low cost api subs over trying to run anything locally. Let them subsidize this shit for you, don’t abuse your own hardware while it’s expensive af to replace, can spend 100s a year on subs and it’ll be less than the hardware costs for that amount of time.
Lumo and some on duck.ai seem fine for the free usage I do. Here’s a fun one that appears accurate, enjoy! duck.ai/share/01cee404-a240-4c14-9a83-584d406c5a68#Ch22BaAoEAejJEeRNZYh-MfEYLFoDCsVZsPd23IGU5w
