What Is Local AI? Running AI Models on Your Own Device
Local AI runs on your own hardware instead of a company server. How it works, what it costs, what it can and cannot do, and how to try it in a browser.
Most AI you have used works the same way: you type something, it travels to a data centre owned by OpenAI, Google, or Anthropic, a very large model processes it there, and the answer travels back. Local AI inverts that. The model itself is downloaded onto your machine, and the thinking happens on your own processor. Nothing you type is transmitted, because there is nowhere for it to go.
That single architectural difference is why people who work with confidential material keep asking about it. This guide covers what local AI actually is, what hardware it needs, where it genuinely beats cloud AI, where it plainly loses, and the easiest way to try it without installing anything.
How local AI actually works
A language model is, in the end, a large file of numbers — the weights. Running a model means loading those numbers into memory and doing a great deal of arithmetic against them for every word produced. Cloud providers do this on racks of specialised accelerators. Local AI does it on the hardware already sitting on your desk.
What made this practical for ordinary machines was not faster laptops but smaller models. Two techniques did most of the work:
- Distillation and better training. A model with 3 billion parameters trained well in 2026 comfortably outperforms a 13 billion parameter model from a few years earlier. Capability per gigabyte has climbed steeply.
- Quantization. Weights are normally stored at 16 bits of precision. Rounding them to 4 bits shrinks the file roughly fourfold with a modest quality cost. A 7-billion-parameter model drops from about 14 GB to nearer 4 GB, which is what turns a server workload into a laptop one.
Once loaded, the model sits in RAM and answers from there. Pull out your network cable and it carries on working, which is the cleanest test of whether a tool is genuinely local.
What you need to run it
Memory is the binding constraint, not raw processor speed. A rough guide:
| Model size | RAM needed (4-bit) | Realistic on | Good for |
|---|---|---|---|
| 1–3 billion | 2–3 GB | Any modern laptop, some phones | Summaries, extraction, rewriting, classification |
| 7–8 billion | 5–6 GB | 16 GB laptop, Apple Silicon | General questions, drafting, light reasoning |
| 13–14 billion | 9–10 GB | 32 GB machine or dedicated GPU | Harder reasoning, longer documents |
| 70 billion and up | 40 GB and up | Workstation, multi-GPU rig | Approaching cloud quality |
Apple Silicon Macs punch above their weight here because the processor and graphics share one pool of memory, so a 16 GB MacBook can hold models that would need a discrete graphics card on a PC. On Windows and Linux, available video memory is usually the number that matters.
Three ways to run AI locally
In your browser, with nothing installed
This is the newest option and by far the lowest-friction. Modern browsers expose your graphics hardware to web pages through an interface called WebGPU, so a website can download a model into its cache and run it on your machine. No installer, no command line, no account. Our AI PDF summarizer and chat with PDF tools work exactly this way: the model arrives once, then your documents are read and answered on your own hardware.
The trade-off is size. Browser-hosted models are small ones, because nobody wants to download 40 GB to read a document. They are strong at summarizing, finding, and extracting, and weaker at multi-step reasoning.
A desktop app
Tools such as Ollama, LM Studio, and Jan install a model runner on your computer and give you a chat window or a local API. This is the sweet spot for most people who want a general-purpose private assistant: you choose from a library of open models, swap between them freely, and everything stays on the machine. Expect a download of several gigabytes per model and a few minutes of setup.
Your own server
Running a model on a machine you control and connecting to it over your own network is the enthusiast and small-business route. It gives you the largest models and full control, at the cost of buying hardware and maintaining it.
Where local AI wins
- Confidentiality. This is the real reason to bother. A contract under NDA, a medical record, an unreleased financial statement, a client list — with a local model these are never transmitted, so there is no retention policy to read, no breach to be caught in, and no question about whether your text became training data.
- It works offline. On a plane, on poor hotel wifi, in a secure facility with no outside network.
- No metered cost. After the download, generation is free however much you use. Cloud APIs bill per token.
- Nothing changes underneath you. A model on your disk behaves the same next year. Hosted models are updated, deprecated, and retired on the provider timetable.
- Regulatory simplicity. For work under HIPAA, GDPR, or attorney-client privilege, not transmitting data at all is a far easier position to defend than transmitting it under contract.
Where local AI loses
Anyone who tells you local models match frontier cloud models is selling something. The honest limits:
- Raw capability. A 3-billion-parameter model is not competing with a frontier model on hard reasoning, subtle code, or long chains of logic. The gap is real and it is large.
- Speed on modest hardware. Without a decent GPU, expect a readable but unhurried pace.
- Context length. Local models generally hold less text at once, which is why good local tools read long documents in sections rather than swallowing them whole.
- No live information. The model knows what it was trained on. No web browsing, no current events.
- Setup, for the desktop route. Choosing a model and a quantization level is real friction, which is precisely why browser-based tools that pick sensible defaults for you have become popular.
Choosing sensibly: match the tool to the document
The useful question is not which AI is best, but which risk you are taking with this particular file. A practical rule:
- Public or trivial material — a published paper, a product manual, a blog draft. Use whatever is most capable. There is nothing to protect.
- Internal but not sensitive — meeting notes, marketing copy. A cloud tool with a no-training setting is a reasonable call.
- Confidential — contracts, medical or financial records, personal data, anything under NDA or privilege. Keep it local. A good-enough summary you can verify beats an excellent one that required handing the document to a third party.
Most people end up using both, which is the sane outcome. Frontier cloud models for thinking hard about public things, local models for anything they would not be comfortable seeing in another company's logs. If you want the detail on what actually happens to a file when you upload it, we wrote that up separately in what happens when you upload a PDF.
Trying it in the next two minutes
You do not need to install anything to see whether local AI is useful to you. Open our PDF summarizer or chat with PDF tool and give it a document. The model downloads once and is cached, so the first run is slower than the ones after it.
Then run the test that settles the question: once the model has loaded, disconnect from the internet and ask it something else. It will answer. That is the whole idea — the intelligence is on your device now, and your document never had to travel to be understood.
If your documents contain personal data you need removed rather than analysed, our automatic redaction tool finds and blacks it out locally too, and data extraction pulls invoice and receipt fields into a spreadsheet without an upload.