Running AI on your own machine: the hardware you actually need

Cloud AI tools are brilliant, but more businesses now want to run AI models on their own hardware, for privacy, predictable costs or simply the freedom to experiment. The good news is that you no longer need a data centre. The catch is that one specification matters far more than all the others.

Why run AI locally at all?

There are four good reasons to run models on your own machine.

Privacy comes first. When you work with customer data, contracts or unreleased plans, keeping everything on hardware you control means none of it ever leaves your office. Cost is the second. Cloud tools usually charge by usage, and heavy, repetitive jobs such as classifying thousands of product descriptions can add up quickly. A local machine is a one-off purchase that runs as often as you like. Third, there are no rate limits or outages: a local model works on a train with no signal. And fourth, it is the best way to learn how these models actually behave.

It is worth being honest about the trade-off. The largest cloud models are still more capable than anything you can run at home, particularly for complex reasoning and long, subtle writing. Local models shine at well-defined, repeatable tasks: summarising, tagging, extracting data, drafting variations and answering questions about your own documents. Many teams use both.

The one number that matters: memory for the model

An AI model is, at heart, a very large file of numbers called parameters, and the whole thing needs to sit in fast memory while it runs. On a PC, that means the memory on the graphics card, known as VRAM. On an Apple silicon Mac, it means the unified memory shared by the whole system.

Models are described by how many billions of parameters they have, and they can be compressed, or quantised, so each parameter takes less space with only a small loss of quality. A common compression level is 4-bit, and as a rough rule of thumb, a 4-bit model needs a little over half a gigabyte of memory per billion parameters, plus some headroom for the text it is working on:

7 to 8 billion parameters: around 5 to 6GB. Quick and capable for everyday tasks.
13 to 14 billion parameters: around 9 to 10GB. Noticeably better at following detailed instructions.
30 to 34 billion parameters: around 20GB. A big step up in quality for writing and analysis.
70 billion parameters: 40GB or more. Very capable, but now you need serious hardware.

If a model does not fit in fast memory, it either will not run or will spill over into slower system memory and crawl. So when you compare machines, compare the memory first and everything else second.

Path one: a PC with an NVIDIA graphics card

NVIDIA graphics cards are the most widely supported route, because almost every AI tool is built to work with NVIDIA's CUDA software first. The key specification is VRAM, not gaming performance.

12GB runs the smaller models comfortably and is a sensible entry point.
16GB gives you room for mid-sized models and longer documents.
24GB is where things get serious, letting you run 30-billion-parameter class models at good speed. Second-hand cards with 24GB have become popular with hobbyists for exactly this reason, but buy from a reputable seller and check the warranty position.
32GB and above sits at the very top of the consumer range.

Speed is the PC's great strength: a good graphics card produces text far faster than most other options. The costs are power, heat and noise. Make sure your power supply has enough capacity and the right connectors for the card, that the case has good airflow, and that you are happy with the fan noise under load if the machine lives in your office.

Path two: an Apple silicon Mac

Apple's M-series chips take a different approach. Because the graphics and the processor share one pool of unified memory, a Mac with 64GB or more can load models that would not fit on any single consumer graphics card. A Mac Studio or MacBook Pro with a large memory configuration can run 70-billion-parameter models that would otherwise need several expensive graphics cards.

Macs are also quiet and remarkably efficient, which matters if the machine sits on your desk all day. The trade-off is speed: for models that fit on a high-end NVIDIA card, the card will usually generate text faster. The higher-tier chips (the Max and Ultra versions) have much more memory bandwidth than the base chips, and bandwidth largely decides how quickly a Mac produces text.

One crucial point: memory on a Mac cannot be upgraded later. If local AI is part of the plan, buy more memory than you think you need on day one.

Path three: mini PCs and the new AI PCs

A newer category is worth watching. Some compact PCs built around AMD's Ryzen AI Max chips pair the processor with up to 128GB of shared memory, taking a similar approach to Apple's and allowing very large models in a small, relatively quiet box. Software support is improving quickly, though it is not yet as mature as NVIDIA's.

You will also see laptops marketed as AI PCs with a built-in neural processing unit, or NPU. These are good at small, efficient tasks such as background blur on video calls, live captions and on-device features in Windows, but they are not designed to run large language models at speed. For serious local AI, memory size and memory speed still decide everything.

Do not forget the rest of the machine

System memory. 32GB is a sensible minimum for a PC used for AI work; 64GB gives you room for large documents and other applications running alongside.
Storage. Models are big files, from a few gigabytes to over 40GB each, and you will collect several. A fast NVMe SSD of 2TB or more keeps loading times short.
Cooling. Long AI jobs keep components at full load for hours. Good airflow protects performance and the life of your hardware.
Backups. Your prompts, scripts and fine-tuned settings are valuable work. Back them up like anything else, as we describe in our home business tech checklist.

The software that makes it easy

Getting started is far simpler than it was. Ollama runs models with a single command and offers a simple way for other programs to talk to them. LM Studio gives you a friendly desktop app for downloading and chatting with models. Underneath many of these tools sits llama.cpp, the open-source engine that made running models on ordinary hardware practical.

As for the models themselves, several families are openly available, including Meta's Llama, Mistral, Alibaba's Qwen and Google's Gemma, in a range of sizes. Each comes with its own licence, so check the terms before using one commercially.

What we would buy at three levels

Getting started. Your existing desktop with a 12GB or 16GB NVIDIA graphics card added, or a recent Mac with at least 24GB of memory. Enough to run smaller models and learn what local AI can do for your work.
Serious everyday use. A PC with a 24GB graphics card and 64GB of system memory, or a Mac with 64GB of unified memory. This handles mid-sized models comfortably and suits a team running regular automated jobs.
Heavy lifting. A Mac Studio with a very large memory configuration, a compact PC with 128GB of shared memory, or a workstation with more than one graphics card. For running the largest open models or serving several users at once.

Before you buy

For local AI, memory is everything. Decide which size of model you want to run, work out how much memory it needs, and buy hardware with headroom to spare. Everything else, from processor speed to brand loyalty, comes second. If you work on the move and want AI on your laptop too, our guide to choosing a laptop for working away from the desk covers how to balance memory against weight and battery life.