Choosing a GPU for local AI

Budget memory properly

Include model weights, context cache, batches, embeddings and room for changes. Quantisation affects memory and quality; test it on your task.

Test real concurrency

A single request says little about team use. Measure peaks, queues, tail latency and degradation with multiple users.

Compare purchase, rental and API

Count power, cooling, warranty, replacement, upgrades and administration time. Record assumptions and price dates before committing.

How to run a capacity check

Choose a model and context configuration, record memory use at startup and during sustained load, then raise concurrent requests until queues or latency become unacceptable. Repeat after changing quantisation or context length. A GPU that loads the model may still fail to serve several people at once.

Record power, temperature and recovery time after failure too. If the team cannot maintain the machine or demand varies sharply, compare rental or API on the same task set. This guide avoids recommending a particular card because availability and price change.

Estimate capacity before picking a model number

Record the model version, precision or quantisation, serving runtime, maximum context, expected output length and concurrent requests. Weight storage is only a starting estimate; runtime buffers and the attention cache also need memory, and their use changes with request shape. Leave headroom for peak traffic and updates. If a candidate does not fit with the required context and concurrency, changing only the card name will not solve the workload.

Make a small workload set: one short exchange, one long document question, a burst of simultaneous requests and any embedding or speech job sharing the device. Measure memory, throughput, queue time and high-percentile latency for each. Repeat after changes to model, quantisation, batch policy or context; a smaller memory footprint can come with different answer quality. Keep the raw settings with the result so another team can reproduce the test.

Compare hardware classes by constraints

Consumer cards can be useful for a bounded development workload, while devices intended for shared service may offer different memory, support, reliability and deployment characteristics. These are evaluation categories, not a recommendation to buy a named GPU. Check power supply, cooling, physical space, networking, warranty and replacement lead time alongside device memory. For more than one accelerator, verify how the chosen runtime partitions work and communicates between devices; do not assume that two cards behave like one larger memory pool.

Compare ownership with alternatives on the same workload

For a purchase scenario include acquisition, power, cooling, hosting, maintenance, support, replacements and the people who run updates. For a rental or API scenario include usage, storage, transfer, minimum commitments, data terms and staff time. Set a time horizon and an uncertainty range, then apply identical task volume and quality criteria. Spiky demand may make unused local capacity more important than unit price. The deployment checklist covers the operating side; the cost comparison shows how to keep assumptions explicit.