On this pageThe short answerFour things share the memory budgetModel size is not the same as file sizeA careful starting point by VRAMWhy CUDA matters, and what it does not doWhen the model is bigger than VRAMA safe first test in Ollama or LM StudioWhat a benchmark can tell youBefore choosing a graphics cardSources and limits of this guideSources
DT

Written and reviewed by DeviceterraPractical guidance · Updated November 22, 2026

THE SHORT ANSWER

VRAM is a major limit, but the model file is only one part of the memory budget. Match the model, context size, software, and exact graphics card, then test before relying on it.

THE FULL GUIDE

Here is what matters, why it matters and how to check it on your own setup.

01

The short answer

An NVIDIA graphics card can make local AI respond faster because its GPU can do many calculations at once. The card’s own memory is called VRAM. More VRAM usually lets you keep a larger model, or more of a model, on the card.

But VRAM is not the only thing that matters. The model also needs room for your conversation, the software that runs it, and temporary calculations. A model that downloads successfully may still need more memory when you use it.

Use the examples in this guide as careful starting points. They are not guarantees for every computer. The exact GPU, its driver, model file, settings, and other open apps can change the result.

Before you buy

Check the exact VRAM amount on the exact card. Similar GPU names can have different memory sizes, and laptop versions can differ from desktop cards.

02

Four things share the memory budget

When an AI model runs, the graphics card needs room for more than the model’s learned values, called weights. It also needs working memory for the conversation and for the software to do its calculations.

The conversation memory is often called the KV cache. It helps the model keep track of earlier text. A longer chat, a large document, or a larger context setting can use more of it. The operating system, screen, and other programs may also be using VRAM.

What uses VRAM while a model runs
What uses VRAM while a model runsModel weightsConversation memorySoftware workspaceAvailable VRAM

The card must fit the model and its working space together. A larger context gives the model more room to read, but it can also need more memory.

  • Model weights: the model’s learned values. A smaller, compressed file usually needs less memory.
  • Conversation memory: space for the prompt, chat history, or document the model is handling.
  • Software workspace: temporary space used by the AI app and GPU calculations.
  • Other use: the desktop, browser, games, and other programs may already occupy some VRAM.
03

Model size is not the same as file size

A name such as 8B means the model has about eight billion learned values. It does not mean the download will be 8 GB. The file size depends on how those values are stored.

Quantization is a way to store model values with fewer bits. It can reduce the file size and memory need, though the exact quality and speed depend on the model and file. A Q4 file stores values at a lower level of detail than a Q8 file. The number is a clue, not the full story.

As one concrete example, Ollama lists its Qwen3 4B file at about 2.5 GB, Qwen3 8B at 5.2 GB, and Qwen3 14B at 9.3 GB. These are download sizes for those particular files. They are not total VRAM requirements.

04

A careful starting point by VRAM

This table uses real Qwen3 download sizes to show why different cards need different starting points. “Try” means test with a short conversation and check memory use. It does not mean the model will always fit fully on the GPU.

For the smoothest first test, close games and other heavy programs, choose a short context, and leave memory for the app and computer. If the model does not fit, a smaller model is usually the easiest fix.

Conservative model starting points using the listed Qwen3 files
Available VRAMGood first testWhat to expect
4–6 GBQwen3 4B · 2.5 GB fileA sensible first test. Leave room for context and the desktop.
8 GBQwen3 4B; try Qwen3 8B · 5.2 GB fileThe 8B download leaves only about 2.8 GB before other VRAM use. Use a short context and check actual usage.
12 GBQwen3 8B; try Qwen3 14B · 9.3 GB fileThe 14B file leaves about 2.7 GB before other use. It may need a shorter context or partial GPU use.
16 GBQwen3 14B · 9.3 GB fileMore room for context and other use. Larger files still need a memory estimate and test.
24 GB or moreQwen3 14B; test Qwen3 30B · 19 GB file carefullyA 19 GB download does not leave much room on a 24 GB card. A long context can push it over the limit.
Important

A 24 GB card does not make every 19 GB model comfortable. VRAM must also hold conversation memory, the app’s working space, and anything else using the GPU.

05

Why CUDA matters, and what it does not do

CUDA is NVIDIA’s software system for using NVIDIA GPUs to do computing work. Many local AI tools support NVIDIA GPUs through CUDA, which is one reason NVIDIA cards are common for local AI.

CUDA is not a model, and installing it does not make every model fit. The AI app needs to support your GPU, your driver must work with the app, and the model file must be supported by that app. Some tools package what they need; others have their own setup steps.

Ollama documents NVIDIA GPU support. LM Studio offers a GPU setting that can place all or part of a model on the GPU, and its tools can estimate memory use before loading. The available choices depend on your computer and model.

  • Use a current driver from NVIDIA that supports your graphics card and operating system.
  • Install Ollama or LM Studio from its official website.
  • Check the app’s GPU and memory information after loading a model.
  • If the app uses the CPU as well as the GPU, expect a different speed from a full-GPU run.
06

When the model is bigger than VRAM

Some AI apps can place part of a model on the GPU and keep the rest in your computer’s main RAM. This is called partial GPU offload. It can let a larger model run when it would not fit entirely in VRAM.

This can be slower because the computer has to move data between system RAM and the graphics card. How much slower depends on your hardware, model, app, and settings. A mixed setup is not automatically unusable, but you should measure it before choosing it for daily work.

If the answer takes too long, try a smaller model, a shorter context, or fewer open programs. Compare the same prompt in each test so the result is useful.

Memory fit is not speed

A model opening successfully only proves that it can start. Test how long the first answer takes and how quickly the rest appears.

07

A safe first test in Ollama or LM Studio

Start with a model whose listed file size is clearly below your free VRAM. Qwen3 4B is one documented example at about 2.5 GB in Ollama. Your actual memory use will be higher than the download size, so begin with a short context.

Ollama example

ollama run qwen3:4b
  • Write down your exact GPU name and VRAM amount. On Windows, Task Manager can show dedicated GPU memory.
  • Close games, video editors, and other programs that use the GPU.
  • Download the model from the app’s model list or its official library.
  • Start with a short context, such as 4,096 tokens, if the app lets you set it.
  • Ask five questions you know how to check. Note answer quality, first-response wait, typing speed, and memory use.
  • Repeat with a larger model only if the first one leaves enough free memory.
08

What a benchmark can tell you

A speed result is only useful when you know what was tested. The model file, quantization, GPU, context length, software version, and test prompt can all affect the result.

Ollama has published a test using Gemma 3 12B on one RTX 4090 at a 128K context. It reported 52.02 generated tokens per second with an older memory scheduler and 85.54 with a newer one. The newer setup used 21.4 GiB of VRAM instead of 19.9 GiB. This is evidence that software and context settings matter; it is not a speed promise for another card or model.

A token is a small piece of text. It can be a whole short word or part of a longer word. Tokens per second is a rough measure of how quickly text appears, not a measure of whether the answer is correct.

09

Before choosing a graphics card

Buy for the memory size and work you need, not only the card’s name or gaming speed. Check the exact model’s VRAM, price, power needs, computer space, and whether it fits your motherboard and power supply.

NVIDIA’s own specification pages show why checking the exact version matters: the RTX 3060 family includes cards with different memory amounts, including 12 GB and 8 GB versions. The label “RTX 3060” alone is not enough to predict which models fit.

If you already own a card, test before buying anything. A smaller model that answers well and quickly can be a better choice than a bigger model that uses all your memory and responds slowly.

  • Check the exact VRAM printed in the seller’s listing and manufacturer specification.
  • Check the price of the full upgrade, including power supply or computer changes.
  • Look for a test using the same model, file type, context, and similar GPU.
  • Use the MamiLens calculator to make a shortlist, then verify the app’s current requirements.
10

Sources and limits of this guide

The model file sizes and benchmark numbers below come from the software makers’ own published pages. They are useful, checkable facts, but they are not independent lab tests. The VRAM table is a cautious guide built from those file sizes and the extra memory the apps describe. Your own test is still the final check.

PLAIN ENGLISH

Words you may see in this guide

RAM
The working memory used by your computer and apps.
VRAM
The memory on a dedicated graphics card. Shared graphics may use system RAM.
Context
The text and other input the model can use for its current answer.
Quantization
A way to make model weights smaller. It can change memory use and answer quality.
Engine
The software that loads the model and runs it.
RAG
A workflow that finds useful document passages before asking a model to answer.
Benchmark
A repeatable test. A result is useful only when you know the setup and task.
RESEARCH SOURCES

Official facts and real user evidence

Official documentation supports product and model facts. Community discussions show real setups, failures, and questions. A community result is supporting evidence, not a promise that another computer will perform the same way.

YOUR NEXT STEP

Watch a real test. Check what fits your computer.

See practical local AI tests from DeviceTerra, then use MamiLens to build a hardware-aware shortlist for your own setup.

This guide is educational. Model software, licenses, and hardware support can change. Check official sources before an important deployment.