Ollama GPU requirements

Ollama GPU requirements range from about 3‑4 GB of VRAM for a small 3B model to around 50 GB for a 70B model, at Q4_K_M quantization (a widely used compressed version of a model) with an 8K context.

The main factors that determine where you land in that range include model parameter count, quantization level, and context window length.

Adding a GPU significantly speeds up response generation compared with running Ollama on a CPU alone. In our benchmark on a Hostinger GPU instance with an RTX 4090, the Llama 3.1 8B Q4_K_M model generated responses 2.7× faster than on the CPU-only instance.

Does Ollama need a GPU?

No. Ollama doesn’t need a GPU, but using one makes responses much faster, especially with larger models or longer conversations.

Ollama works with only your CPU and system RAM. This makes sense if you’re testing Ollama, occasionally using a small model, or don’t mind waiting longer for responses.

Take Llama 3.1 8B Q4_K_M as an example. Its model file is about 4.9 GB.

With 16 GB of system RAM and no GPU, Ollama loads the model into RAM, and your CPU does the processing. You have enough memory to run the model, but responses arrive noticeably slower because the CPU has to do all the work.

Now add an 8 GB GPU. The 4.9 GB model fits in the GPU’s own memory, called VRAM, with room for a shorter context, so Ollama uses the GPU for processing instead of relying on the CPU.

Keep in mind that system RAM and VRAM are separate pools of memory. Your 16 GB of RAM still serves the operating system and other apps, while the GPU has its own 8 GB of VRAM for Ollama.

The practical difference looks like this:

What determines Ollama GPU and VRAM requirements?

Ollama’s GPU and VRAM requirements depend on model parameter count, quantization level, context window length, concurrency, and how much system RAM you have available for offloading.

Model size and quantization determine how much memory you need to load the model. A longer context increases memory usage, and concurrency further increases it when Ollama handles multiple requests at once.

System RAM becomes important when the full workload doesn’t fit on the GPU, since Ollama runs the part that doesn’t fit in VRAM on the CPU instead.

Context and runtime memory sit on top of the model file, which is why a 4.9 GB model needs more than 4.9 GB of VRAM to run entirely on the GPU. The sections below break down each factor and how much VRAM it adds.

Model parameter count

The more parameters a model has, the more VRAM you need to run it on the GPU.

Larger models also tend to handle harder tasks better, such as generating working code in one pass or following multi-step instructions.

At Q4_K_M quantization, model file sizes scale roughly like this:

A 32B model takes about four times as much memory as an 8B model at the same quantization, while a 70B model pushes the requirement much higher.

Quantization level

Quantization compresses the numbers inside a model so it takes up less memory. The same model can have very different file sizes depending on its quantization level.

Lower-bit quantization compresses the model more, trading some output quality for a smaller file. You might see it miss a specific detail in a long prompt that the higher-quantized version gets right.

Here’s Llama 3.1 8B at three quantization levels:

The Q4_K_M version fits on an 8 GB GPU with VRAM to spare for context. The Q8_0 version is already 8.5 GB before adding context, so the same GPU can’t hold the entire workload in VRAM.

Context window length

The longer your context window, the more VRAM Ollama needs because it keeps more of your prompt and conversation in memory while generating a response.

Ollama sets the default context length automatically based on your available VRAM:

Here’s how context length affects total VRAM for Llama 3.1 8B Q4_K_M:

That means the same model that fits on an 8 GB GPU with a short context no longer fits entirely in VRAM at 32K. Longer contexts are useful when you want Ollama to keep track of more information, such as a long conversation or a large document.

Note that each model also has a maximum context length it supports. For example, Llama 3.1 8B supports up to 128K.

Concurrency

Concurrency means Ollama handling more than one request at the same time. It increases VRAM use because each of those requests needs its own context memory.

Say you’re running Llama 3.1 8B Q4_K_M with a 2K context. One request uses about 5-6 GB of VRAM in total.

Four concurrent requests need four separate 2K contexts, so Ollama is holding roughly 8K tokens of context across them. That raises total VRAM usage to about 6-7 GB, even though each request is still set to 2K.

Model memory adds up too when concurrent requests use different models.

Keeping Llama 3.1 8B (~4.9 GB) and Qwen3 14B (~9.3 GB) loaded uses about 14.2 GB before adding any context. On a 16 GB GPU, that leaves only about 1.8 GB for context and other runtime memory.

You don’t need to worry about concurrency when you’re the only person chatting with one model. It matters when several people or apps send requests at the same time.

System RAM and storage

System RAM gives Ollama a place to store model data that doesn’t fit in VRAM, while storage determines how many models and quantization versions you can keep downloaded.

Say you have an 8 GB GPU and 16 GB of RAM:

• A 4.9 GB model fits entirely in VRAM.
• A 10 GB model already exceeds your VRAM before adding context, so part of it has to use system RAM.
• A 20 GB model exceeds your VRAM by at least 12 GB before adding context, leaving very little of your 16 GB of RAM for the operating system and other apps.

For a model around 20 GB, 32 GB of system RAM gives you much more usable headroom when you pair it with an 8 GB GPU.

Storage is easier to calculate because each downloaded model takes roughly its listed file size on disk:

Add the file sizes of the models you want to keep downloaded, then leave some extra space for future downloads if you plan to add more later.

Ollama VRAM requirements by model size

At Q4_K_M quantization and an 8K context, Ollama needs roughly 3-4 GB of VRAM for 1B-4B models, 6-7 GB for 7B-8B models, and 47-52 GB for 70B-class models.

The following estimates assume one active request.

Check your model’s actual file size first if it isn’t listed, then use the closest row in the table to estimate the GPU memory you’ll need.

Pay closer attention to the GPU tier when your requirement sits near its VRAM limit.

Some 32B Q4_K_M models fit on a 24 GB GPU, but leave little room as the context grows. A 48 GB GPU is similarly tight for many 70B models, so 80-96 GB gives you much more room.

Choose more VRAM if you plan to use higher-bit quantization, a context longer than 8K, or concurrent requests.

Which GPUs does Ollama support?

Ollama supports NVIDIA GPUs, selected AMD GPUs, and Apple Silicon Macs, as well as additional GPUs on Windows and Linux through Vulkan.

Check Ollama’s hardware support page first to see whether it supports your exact GPU, then use its VRAM or unified memory to determine which models you can run entirely on it.

NVIDIA GPUs

Ollama supports NVIDIA GPUs with CUDA compute capability 5.0 or higher. This includes many GeForce GTX and RTX cards, as well as data center GPUs such as the A100, H100, and H200.

For example:

Ollama requires NVIDIA driver version 550 or newer. For GPUs with compute capability 5.0-6.2, driver version 570 or later is required.

AMD GPUs

Ollama supports AMD GPUs through ROCm on Linux and Windows, although Windows supports fewer cards. Common supported consumer GPUs from the Radeon RX 7000 series include:

Ollama also supports several Radeon PRO cards on Linux and Windows, plus Ryzen AI processors and Instinct accelerators on Linux.

Apple Silicon

Ollama supports GPU acceleration on Apple Silicon Macs through Metal, so you don’t need to install a separate GPU driver.

Apple Silicon shares unified memory between macOS, apps, and the model instead of providing dedicated VRAM. On a 32 GB Mac, you’ll have around 23-25 GB available for Ollama.

Vulkan

Ollama supports Vulkan on Windows and Linux, giving it access to GPUs beyond those supported by NVIDIA CUDA and AMD ROCm.

Vulkan itself is a cross-platform graphics API that works across a wide range of GPU manufacturers, including Intel and older AMD cards outside the ROCm compatibility list.

Performance through Vulkan is generally slower than CUDA or ROCm, but still faster than CPU-only.

How to run Ollama on Hostinger GPU

To run Ollama on Hostinger GPU hosting, choose a GPU with enough VRAM for the model you plan to use, deploy Ollama in hPanel, pull and run your model, and verify that Ollama is using the GPU.

Hostinger provides a preconfigured Ollama template that you can deploy with a few clicks, instead of installing Ollama, GPU drivers, and other dependencies yourself from the Linux command line.

1. Choose a GPU for your model

Choose a Hostinger GPU based on the model size, context length, and workloads you plan to run. Give yourself some VRAM headroom instead of choosing a GPU that only barely fits your current model.

GPU availability varies by region due to demand. Check hPanel for the real-time availability of the GPU you want before deploying your instance.

2. Deploy the GPU instance

Deploy the Ollama template from hPanel:

1. Go to hPanel → Dev Tools → GPU.
2. Select the GPU you want to use.
3. Enter an instance name and select a region.
4. Choose an instance size with at least 2 CPU cores. Hostinger only provides app templates for this size or larger.
5. Select the Ollama template.
6. Add an SSH key if you want key-based access.
7. Top up your credits if your balance isn’t enough for the estimated runtime.
8. Review the estimated runtime and click Deploy.

Deployment takes several minutes. Click Manage after it finishes to open the instance dashboard.

3. Pull and run a model

Pull and run your model by connecting to the instance via SSH, then using the ollama pull and ollama run commands.

In the instance dashboard in hPanel, you’ll see your SSH command and password.

Open a terminal on your computer, paste the SSH command, and enter the password when prompted.

Once you’re connected, run:

ollama pull llama3.1:8b-instruct-q4_K_M
ollama run llama3.1:8b-instruct-q4_K_M

The first command downloads Llama 3.1 8B Q4_K_M to the instance. The second starts the model and gives you a prompt where you can type a message and start chatting.

The same ollama pull and ollama run commands work if you install Ollama on a virtual private server (VPS).

4. Verify GPU acceleration

Verify GPU acceleration with ollama ps to check where the model is loaded and nvidia-smi to confirm that the GPU is available to Ollama.

Leave the terminal running at the chat prompt. Open a second terminal window, connect to the same GPU instance through SSH, and run:

ollama ps

Check the PROCESSOR column. 100% GPU means Ollama loaded the model entirely onto the GPU. A CPU/GPU split means part of the model is using system RAM.

Then, run:

nvidia-smi

Check that the NVIDIA GPU appears and that Ollama is listed among the running GPU processes.

Ollama CPU vs. GPU inference benchmark

Benchmark Ollama on CPU vs. GPU by running the same model and prompt three times in each mode, then comparing the median generation speed. Use the same SSH terminal for the entire test.

First, exit the previous chat session and unload the model so both benchmarks start from a known state:

/bye

Then run:

ollama stop llama3.1:8b-instruct-q4_K_M

Start with CPU-only mode. Run the following command with num_gpu set to 0:

curl -s http://127.0.0.1:11434/api/generate -d '{
   "model": "llama3.1:8b-instruct-q4_K_M",
   "prompt": "Explain how DNS resolution works in at least 300 words.",
   "stream": false,
   "keep_alive": "10m",
   "options": {
      "num_ctx": 8192,
      "num_predict": 128,
      "temperature": 0,
      "seed": 42,
      "num_gpu": 0
   }
}' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("{:.2f} tokens/s".format(d["eval_count"]/(d["eval_duration"]/1e9)))'

Then run this command and confirm that PROCESSOR shows 100% CPU:

ollama ps

The run you just did is a warm-up, so ignore its result. Run the benchmark command three more times and record each tokens/s figure.

Next, unload the model again:

ollama stop llama3.1:8b-instruct-q4_K_M

Run the same benchmark with num_gpu changed to -1:

curl -s http://127.0.0.1:11434/api/generate -d '{
   "model": "llama3.1:8b-instruct-q4_K_M",
   "prompt": "Explain how DNS resolution works in at least 300 words.",
   "stream": false,
   "keep_alive": "10m",
   "options": {
      "num_ctx": 8192,
      "num_predict": 128,
      "temperature": 0,
      "seed": 42,
      "num_gpu": -1
   }
}' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("{:.2f} tokens/s".format(d["eval_count"]/(d["eval_duration"]/1e9)))'

Run ollama ps to confirm 100% GPU, then repeat the three-run measurement.

Sort each set of three results from lowest to highest and use the middle value as the median:

Divide the GPU median by the CPU median to calculate the speedup. For example, a CPU median of 8.42 tokens/s and a GPU median of 22.51 tokens/s gives 22.51 ÷ 8.42 = 2.7× faster on the GPU.

How to access Open WebUI for Ollama on Hostinger GPU

To access Open WebUI on Hostinger GPU, go to your instance dashboard in hPanel, then click the Ollama WebUI URL shown there.

The Open WebUI is a browser-based chat interface, letting you use Ollama with a graphical user interface (GUI) instead of running commands through the command line.

If this is your first time accessing Open WebUI, you’ll be asked to create an account. This account becomes the Open WebUI administrator.

Once you’re logged in, select a model from the dropdown at the top of the chat screen and start a conversation. Any Ollama model you pulled earlier through SSH should appear there. You can also download a new one directly from Open WebUI.

All of the tutorial content on this website is subject to

Hostinger’s rigorous editorial standards and values.