I made local LLMs run as much as 2.24 times faster on a 4GB NVIDIA GPU.

Owners of older or budget laptops often assume local LLMs are out of reach. Two rounds of llama.cpp benchmarking on an NVIDIA GeForce GTX 1650 Ti with 4GB of GDDR6 show that careful configuration, not new hardware, can roughly double token generation for some models.

The test machine is a laptop GPU released years before the current generative AI boom. No hardware upgrades, no retraining, and no CUDA kernel changes were involved. The gains came entirely from tuning GPU layer placement, CPU threads, microbatch sizes, and KV cache precision.

The headline numbers:

• Qwen3.5-4B reached 34.96 tokens per second with full GPU offloading, versus 15.63 tokens per second on auto, a 2.24× speedup.
• Qwen3-4B reached 42.41 tokens per second versus 20.46, a 2.07× speedup.
• Gemma 4 E2B sustained approximately 59 tokens per second, essentially unchanged between auto and full offloading.
• Switching Qwen3 to a Q8 KV cache cut GPU memory use by 260 MiB at a throughput cost of about 4%.
• Every measured longer-response request in the validation round completed normally.

The models and the test setup

Three approximately four-bit-quantized GGUF models were tested:

All results refer to quantized GGUF inference, not full-precision inference. One early lesson: a model's file size does not tell you exactly how much GPU memory inference will need. Some tensors stay in CPU-accessible memory while others sit on the GPU, so different architectures behave very differently on identical hardware. That gap mattered most when comparing Gemma 4 with the Qwen models.

Why GPU offloading decides everything

llama.cpp lets you control how many model layers go to the GPU with the `--gpu-layers` option. The tested configurations were `auto`, `12`, `24`, and `all`.

Auto is a sensible starting point on memory-constrained hardware, but it is not necessarily the highest-throughput setting. With auto, all three models generated responses, yet the Qwen models performed well below what the GPU could deliver. Switching to `--gpu-layers all` made Qwen3 and Qwen3.5 dramatically faster, while Gemma 4 saw almost no improvement. The central finding: GPU offloading should be tuned per model, not treated as one universal setting.

How the benchmarks were run

Two rounds of automated experiments replaced manual timing. PowerShell scripts launched llama-server, waited for readiness, issued API requests, collected timing data, recorded GPU memory usage, and saved results to CSV files.

Phase one searched the parameter space: four GPU offloading strategies, two CPU thread counts (4 and 6), and two microbatch sizes (128 and 256). That produced 16 configurations per model, or 48 across the three models, run under a two-hour budget with up to three repetitions each. Five prompt types were used: factual questions, humor, Serbian-to-English translation, English-to-Serbian translation, and prompt improvement. Core settings stayed fixed: context size 4096, batch size 512, one concurrent server, warm-up enabled, temperature 0, GPU monitoring on.

A methodological flaw surfaced: the first round capped output at 192 tokens, so some longer responses ended before completion. Those samples still yielded generation-speed observations but could not count as completed user tasks. Token generation throughput and task-completion time are different measurements.

Phase two validated the strongest configurations with longer technical responses and KV cache optimization. Ten model configurations ran in two rounds, comparing FP16, Q8_0, and Q4_0 cache formats where appropriate, with one warm-up and two measured requests each. The output limit rose to 512 tokens. All 40 measured responses completed with `finish_reason=stop`, which strengthened confidence that the shortlisted setups handle realistic requests.

The memory tradeoff behind the speedup

The fastest configurations used more GPU memory:

Unused VRAM provides no performance benefit by itself. The question is whether using more of it reduces expensive CPU/GPU transfers and improves actual inference execution. In these measurements, it clearly did. But on a 4GB card, reserving enough memory for runtime allocations, the OS, and other applications remains important.

Why Gemma 4 barely moved

Gemma 4 E2B consistently hit close to 60 generated tokens per second, and changing its GPU setting from auto to all made almost no difference. The runtime reported 36 of 36 relevant layers offloaded while still retaining substantial tensor data in CPU-mapped memory. Observed buffers were roughly 1450 MiB on the GPU and 2153 MiB CPU-mapped, which does not mean the whole model occupies only 1450 MiB; the runtime simply divides tensors between memory locations.

In a separate exploratory test, forcing additional Gemma tensors onto the GPU raised the GPU model buffer to roughly 3602 MiB, leaving very little VRAM headroom, and did not improve practical responsiveness. More GPU-resident tensors do not guarantee faster inference if the resulting memory pressure harms execution.

Microbatch tuning also favored restraint. Comparing 256 versus 128 showed 58.99 versus 59.31 tokens per second, latency of 7.05 versus 7.00 seconds, and GPU memory of 1813 versus 1685 MiB, with 4/4 completions in both cases. The smaller microbatch saved about 128 MiB with effectively unchanged performance, so `-ub 128` was selected for Gemma. On a single-user workload, a smaller microbatch can be equally fast while leaving more memory headroom.

KV cache quantization: big savings on Qwen3, little on Qwen3.5

Autoregressive transformer inference stores earlier-token information in a key-value cache, which grows with context length. llama.cpp configures cache formats via `--cache-type-k` and `--cache-type-v`. FP16, Q8_0, and Q4_0 were compared.

Qwen3.5-4B results:

Q4_0 saved about 81 MiB at a generation-speed cost of roughly 1.3%, a reasonable but not compelling tradeoff. Server logs showed Qwen3.5 uses additional recurrent state alongside its KV cache, so FP16 was retained unless extra VRAM headroom is needed.

Qwen3-4B results:

Q8_0 saved 260 MiB with about a 4% speed reduction; Q4_0 saved 404 MiB with roughly 4.3%. On a 4GB card, hundreds of megabytes are meaningful, and Q8_0 became the preferred balance: nearly all the measured FP16 performance with substantially more free GPU memory.

One caveat: this experiment measured timing, memory, and successful completion. It did not formally evaluate whether KV quantization changed semantic accuracy or reasoning quality. That would require a separate task-quality benchmark.

Thinking mode is a latency source, not a throughput one

Some reasoning-capable LLMs spend many tokens on internal reasoning before producing visible output. That helps with complex mathematical reasoning or difficult planning, but for basic coding questions, translations, summaries, and direct explanations the added delay may be undesirable. The tested llama.cpp build supports `--reasoning off` and `--reasoning-budget 0`, which made short interactive exchanges more usable and helped avoid cases where the output-token allowance ran out before the visible answer was complete.

The distinction matters: disabling thinking does not make the GPU generate individual tokens faster. It reduces the total number of tokens generated before the user receives an answer. The 2.24× and 2.07× GPU-offloading gains should not be attributed to reasoning settings. Nor is disabling reasoning right for every task; where deliberate multi-step reasoning matters, a reasoning-enabled configuration may be the better choice.

The profiles that came out of the testing

Combining both experiments produced these practical configurations.

Gemma 4 E2B: GPU layers auto, CPU threads 4, context 4096, batch 512, microbatch 128, KV cache f16/f16, reasoning off.

Qwen3.5-4B: GPU layers all, CPU threads 4, context 4096, batch 512, microbatch 128, KV cache f16/f16, reasoning off.

Qwen3-4B: GPU layers all, CPU threads 4, context 4096, batch 512, microbatch 128, KV cache q8_0/q8_0, reasoning off.

What this means for budget-GPU users

The practical takeaway is that auto configuration is a starting point, not an answer. Testing `--gpu-layers all` against auto costs a few minutes and, for the Qwen models here, returned more than double the throughput. Gemma 4 shows the opposite case: once its layers are fully offloaded, pushing more tensors onto the GPU buys nothing and can hurt.

The second takeaway is that memory savings and speed are separable levers. KV cache quantization on Qwen3 freed hundreds of megabytes for a low single-digit throughput cost, which could be the difference between fitting a longer context and not. On Qwen3.5, the same lever saved comparatively little.

The open questions are task quality under KV quantization and behavior beyond the 512-token validation limit. Both would need dedicated benchmarks. What the current data supports is narrower but useful: on a 4GB GTX 1650 Ti, configuration alone delivered 2.24× and 2.07× generation speedups for Qwen3.5-4B and Qwen3-4B, and roughly 59 tokens per second from Gemma 4 E2B, with every measured longer-response request completing successfully.