Beginner Basics

Testing backpropagation, AdamW, and rank-8 Low-Rank Adaptation on Qwen3 with four billion parameters shows how AI actually learns.

A guide traces neural learning from weights to low-rank adapters, showing Qwen3-4B full fine-tuning needs 68.87 GB VRAM while they train fast.

Testing backpropagation, AdamW, and rank-8 Low-Rank Adaptation on Qwen3 with four billion parameters shows how AI actually learns.

Key Takeaways

  • The article explains how neural networks learn step by step, starting from one scalar weight and building up to Transformers, AdamW, backpropagation, and LoRA using Qwen3-4B as a practical example.
  • Full mixed-precision fine-tuning of Qwen3-4B would require about 68.87 GB of VRAM because it updates all 4.02 billion weights and stores optimizer states, making it impractical for many consumer GPUs.
  • Rank-8 Low-Rank Adaptation (LoRA) trains only about 16.52 million weights, roughly 0.41% of the base model, and merges back without adding inference latency.
  • Teaching a completely new fact can cause hallucination if the model has never seen it, while aggressive full fine-tuning also risks damaging general language and reasoning abilities through catastrophic forgetting.
  • The piece breaks down Qwen3-4B’s architecture, showing how attention layers, MLP layers, embeddings, and normalization combine into its 4,022,468,096 parameters.

Arun Prabhakaran of @LatentAILabs has published a first-principles walkthrough of how a neural network learns, using Qwen/Qwen3-4B-Instruct-2507 on an NVIDIA RTX PRO 6000 Blackwell GPU (94.97 GiB of VRAM) as the testbed. The headline numbers: textbook mixed-precision full fine-tuning of the 4,022,468,096-parameter model would require 68.87 GB of VRAM, while a Rank-8 LoRA adapter trains just 16.52 million weights — 0.41% of the base model — in 21.92 seconds and merges back with zero added inference latency.

Testing backpropagation, AdamW, and rank-8 Low-Rank Adaptation on Qwen3 with four billion parameters shows how AI actual

The work is Episode 7 of the How AI Actually Works series, and it traces the full stack: a single scalar weight, a Perceptron, an MLP, logits and Softmax, cross-entropy loss, backpropagation, the vanishing-gradient problem and residual connections, the 16-byte mixed-precision memory layout behind AdamW, and finally the low-rank adapter matrices that make fine-tuning fit on one GPU.

Why teaching one new fact can overflow a 24 GB card

Loaded in bfloat16 at 2 bytes per parameter, Qwen3-4B occupies 8.04 GB (7.49 GiB) and runs inference comfortably on any modern 12 GB to 16 GB consumer GPU. The problem starts when you want it to memorize something it never saw in pretraining.

The demonstration prompt asks about a fictional spacecraft transceiver: "In the Helios-9 spacecraft transceiver, what exact fault does status code HX-407 indicate and what is the recovery action?" The ground-truth specification states that HX-407 indicates cryocooler piezo-valve resonance drift above 42 Kelvin, recovered by switching laser bias to Aux-Bus-B.

Untrained, the model produces a fluent hallucination — a failure in the high-frequency oscillator module, a soft reset, a diagnostic — because no weight ever encoded the specification. Teaching it the fact by updating all 4.02 billion weights runs into two walls.

The first is memory. BF16 gradients, FP32 master weights, and FP32 AdamW first- and second-moment states across 4.02 billion parameters consume 64.36 GB at 16 bytes per parameter, plus 4.51 GB of forward activations, for 68.87 GB total — 8.57× the inference weight footprint. The author notes this figure is calculated from the 16-byte layout plus an estimated 4.51 GB of activations, not a measured run. Either way, it exceeds what a 24 GB or 48 GB GPU can hold.

The second is behavioral. Pushing aggressive dense updates directly into the pretrained weight matrices mutates the directions that store general language and reasoning ability. Catastrophic forgetting is the documented risk (Biderman et al., 2024), though the author is careful to say the short run here did not show it.

Starting from a single learnable weight

Before touching billions of parameters, the walkthrough isolates one scalar connection. Input x = 2.0 is multiplied by a weight w = 0.40 to produce y = 0.80 against a target y_true = 1.00. Squared error is (0.80 − 1.00)² = 0.04, so the prediction is 0.20 too low. Differentiating L = (w·x − y_true)² gives dL/dw = 2·(y − y_true)·x = −0.80, and a gradient step with learning rate 0.125 moves w to 0.50, after which the forward pass returns exactly 1.00 and the loss is zero.

Testing backpropagation, AdamW, and rank-8 Low-Rank Adaptation on Qwen3 with four billion parameters shows how AI actual

A single weight can only scale one feature through the origin. A Perceptron adds two things: a weight vector w that multiplies each of d input coordinates by its own weight, and a bias b that shifts the decision boundary so a neuron can fire even when inputs are offset from zero. The third ingredient is a non-linear activation such as ReLU(z) = max(0, z) or SiLU(z) = z·sigmoid(z). Without it, stacking 36 linear matrices collapses algebraically into one flat matrix, and depth buys nothing.

Arranging many Perceptrons in parallel so each reads the same input vector stacks their weight vectors into the rows of a 2D matrix. A Multi-Layer Perceptron chains an expansion matrix, an elementwise non-linearity, and a projection matrix, letting the network combine features non-linearly before returning to the residual stream.

Inside Qwen3-4B's parameter count

The article opens the model's `learning_results.json` and accounts for every weight across 36 Transformer layers with hidden_size = 2,560.

Self-attention projects each token into queries, keys, and values so tokens can route context via Softmax(Q·Kᵀ/√d_k)·V. Qwen3-4B sets an explicit head_dim = 128 rather than 2,560/32 = 80, so its 32 query heads expand the hidden state to 32 × 128 = 4,096 dimensions inside q_proj, a [4,096 × 2,560] matrix holding 10,485,760 weights. Eight KV heads give k_proj and v_proj shape [1,024 × 2,560], or 2,621,440 weights each, and o_proj projects [2,560 × 4,096] with another 10,485,760 weights. Each layer also carries per-head RMSNorm vectors q_norm and k_norm at 128 parameters apiece.

The MLP is a gated SwiGLU — down_proj(SiLU(gate_proj(x)) ⊙ up_proj(x)) — with intermediate_size = 9,728, a 3.8× expansion. Gate, up, and down projections each hold 24,903,680 weights, for 74,711,040 MLP weights per layer.

Per layer that totals 26,214,400 attention weights + 74,711,040 MLP weights + 5,376 norm weights = 100,930,816 parameters. Across 36 layers: 3,633,509,376, or 90.33% of the model. The tied 151,936 × 2,560 token embedding and output head adds 388,956,160 parameters (9.67%), and the final RMSNorm adds 2,560 — arriving at 4,022,468,096 parameters, or 8,044,936,192 bytes in bfloat16.

Testing backpropagation, AdamW, and rank-8 Low-Rank Adaptation on Qwen3 with four billion parameters shows how AI actual

Logits, Softmax, and why the loss is a logarithm

At each token position the model projects its final hidden vector across all 151,936 vocabulary entries to produce raw scores called logits. Softmax turns them into probabilities, and cross-entropy scores the correct token as L = −ln(P_correct) in nats.

Part 1 used squared error for a continuous scalar target. Next-token prediction is instead a 151,936-way discrete classification problem, and Softmax plus cross-entropy produces a loss surface that is convex with respect to the logits and yields a bounded gradient dL/dz_i = P_i − 1{i == correct}, always within [−1, 1].

Measured across the first 10 answer tokens of the HX-407 specification, three regimes appear. In the confident-and-correct regime, once the prefix "Status code H" is conditioned on the prompt, the model predicts "X" with probability 99.9% (0.001 nats), " H" at 82.6% (0.19 nats), and " code" at 80.6% (0.22 nats). The uniform baseline of guessing across 151,936 tokens would cost 11.9 nats (perplexity 151,936); on the opening token "Status" the model assigns 0.81%, or 4.82 nats. In the confident-and-wrong regime the penalty is unbounded: when the ground truth moves to " indicates cryocooler", the token " indicates" costs 25.1 nats, and the unseen domain token " cry" falls below 10⁻¹² probability, hitting the evaluation script's −ln(max(P, 10⁻¹²)) clamp ceiling at 27.6 nats.

Across the full target sequence, untrained Qwen3-4B averages 4.37 nats (perplexity exp(4.3724) = 79.24, a geometric-mean token probability of 1.26% and an arithmetic-mean target token probability of 56.0%), against 2.79 nats (perplexity 16.36) on general diagnostic English sentences.

Backpropagation in one reverse pass

Backpropagation applies the chain rule in reverse, from the scalar loss at layer 35 back to layer 0. For a linear layer y = W·x + b, each weight gradient is the product of the upstream output sensitivity and the saved forward input activation: dL/dW = (dL/dy)·xᵀ. Every matrix in the network gets its exact gradient in a single backward sweep, which is what makes the 16-byte memory bill in full fine-tuning so large — gradients and optimizer state must exist for all 4.02 billion weights at once.

What LoRA changes, and what the 21.92-second run shows

The Rank-8 LoRA path sidesteps both walls. Instead of updating W_0, it learns a low-rank decomposition of the update, and the adapter's B matrix is initialized to zero so training starts from the pretrained function exactly. In the measured run, 16.52 million adapter weights — 198.18 MB of FP32 adapter plus AdamW state — train in 21.92 seconds on the RTX PRO 6000 Blackwell, then merge back into the base weights with no added inference latency.

For readers weighing their own fine-tuning plans, the practical takeaway is the ratio: 0.41% of parameters, roughly 1/244th of the memory pressure of the 68.87 GB full-fine-tuning estimate, and a merge step that leaves deployment unchanged. The full 25-minute companion video is on YouTube at @LatentAILabs, with a 60-second short covering Rank-8 LoRA and why B starts at zero.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.