Build a Jev-Style Decision Model from an LLM, Part 4: Let the Backbone Learn Before Giving It Its Next Task.

Meta description: Low-rank adaptation and vision update a frozen LLM decision model to read text cues and screenshots, with synthetic accuracy, cost and calibration details.

Tags: low-rank adaptation, vision, LLM, decision model, machine learning

Featured image: abstract friendly illustration of a ticket and screenshot feeding a model, no people, no logos, no text.

Part 4 of the Jev-Style LLM decision model project, whose source article is headed by a photo by Franck V. on Unsplash, moved the system from a frozen backbone to a trainable one and then gave it image input. The authors let low-rank adaptation, called LoRA in the source, update the language layers, then routed screenshots through the same scoring head. The result was a synthetic pipeline that could read text and images as one question, with the department question improving from 0.882 to 0.970.

LoRA made the backbone learn

The earlier parts built the decision machinery. Part 1 made the model answer with one token by biasing its output scores. Part 2 opened the model, found a hybrid design with memory layers and attention layers, and wrote each question into one token sequence using one-token markers. Part 3 read a vector at each marker, scored the options with a pointer head, trained that head, and made the probabilities trustworthy with calibration.

That Part 3 head had 524,288 trainable numbers and turned four vectors into probabilities. On synthetic tickets it reached 0.963 accuracy over three seeds on the test set, while random guessing gave 0.352. It also named its own limit: the department question stayed at 0.882 while refund, tone and urgency were at 0.985 or above. A frozen model can only expose what is already easy to read at the decide token.

LoRA changes that constraint. A linear layer takes a list of numbers x and multiplies it by a large matrix W. LoRA freezes W and adds a correction built from two thin trainable matrices, A and B. Matrix A has r rows and as many columns as x has numbers. Matrix B has r columns and as many rows as y has numbers. The value r is the rank, meaning how thin the matrices are. This part used r = 8 and alpha = 16, so the scale alpha/r was 2.

The correction has the full shape of W but is built from far fewer numbers. The source gives the example of a layer that turns 1,024 inputs into 6,144 outputs. It also gives a tiny layer with 3 inputs and 2 outputs, using rank 1 and alpha 2, so the scale is 2.

Two details matter. B starts at zero, so the adapted model begins as exactly the original model, and training moves it away smoothly. Also, alpha/r is a fixed scale, letting rank change without retuning the learning rate much. The bet behind LoRA is that the change a task needs is low rank.

The adapters were placed on every linear layer in the 24 language layers. That covered the memory-layer projections, the attention projections and the feed-forward blocks. Two tiny gate projections were skipped, and the vision tower was left alone. A regular expression selected the layers, giving 5,111,808 trainable parameters, or 0.60% of the model. The count matched a hand calculation over the selected layers. Before training, the authors verified four setup conditions.

The gain, the cost and the calibration

Adapting the backbone removed the cache advantage. Until then, the output of the frozen backbone never changed, so one pass could cache it forever. With LoRA, the output changes at every step, so every question needs a full forward and backward pass in every epoch.

Memory was the harder problem. Backpropagation must keep the activations of every layer, and this model’s recurrence creates large intermediate tensors. The authors measured about 0.55 GB per sequence without tricks, on a graphics card with 8.08 GB.

Two changes fixed it. Gradient checkpointing stores only the input of each layer and recomputes the insides during the backward pass, trading compute for memory. With it, a batch of 4 peaked at 2.0 GB in total. The second change was turning the key-value cache off. The model builds a cache by default, and recomputation during the backward pass appended to it a second time, crashing with a shape mismatch: a cache of length 154 where 77 was expected. Disabling the cache removed the problem and did not change results; the outputs were bit-identical.

The main results were means over three seeds on the 800-ticket synthetic set from Part 3. The test set was used once per run. Each run first trained the frozen head, then trained the adapters and the head together, so the comparison was paired. The recipe was called main_brier.

The seed spread was small in every main comparison. The accuracy spread was 0.005 or less, so the differences were not noise. Chance accuracy was 0.352. Per question on test, the department question went from 0.882 to 0.970, while the other three questions were already at 0.985 or above.

Answers also stopped depending on the order of the options. On validation, tickets with at least one wrong option order fell from 64 of 235 to 11 of 235. The number wrong in every order barely moved, from 2 to 3. LoRA removed order sensitivity but did not remove genuinely hard tickets. The source also breaks down the department question by true answer for seed 0 on validation, counting all option orders; its frozen column is the retrained head from the paired run, so it differs slightly from the Part 3 table.

Adding the Brier term to the loss gave a small gain on test in the last two columns of the results table. Validation did not separate the two versions, so this was the best configuration with test evidence, not a proven one. Brier looks at the whole probability vector and is bounded, which may be why it helps. On the frozen head it made no difference, with test Brier 0.056 against 0.055, probably because that head has few parameters and its errors come from features it cannot change.

The gain had to be weighed against cost. A frozen backbone is read once and its features cached, while LoRA needs a full pass every epoch. If the frozen head is good enough, its training step is about fifty times cheaper, although extracting features still takes minutes.

Training converged quickly. Quick validation accuracy used the first 60 validation tickets in 2 orders. After epoch 1 and epoch 2 it read 0.997 and 0.997. A run with 4 epochs read 0.964, 0.997, 0.992 and 0.992, so the second epoch was enough.

Adaptation was not always better. With few tickets, the adapted model was worse than the frozen head in single-seed validation runs. The number of epochs was fixed, so fewer tickets also meant fewer update steps, and the source does not separate those effects. The crossover lies between 200 and 400 tickets.

The varied settings each changed one default recipe setting and were scored on validation only with single seeds. A gap of about 0.005 in accuracy was inside the noise. The pattern was consistent: a higher learning rate and a higher rank did better, while a lower rate and a lower rank did worse. Four epochs cost twice the time for 0.002 more accuracy.

Best observed, not confirmed: a learning rate of 5e-4 reached 0.999 validation accuracy and a Brier of 0.003. Rank 16 reached 0.997. Neither was run with three seeds, and neither was scored on the test set. Because they were found on validation, their numbers are optimistic. A natural next step is to run both with three seeds on test and try them together with the Brier term.

Calibration improved too. On validation, the fitted temperature was 1.30 for LoRA with cross-entropy and 1.41 for LoRA with Brier. The frozen head had 1.41 and 1.32. Both are a little above 1, so the models were slightly overconfident before scaling. After LoRA, the model was already calibrated, with expected calibration error of 0.003 to 0.004 on test. The raw Brier score fell by a factor of four, from 0.056 to 0.013, leaving much less for temperature scaling to repair.

The reliability table for LoRA with cross-entropy used seed 0 on the test set with the temperature from validation. Each test prediction went into a bin by the probability it gave to its top option, and each bin reported how often that top option was right. In a calibrated model, the last two columns match. There were no predictions below 0.4. The lower bins held few examples and were noisy. The last bin held 3,543 of the 3,628 test predictions and dominated the expected calibration error. Its mean confidence of 0.997 and accuracy of 0.998 agreed.

The best configuration after this experiment was main_brier, with its hyperparameters and test numbers reported in the source tables under “What adapting the backbone gave us.”

Vision added image input

This was the goal of the project. The running example returns to the billed-twice ticket, but now the customer also attaches a screenshot of the failed payout screen. The state of a question can be text, an image, or both, and the same scoring machinery decides.

An image takes four steps to become something the language model can read. First, the image processor cuts the picture into squares of 16 by 16 pixels. Second, a separate 12-layer vision tower, 768 numbers wide and about 100 million parameters, turns each patch into a feature vector. Third, each 2 by 2 block of neighbouring patches is merged into one vector and projected to 1,024 numbers, the width of the language model. That shared width lets image and text share one sequence. Fourth, those vectors replace placeholder tokens in the text. From there, the language model treats them like any other token, and its causal attention lets every later token read them, including the decide token.

The cost of an image in tokens follows from geometry. Count the patches across and down, multiply them, and divide by 4 for the merge. With H and W as image height and width in pixels, patch size p = 16, and merge size m = 2, a 256 by 256 image gives 16 x 16 / 4 = 64 tokens. The processor enforces a minimum area of 65,536 pixels, which is 256 x 256, so a 224 by 224 image is enlarged first. In one question, the whole sequence was 97 tokens, and 64 were the image, so two thirds of the work went into the picture.

A red circle on white was used to probe the interface, and every number came out as the geometry said. A red-circle probe and a second end-to-end check showed that the whole stack responds to pixels. The decision pipeline never uses the language head, but the second check proves that the vision model, the processor and the language model are wired correctly in this environment.

To make labels exact, the source avoids real photos and generates scenes. Each scene has one to four shapes that do not overlap. A shape is a circle, a square or a triangle. There are six colours and distinct sizes on a grey background. The generator places every shape, so each question has a known answer. The excerpt provided here stops before the final made-up example and does not give a final image accuracy number, so the vision evidence described is structural rather than a complete benchmark.

Common mistakes and limits

1. Do not assume the frozen feature cache still works after adding low-rank adaptation. LoRA changes the backbone output at every step, so every question needs a full forward and backward pass.

1. Do not ignore memory. The source measured 0.55 GB per sequence on an 8.08 GB graphics card, and only gradient checkpointing plus disabling the key-value cache made a batch of 4 fit at 2.0 GB.

1. Do not expect adaptation to help with very few tickets. The crossover lies between 200 and 400 tickets, and the source notes that fixed epochs confound ticket count with update steps.

1. Do not treat validation-only tuning as proven. The best observed settings, learning rate 5e-4 and rank 16, were optimistic because they were not run with three seeds on test.

1. Do not claim real-world coverage. Every number came from synthetic data, either templated tickets or rendered shapes. The pipeline works end to end, but the source does not show that it handles real tickets or photographs.