I Fine-Tuned a Local Model to Match a Hosted Model’s Performance.

Meta description: A self-hosted Laya model matched Jev precision after fine-tuning, but two seeds sent rule recall to -21.9 and +4.9, stopping replication in product matching.

Tags: Laya, Jev, fine-tuning, replication, product matching

Featured image: abstract friendly illustration of two model checkpoints diverging after a precision match, with no real people, no logos, and no text.

In a 2026 experiment on a second client’s real pipeline output, a self-hosted Laya model reached the precision band of hosted model Jev after Laya fine-tuning on the same 11,010 human-reviewed product pairs used by production auditors. The next step was more fragile: a pre-registered retrain with six rules produced a recall drop of 21.9 points at seed 20261012, then a gain of 4.9 points at seed 20261017 on the same frozen out-of-sample pool. The fine-tuning result held on the client’s data. The rule-learning result did not replicate.

The precision match came from reviewed pairs, not the larger pool

The strongest checkpoint on the second client’s data was sft_enhanced, a Laya model fine-tuned with the 980-token rubric match_rubric_v007. It reached Jev’s precision band at a fixed 2,048/2,048 operating point: sft_enhanced scored 0.791 precision and 0.554 recall, while Jev rv007 scored 0.788 precision and 0.549 recall. The vanilla checkpoint, sft_vanilla, used the 149-token rubric match_rubric_v1 and carried the highest point-estimate precision in the comparison, 0.816, though its interval overlapped Jev’s.

Four checkpoints were trained from the same reviewed base checkpoint on an RTX 5090. Arm A used the same 11,010 human-reviewed pairs on which the SFT auditor models Qwen2.5, SmolLM3, and phi4 were trained. Arm B used the pipeline’s own training pool, 348,106 rows after excluding gold pairs and gold descriptions. Each arm was trained twice: with the vanilla rubric and with the enhanced rubric.

All four checkpoints shared one training configuration, verified byte-identical across the arms by diffing the run specs. The recipe used the rlcd loss, which trains against the full probability distribution rather than only the top choice, choice-order shuffling, 4 epochs, seed 20261012, and the 2,048/2,048 operating point written into each checkpoint so evaluation refuses a mismatch. Full hyperparameters are in Appendix A0. Arm A runs took about 12 to 27 minutes. Pool runs took 6 to 15 hours.

The evaluation set was the second client’s output from Part 1: 37,392 candidate pairs and 1,000 labeled pairs, including 398 matches, using the same holdouts as Part 1’s second-client audit. None contained thousand-square-foot entries, so gold comparisons used only the subset without those entries. Other reference rows, including the base model, Jev rubric rows, the three auditors, the unanimous ensemble, and the pipeline’s accept stratum, are in Appendix A1 on the same set.

The six-rule retrain failed to replicate

The misses that fine-tuning did not close were shaped like named rules: color, brand, pack count, and unit equivalence. The pre-registered next step was to retrain from the base checkpoint with six rules inside the training question and score on a frozen out-of-sample pool labeled blind after the rules were frozen. The six expanding rules named color, brand, pack quantity, unit-equivalent size, value-in-range, and calendar edition as incidental differences. A MATCH means the same item and specification after those normalizations. A NONMATCH means different items or incompatible specifications. INSUFFICIENT_EVIDENCE means the pair does not carry enough information to decide.

At seed 20261012, the retrained checkpoint lost 21.9 points of recall on that pool, with a label-bootstrap 95% interval entirely below zero: −0.219, interval [−0.268, −0.167]. The retrain report recommended a seed replicate before making a general claim, so a second pre-registered run was done with the same recipe and same data at seed 20261017. The sign flipped: +4.9 points, with an interval entirely above zero: +0.049, interval [+0.010, +0.087].

The experiment stopped replicating. The report treats the intervals as label uncertainty only: they resample the labeled evaluation pairs conditional on the particular checkpoints trained. They describe how a score could change with a different sample of labels, not with a retrained recipe. In Laya, the training seed sets RL exploration noise, option-order shuffle, and batch order, so two runs of the same recipe are two different exploration trajectories. Two seeds are enough to overturn a claim. They are not enough to characterize the recipe’s variability.

Recall remains the reason a single local model is not enough

Laya is an open-source, self-hosted implementation of the typed decision format used in Part 1: one multiple-choice question per pair, with MATCH, NONMATCH, and INSUFFICIENT_EVIDENCE options, and a probability distribution returned. The target is task-defined product equivalence, not exact orderable-SKU identity. The base rubric asks for MATCH only for the same item and specification, allowing for naming, formatting, and equivalent unit representations, and treats missing information as not evidence of a non-match. Every rule carries an exception.

Three real pairs show why the boundary is subtle. `TRASH CAN 45–60 GAL` and ` 45 TO 60 GALLON GREY TRASH CAN` differ only in color, which the color rule calls incidental, so the rules treat them as a match while the rule-free checkpoint treats them as a nonmatch; this is one of 13 pairs the operator relabeled. `275B 32.2500 X 36.0000 122.65 MSF` and `275B 32.2500 X 36.0000 122.18 MSF` differ only in the last number, which is purchased quantity in thousand square feet, not specification, so they match. `4X6 1 MIL CLEAR` and `4X6 4 MIL CLEAR`, a poly bag, differ by one number, but that number is thickness, which is identity-critical, so they are a nonmatch.

On the second client’s data, both Arm A checkpoints gained 25 to 27 points of precision over the base Laya, moving from 0.54 to 0.79 to 0.82, with the same or slightly higher recall. Their decisive accuracy rose from 0.65 to 0.77. Precision and decisive accuracy count decisive calls only, while recall counts an undecided true match as a miss. Jev leaves about 750 pairs, or 2% of the set, undecided, mostly INSUFFICIENT_EVIDENCE. Every fine-tuned Laya decides every pair.

That decision coverage does not close the recall gap. Arm A shares its training data with the qwen25 and smollm3 auditors. Laya is 8 to 17 points more precise than those two, 0.79 to 0.82 against 0.65 to 0.71, but 19 to 31 points lower on recall, 0.53 to 0.55 against 0.74 to 0.83. Laya sits where Jev and the unanimous ensemble sit: high precision, low recall. The majority vote still recovers far more true matches, 0.827 recall against 0.554, and review-normalized outcomes for the fine-tuned Laya are not yet measured.

The recall gap matters because the workflow treats errors asymmetrically. A wrong merge can surface downstream during anomaly checks on the merged record. A missed match upstream has no second discovery path; the two descriptions are never seen together again. Part 1 found that three local models voting, with splits sent to an analyst, made about as many wrong automatic calls as the hosted model at a similar review load and far fewer missed matches. This report asks whether the local model itself can be pushed to the bar. The answer so far is partial: Laya can reach Jev’s precision band, but it does not reach the majority ensemble’s recall.

Larger pool training did not transfer

The rubric axis also came out as a computed fail on the gold slices. Vanilla, at 149 tokens, beat enhanced, at 980 tokens, per subset on 5 of 6 comparisons, so the richer rubric added no value on Arm A. On the second client’s data, vanilla was again more precise, while enhanced traded about 2.5 points of precision for about 3 points of recall. Both were one seed per checkpoint, and the seed study is the yardstick for how far a gap like that is from being established.

The Arm A data started with 16,033 candidate rows. Normalization, duplicate dropping, conflict-key dropping, and a gold-exclusion step left 11,010 rows, one per distinct pair key, with 26.24% MATCH. Every row was human-reviewed, and none was gold. The stage arithmetic closes both ways, on rows and on keys.

The Arm B data started with the pipeline’s labeled pool: 613,509 rows, 41.27% positive, close to gold’s 41.3%. After excluding gold pairs and descriptions that leak into gold, and carving an entity-disjoint 20,000-row validation set, 348,106 training rows remained. The exclusions did not preserve the prevalence: leaked-description rows were mostly negative, so the final training split was 55.25% MATCH, or 192,328 of 348,106 rows. The pool labels also come from the same 2026-10-07 fit run that produced gold, whose 20,000 pairs were drawn from exactly this pool, so gold is in-sample for Arm B by construction. The arms differ in prevalence, source, scale, and label provenance at once, and nothing in the report can isolate which difference matters.

On the in-pool gold subset, pool checkpoints scored about 0.95 precision and 0.95 recall, far above Arm A’s 0.81 to 0.86 precision and 0.69 recall. On the client’s real output, their precision fell to about 0.76, and their recall collapsed to 0.34 to 0.37, 16 to 21 points below Arm A, with both recall intervals clear of Arm A’s. They also called MATCH on about a third fewer pairs, 6,000 to 7,000 against 9,500 to 10,500.

Training-data leakage is an unlikely explanation for the gap between the arms: less than 1% of this client’s pairs or descriptions appear in either arm’s training split. The gold score is in-sample for Arm B by design, so it measures fit to that distribution rather than transfer. On genuinely new data, the pool checkpoints fall back to NONMATCH. Because the arms differ in several ways at once, the result is descriptive rather than causal.

The campaign takeaway is narrow: the Arm B report had chosen pool_vanilla as the best operating point on gold, and this result does not support carrying that choice forward. When in-pool and out-of-sample disagree, the out-of-sample number is the one a production decision should lean on.

Fresh labels confirm the limits, not a rule win

The whole comparison was repeated on two more partitions of the same March 2026 link run, set1 and set2, whose 1,000-pair holdouts were labeled fresh and blind after every checkpoint was frozen. set1 holds 17,099 pairs, estimated 49.5% matches. set2 holds 17,686 pairs, estimated 42.4%. The pooled intervals are about ±0.04 to ±0.05.

A scope note matters for how readers should treat these fresh labels. They were a frozen holdout for the initial pre-registered rule test. The seed and epoch studies, the paired Laya-Jev comparison, and the choice of which checkpoint to retrain reused them afterward. For those later analyses, set1 and set2 are not independent confirmation. They become confirmatory only after a new holdout or an independent replication.

For teams evaluating self-hosted decision models, the practical signal is clear. Laya fine-tuning on reviewed pairs can make a local model competitive with a hosted model on precision at a fixed operating point. It does not automatically make the model a better routing decision, because the ensemble still recovers more true matches. It also does not make a six-rule retrain safe to ship. The next reliable step is not more rubric text alone. It is seed replication, training-set uncertainty separate from label uncertainty, and a genuinely new holdout before anyone claims that the rules landed.