Less Size, Stronger Recall: How Qwen2.5-1.5B Beats the 3B Model (With Four Hypotheses)

Meta description: Qwen2.5 1.5B beat 3B on SRRM long-context memory, with 86.7% retrieval and 45.7% reconstruction, according to new independent research on Academia.edu.

Tags: SLMs, long-context memory, Qwen2.5, model evaluation, edge AI

Featured image: abstract friendly illustration of two small model blocks holding overlapping cards in a long context window, no real people, no logos, no text.

This article is for developers, researchers, and AI engineers choosing a Small Language Model for local devices. It explains a new long-context memory benchmark, SRRM, and why it found that Qwen2.5–1.5B outperformed Qwen2.5–3B on both targeted retrieval and full context reconstruction. The paper, titled “SRRM: A Retrieval-Reconstruction Benchmark for Evaluating Long-Context Memory in Small Language Models,” was published on Academia.edu by Christopher Alberto Hamonangan.

Many large language models and small language models advertise context windows reaching 128K tokens or more. Because SLMs are increasingly deployed on consumer-grade hardware such as laptops, smartphones, and edge devices, teams often assume that a larger parameter count automatically means better memory. The SRRM results challenge that assumption under controlled tests.

A benchmark that checks recall and reconstruction

Traditional long-context evaluations often rely on Targeted Retrieval tasks, such as Needle-in-a-Haystack or single-fact question answering. These tests measure whether a model can find an isolated piece of data when explicitly prompted. They do not answer a harder question: can the model preserve and rebuild the entire set of facts it was shown?

SRRM was designed as a lightweight, reproducible, open-source benchmark to test long-context memory from two angles. The Sequential Retrieval Test asks explicit, isolated questions about previously presented statements, such as “What is John’s WiFi password?” The Summary Reconstruction Test asks the model to reconstruct all previously mentioned facts line by line without item-specific cues.

The benchmark also reports a Memory Preservation Ratio, defined as Mean Critical Information Recall divided by Mean Retrieval. In plain terms, it asks how much of what a model can retrieve survives when the model is asked to rebuild the whole context.

Why the smaller Qwen model led the metrics

The researchers evaluated three members of the Qwen2.5 family: Qwen2.5–0.5B, Qwen2.5–1.5B, and Qwen2.5–3B. They tested the models under identical experimental conditions and hardware configurations across context loads containing 10, 25, 50, 75, and 100 factual statements. All models received identical prompt templates, dialogue context sequences, and inference configurations, so model scale was the main variable being compared.

To test memory across different kinds of knowledge, the synthetic dataset used four semantic categories: Hair Color, Family Relationship, Activity, and WiFi Password. The examples included “James’s hair color is black,” “Sarah’s younger sister is Emily,” “Mary went to Tokyo,” “Kevin ate pizza,” and “John’s WiFi password is OceanBlue92.” The WiFi Password category is intentionally harder because arbitrary alphanumeric strings carry little semantic context.

The statements were merged through a deterministic Round-Robin interleaving strategy. That process distributed the four categories evenly throughout the benchmark sequence and assigned each item a sequence identifier and a category identifier, producing a standardized dataset that could be partitioned into different context sizes.

Conventional scaling expectations would suggest that the 3B model should dominate memory tasks. The reported results point the other way. Qwen2.5–1.5B achieved 86.7% mean retrieval accuracy, while Qwen2.5–3B scored 72.2% and Qwen2.5–0.5B scored 65.1%.

The reconstruction gap is the more striking part. Qwen2.5–3B’s Critical Information Recall dropped to 18.5%, while Qwen2.5–1.5B retained 45.7%. The 3B model also showed a 74.3% relative gap between retrieval and reconstruction, producing an MPR of 0.26. The 1.5B model preserved over half of its retrieved knowledge, with an MPR of 0.53. Across all evaluated models, retrieval performance was substantially higher than reconstruction performance.

Four hypotheses for the reversal

Because the models were tested with identical prompts, dataset ordering, context scaling, and inference configurations, the authors argue that the reversal is unlikely to be caused by measurement noise or prompt-engineering artifacts. They offer four hypotheses.

First, Context Utilization Efficiency suggests that the 1.5B architecture may have a more balanced attention mechanism for mid-to-long context loads, keeping relevant factual tokens accessible across layers during both retrieval and generation.

Second, Memory Representation Strategy proposes that internal representations may differ across model scales. The 1.5B model might maintain more discrete factual representation vectors, while larger models shift toward denser contextual abstractions.

Third, Information Over-Compression is presented as the leading hypothesis. The larger 3B model may compress multi-turn dialogue contexts aggressively. That compression may be enough to answer targeted single-item prompts, but it may destroy the fine-grained factual boundaries needed to list every statement during unprompted reconstruction.

Fourth, Scaling-Related Internal Mechanisms suggests that long-context memory preservation may not scale monotonically with parameter count alone. Optimization dynamics, pre-training data mixtures, or layer-specific attention head allocations may create favorable sweet spots in specific model variants.

What developers should do next

The practical takeaway is that high accuracy on retrieval-style benchmarks does not guarantee coherent multi-turn assistant dialogues or reliable long-document summarization. Single-fact retrieval metrics can look strong while a model still fails to preserve a full set of facts.

For teams running models locally, the paper suggests Qwen2.5–1.5B may be a better memory-focused choice than Qwen2.5–3B under these test conditions. In edge computing, local retrieval-augmented generation pipelines, and persistent conversational agents, upgrading from 1.5B to 3B does not guarantee better memory. It may even degrade performance when the task requires recalling many facts from a long context.

Readers should still treat this as a research signal rather than a production guarantee. The paper is available on Academia.edu, and the author notes that the preprint is on hold on arXiv and HAL pending author identity and institutional verification, with formal peer review to follow. The test set is synthetic and focused on four factual statement types, so it may not capture all long-document behavior, reasoning, or instruction-following. Still, it gives developers a clear prompt to test reconstruction, not just retrieval.

The paper also asks whether similar scaling anomalies appear in other model families, such as Llama, Mistral, and Gemma.

Who should use it / who should skip it

Who should use it: teams choosing SLMs for local or edge deployments, researchers evaluating long-context memory, and application developers building persistent chat assistants or local retrieval-augmented generation pipelines. It is useful when a model must recall many details from a long conversation, not just answer one lookup question.

Who should skip it: readers looking for a general ranking of all model capabilities, or teams whose work is mainly short-context generation, coding, or single-question retrieval. Because the benchmark is synthetic and the paper is not yet peer reviewed, the numbers should not be used as the sole basis for replacing a production model.