Model Comparisons

Do multimodal models reach consistent decisions when presented with equivalent optical spectra?

Multimodal model answers on optical spectra swing with format, often fail abstention, and low-rank tuning may teach shortcuts rather than reasoning.

Key Takeaways

  • Multimodal models can change answers by up to 41 percentage points when the same optical spectrum is shown in equivalent formats, such as wavelength tables, photon-energy tables, or plots.
  • Many models do not reliably abstain when the evidence needed for an answer is missing, even when the correct response should be “undetermined.”
  • LoRA fine-tuning can teach a superficial shortcut: the model may abstain whenever input data are truncated, regardless of whether the missing part actually matters.
  • Training with answerable truncations helps models distinguish relevant missing evidence from irrelevant missing data and reduces false abstention behavior.
  • Scientific model evaluation should test representation invariance and missing-data behavior, not just average accuracy on complete inputs.

For anyone using multimodal models to read measured optical spectra, the key question is whether a spectrum representation is stable: if the same silicon metasurface data appears as a nanometer table, a micrometer table, scientific notation, an electron-volt table, or a photon-energy plot, the answer should not swing. A reported test on silicon metasurface spectra (MOCLIP) and gold nanoparticle spectra shows it can swing by up to 41 percentage points within one model when only the representation changes. The same test also checks whether models say so when the evidence needed for an answer is missing.

Do multimodal models reach consistent decisions when presented with equivalent optical spectra?

Why representation matters for spectrum answers

A spectrum can be shown to a model as a table in nanometers or as a plot in photon energy. The decision should not depend on which view the model sees. The evaluation tested representation invariance and abstention on measured optical spectra of silicon metasurfaces, labeled MOCLIP, and on gold nanoparticles, with labels computed from the data.

Each spectrum was presented in six equivalent forms: four tables using nm, μm, scientific notation, and eV, plus two plots using nm and eV. The test also included partial spectra where data were removed from inside or outside the queried range. The decision models evaluated were d3-nano and Jev 1.13, three GPT-6 Luna configurations, and the Qwen3.5-2B base model.

The reported results show that representation choice is not a cosmetic detail. Changing only the representation moved accuracy by up to 41 percentage points within one model, with the largest shifts mostly appearing on photon-energy tables and plots. With the same evidence, asking in photon energy instead of wavelength helped some model-view pairs by 16-50 points and hurt others by up to 25.

When evidence is missing, abstention is the answer

The second part of the test focused on whether models can recognize missing evidence. If the relevant extremum is removed from a spectrum, the correct behavior is not to guess. The model should answer that the result is “undetermined.”

The evaluation found that unadapted models rarely gave that abstention answer. This matters because scientific decisions often depend on whether a model knows the limits of its own evidence, not only on whether it can answer when the data are complete. A model that gives a confident answer from incomplete data can be worse for a user than a model that stops and says the requested range is missing.

Do multimodal models reach consistent decisions when presented with equivalent optical spectra?

LoRA fine-tuning can teach a shortcut

The study also examined LoRA fine-tuning of d3-nano. LoRA is a lightweight adaptation method that updates a smaller set of parameters compared with full model retraining.

The fine-tuned model improved comparisons on the trained table formats. But much of its gain on unanswerable questions came from a shortcut: it abstained whenever the input was truncated, regardless of whether the missing part actually mattered. That behavior can look like abstention skill while still failing when a truncated spectrum remains answerable.

To test this, the researchers trained with answerable truncations, meaning some partial spectra still contained enough evidence for a valid answer. This approach avoided the shortcut in all six seeds. On held-out structures, it produced a 90-100 point gap between abstention after inside cuts and abstention after outside cuts, showing that the model had learned to distinguish relevant missing data from irrelevant missing data.

A control condition used the same amount of extra original-task data but did not train on answerable truncations. In that case, five of six seeds kept the shortcut. On gold spectra, the two training sets performed similarly.

What this means for users of scientific models

The broader point is not that one model is simply better than another. The point is that multimodal language models used for scientific decisions should be tested on equivalent representations and on missing evidence, not only on average accuracy.

Do multimodal models reach consistent decisions when presented with equivalent optical spectra?

For researchers, this means a model that scores well on one table format may fail when the same spectrum is shown as a plot or in photon-energy units. For engineers building spectrum-analysis workflows, it means the interface should be tested across the representations users actually paste, upload, or query. For anyone relying on model answers, it means a missing-data check should be built into evaluation: can the model say “undetermined” when the queried range has been cut, and can it still answer when the cut removes irrelevant data?

The release includes the tasks, per-response results, and analysis code, so the findings can be inspected case by case rather than judged only from summary scores.

Meta description: Tests of silicon and gold optical spectra show model answers can shift by 41 points when representation changes, and missing data often triggers abstention.

Tags: spectrum representation, abstention, multimodal models, optical spectra, LoRA fine-tuning

Featured image: Abstract spectrum chart with smooth curves and grid lines, friendly illustration, no people, no logos, no text.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.