Tool Reviews

Equal Character Error Rates Can Still Mean Unequal Damage, So a Severity-Weighted Character Error Rate Metric Is Needed for Optical Character Recognition.

semCER is a severity-weighted extension of character error rate for evaluating text recognition outputs. Instead of charging every character error equally, it scores each edit from 0 to 1, distinguishing harmless typos from mistakes that change meaning. It keeps the same alignment and denominator as character error rate, so it explains rather than replaces the original metric. Results suggest edit-level severity tracks human ratings in English, Vietnamese, and Chinese, while whole-line normalized character error rate still summarizes overall error volume. Across 16 languages, severity varies by writing system, showing multilingual evaluation cannot rely on universal assumptions. The project releases code, a local 0.8 billion-parameter model, and 61 million judged edits, with clear limits on reader studies and small score gaps.

Equal Character Error Rates Can Still Mean Unequal Damage, So a Severity-Weighted Character Error Rate Metric Is Needed for Optical Character Recognition.

Key Takeaways

  • semCER adds a 0-to-1 severity score to each character error while keeping character error rate’s alignment and denominator, helping explain which OCR mistakes matter to readers rather than just counting all errors equally.
  • Two systems with the same character error rate can still cause different damage; severity breakdowns can show whether errors are mostly harmless or meaning-changing, making semCER a diagnostic complement to character error rate.
  • Severity is not universal across languages or writing systems; for example, a wrong diacritic is minor in Spanish but can be nearly as costly as a wrong letter in Telugu, so multilingual evaluation must account for reader impact.
  • The project releases open resources, including an LLM-distilled 0.8B local model, 61 million judged edits, reader ratings, code, and a quickstart notebook.
  • Current limits are important: reader validation covers only English, Vietnamese, and Chinese, and severity differences below about 0.1 should not be treated as meaningful.

Meta description: semCER scores each character error from 0 to 1 to show which text mistakes matter, while character error rate remains a useful overall measure for human readers.

Equal Character Error Rates Can Still Mean Unequal Damage, So a Severity-Weighted Character Error Rate Metric Is Needed

Tags: character error rate, text evaluation, multilingual evaluation, open datasets, model evaluation

![Abstract friendly illustration of text lines with severity markers, no logos or text](semcer-severity-illustration.jpg)

The author of semCER has released a metric that scores each character edit from 0 to 1 instead of treating every character error as equally costly. It is aimed at people who use character error rate to evaluate text quality and want to know which mistakes human readers notice. semCER keeps the same alignment and denominator as character error rate, but adds a severity score for each edit.

Why severity matters

Character error rate charges every character error 1. A typo costs the same as a wrong digit in a price or a wrong name. That can make a simple error count look more important than it is, because not every mistake changes what a reader understands. semCER addresses this by assigning each edit a severity from 0, for a harmless change, to 1, for a change that loses meaning.

Equal Character Error Rates Can Still Mean Unequal Damage, So a Severity-Weighted Character Error Rate Metric Is Needed

The practical value is diagnostic. If two systems have the same number of character errors, a severity breakdown can show whether the errors are mostly harmless or mostly damaging. The author does not present semCER as a way to replace the original metric, but as a way to explain it.

What the results show

According to the author, edit-by-edit severities track human readers' ratings in English, Vietnamese and Chinese. For whole lines, normalized character error rate tracks readers as well as semCER or better. The author therefore describes semCER as a breakdown that sits beside character error rate, not a replacement. It tells evaluators which errors hurt, while the original metric still summarizes overall error volume.

Across 16 languages, the same error type receives very different severities. A wrong diacritic costs almost nothing in Spanish but nearly the cost of a wrong letter in Telugu. That result matters for multilingual text evaluation because severity is not universal across writing systems. A score tuned to one language may not describe reader impact in another language.

Equal Character Error Rates Can Still Mean Unequal Damage, So a Severity-Weighted Character Error Rate Metric Is Needed

Model, data, and limits

The severity labels come from an LLM judge using DeepSeek V4 Flash. The author says these judgments were distilled into a 0.8B model that runs locally. The open release includes 61 million judged edits and reader ratings.

The project has clear limits. Reader studies cover only English, Vietnamese and Chinese. The author also warns that severity differences below about 0.1 are not reliable. That means small gaps in scores should not be read as meaningful differences. The metric is strongest when used to understand patterns of error severity, not to claim exact reader harm from tiny score changes.

Open resources and next signals

Everything is open. The code can be installed with pip install semcer and is hosted at https://github.com/nguyenx2t/semcer. The local model is available at https://huggingface.co/nguyenx2t/semcer-qwen3.5-0.8b. The dataset of judged edits and reader ratings is at https://huggingface.co/datasets/nguyenx2t/semcer-judged-edits. A quickstart notebook is at https://colab.research.google.com/github/nguyenx2t/semcer/blob/main/notebooks/quickstart.ipynb.

The author welcomes feedback and bug reports, especially from people working on languages that have not been checked against reader ratings. The key question now is whether semCER helps evaluators prioritize fixes without discarding the simple whole-line comparison that character error rate already provides. For readers following the release, the useful signal is not a new replacement score, but a clearer map of which errors actually change meaning.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.