TenetBench is a Russian corpus anchored to specific clauses for evaluating language models against professional codes of ethics.
TenetBench offers a Russian benchmark, Zenodo data, and verdicts to inspect LLM conflicts with professional ethics clauses.
Key Points
- TenetBench evaluates whether LLM-generated text conflicts with specific clauses of Russian-language professional codes of ethics.
- The benchmark compares conflicting requests with benign alternatives built from the same scenario to isolate whether models produce clause-violating outputs.
- In the main evaluation, 11 of 14 model and reasoning-mode configurations generated conflicting content on most of 279 requests, usually without warnings, as judged by a frozen LLM evaluator.
- The public Zenodo release includes 2,235 task pairs across 10 professional codes, 7,303 model-judge verdicts, human annotations for 1,260 items, and the generation/evaluation pipeline.
- The dataset and code allow researchers and evaluators to independently inspect how LLM outputs relate to professional-code conflicts.
Meta description: TenetBench tests whether LLM-generated text conflicts with Russian-language professional codes. A preprint reports 11 of 14 configurations produced conflicts.
Tags: TenetBench, LLM evaluation, professional codes, Zenodo dataset, model reasoning
Featured image: Abstract friendly illustration of two answer cards compared by a fixed evaluator, with no real people, no logos, and no text.
TenetBench tests LLM output against professional codes
A preprint introduces TenetBench, giving researchers and evaluators a benchmark that tests whether LLM-generated text conflicts with specific clauses of Russian-language professional codes. The setup pairs each conflicting request with a benign alternative based on the same scenario. In the paper's main evaluation, each configuration was tested on 279 conflicting requests. 11 of 14 model and reasoning-mode configurations produced conflicting content on most requests, usually without a warning, as assessed by a frozen LLM judge.
How conflicting and benign requests are compared
The benchmark isolates one comparison: a request that can lead to clause-conflicting output and a benign alternative for the same scenario. This design lets the evaluation focus on whether a model produces conflicting text when given a conflicting request, compared with the benign alternative. The reported main result covers 14 configurations, each facing 279 conflicting requests.
Dataset and verdicts available on Zenodo
The release is available on Zenodo. It contains 2,235 task pairs covering 10 professional codes, 7,303 model-judge verdicts across 20 model and reasoning-mode configurations, and human annotations on 1,260 request-response items. The generation and evaluation pipeline is also provided. Data: CC BY 4.0. Code: Apache-2.0. DOI: https://doi.org/10.5281/zenodo.22678484.
What readers can inspect
The package gives researchers and evaluators a public way to inspect whether LLM output can conflict with Russian-language professional code clauses. It combines task pairs, model-judge verdicts, and a human-annotated subset, so the 11-of-14 main-evaluation result can be checked against the underlying prompts and scoring outputs.
Comments (0)