Youtu-Parsing-Omni: a single encoder and a single schema for all modalities
Youtu-Parsing-Omni is a compact 5B omni-modal parsing model that converts documents, natural images, charts, flowcharts, geometry figures, audio and video into a single structured JSON envelope. A task prompt selects one of seven parsing families, while the unified schema captures layout, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events and camera motion alongside captions and reports. It reports 96.96 Overall on OmniDocBench v1.6 and 75.08 Avg. on OmniParsingBench, offering a way to consolidate multi-pipeline document and media analysis.
Key Points
- Youtu-Parsing-Omni is a compact 5B omni-modal parsing model that converts documents, images, charts, audio, and video into one structured JSON output.
- It uses a unified schema for seven parsing tasks, combining perception details such as layout, OCR, ASR, bounding boxes, and timestamps with cognition outputs like captions, narratives, and reports.
- An omni encoder handles visual, audio, and interleaved audio-visual inputs in a single model, reducing the need for separate document, image, audio, and video pipelines.
- The model reports strong benchmark results, including state-of-the-art performance on OmniDocBench v1.6 and the best open-weight result on OmniParsingBench, while remaining competitive in specialized domains such as chemical structures and music scores.
- Practical value lies in simplifying multimodal parsing workflows, though availability, license, training data, hardware requirements, and latency are not specified and should be checked before production use.
For developers and researchers building document, media and video analysis pipelines, Youtu-Parsing-Omni is a compact 5B omni-modal parsing model that converts one input into one structured JSON envelope. The input can be a document page, a natural image, a chart, a flowchart, a geometry figure, an audio clip or an audio-visual video. The returned JSON is meant to cover both perception and cognition: it records what is visible or audible, and it also describes what the model understands from those signals.
One Input, One Machine-Readable Response
The model is built around a single output format. JSON is a machine-readable data format, so the goal is to give downstream software a predictable structure rather than a loose block of text.
The perception layer tracks layout elements, text, tables, formulas, bounding boxes (bbox), timestamps, automatic speech recognition (ASR), optical character recognition (OCR), acoustic events and camera motion. The cognition layer adds captions, narratives and reports.
A task prompt tells Youtu-Parsing-Omni which parsing family to use. That unified schema is the core idea: seven parsing families can share one JSON envelope instead of requiring separate output contracts for documents, images, charts, flowcharts, geometry, audio and video.
What Each Input Task Captures
| Input | Task | Modality / subtype | Key contents |
|---|---|---|---|
| Document page | document | image / document | layout elements with bbox; text, LaTeX, OTSL tables, Markdown charts, Mermaid flowcharts and reading order |
| Natural image | natural_image | image / natural_image | entities and text with bbox, tags, captions and a global description |
| Chart | graphics_chart | image / document | one chart element: Markdown table, notes and caption |
| Flowchart | graphics_flowchart | image / document | one flowchart element: Mermaid and caption |
| Geometry figure | graphics_geometric | image / document | one geometric element: points, lines, arcs, shapes, geometric relations and measurements |
| Audio | audio | audio / - | vocal / non-vocal segments with timestamps, speakers, ASR, timbre / scene captions and acoustic events |
| Natural video | natural_video | video / natural_video | temporal segments with visual elements, actions, interactions, camera motion and audio track |
| Text-rich video | textrich_video | video / text_rich_video | segments with OCR and ASR, plus a Markdown structured_report of the whole video |
Omni Encoder and Serving Details
The model uses an omni encoder for image, audio and interleaved audio-visual video inputs. In the video case, frames and the audio track can be handled inside a single model rather than through separate visual and audio pipelines.
For serving, the material says the package includes a vLLM plugin, pinned serving settings, task prompts and inference examples. Pinned settings can help keep serving behavior consistent across deployments, while the included prompts and examples are intended to make the model easier to run without custom integration work.
Benchmark Signal and Practical Meaning
The reported results position Youtu-Parsing-Omni as a strong small-model option. It is described as state-of-the-art on OmniDocBench v1.6 with 96.96 Overall. It is also described as the best open-weight model on OmniParsingBench with 75.08 Avg., second only to Gemini-3-Pro. In narrower domains, the material says it is competitive with specialized models on chemical-structure recognition in ChemOCR and music-score recognition in PDMX-Synth.
The practical value is mainly about consolidation. Teams that currently stitch together document parsing, OCR, ASR, chart extraction, flowchart conversion, geometry analysis and video reporting may benefit from a single prompt-driven model that returns one structured output. For readers evaluating it, the key question is whether a compact 5B model can reduce pipeline complexity without giving up too much accuracy on specialized tasks.
The provided material does not specify availability date, license terms, full training data, hardware requirements or latency, so those details should be checked before production use.
Meta description: Youtu-Parsing-Omni, a compact 5B omni-modal parsing model, turns documents, images, audio and video into one structured JSON output for seven parsing tasks.
Tags: Youtu-Parsing-Omni, omni-modal parsing, JSON extraction, multimodal AI
Featured image: Abstract illustration of a document, chart, audio wave and video frame merging into one JSON block, friendly style, no people, no logos, no text.

Comments (0)