Tool Reviews

A Technical Report on Fish Audio Drama 3

Fish Audio Drama 3 is an open-source 35B MoE speech model for directed audio, multi-character scenes, revisions, edits, and streaming.

A Technical Report on Fish Audio Drama 3

Key Takeaways

  • Fish Audio Drama 3 is an open-source speech model that turns text into directed audio performances, including multi-character scenes, revisions, and streaming audio.
  • It combines new capabilities—multi-turn revision, scene dubbing, editing by infilling, and streaming text input—in one model rather than separate systems.
  • The model uses a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token, paired with a hybrid linear-attention design and a codebook multi-token-prediction audio decoder.
  • Training includes three-stage pretraining, instruction fine-tuning, and large-batch reinforcement learning using a single learned model of human preference as its only reward.
  • It achieves higher single-stream inference speed than Fish Audio S2—271 frames per second versus 177 on one H200—despite having far more parameters, with weights and tools available on GitHub, Hugging Face, and SGLang-Omni.

Meta description: Fish Audio Drama 3 is an open-source speech model that turns text into directed performances, multi-character scenes, revisions, and streaming audio.

A Technical Report on Fish Audio Drama 3

Tags: fish audio, speech generation, open-source, audio drama, AI audio

Featured image: abstract friendly illustration of text becoming a directed audio scene with waveform, microphones, and character silhouettes; no real people, no logos, no text.

Fish Audio Drama 3 is an open-source speech generation model for creators, developers, and researchers who want text to become directed audio performances. It can clone voices from multiple references, follow natural-language direction, revise a take when the next turn gives a note, voice multi-character scenes, edit existing speech by infilling, and synthesize long-form and streaming audio.

A Technical Report on Fish Audio Drama 3

New capabilities in one model

The model does not split these jobs into separate systems. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, and they are all in one model. That makes the release more than a text-to-speech upgrade: it changes how users can guide a performance over several turns.

For readers, the practical meaning is that a workflow can remain conversational. A user can ask for a different take after hearing a line, add another character, or request a change while generating longer scenes. Scene dubbing covers multi-character scenes, while editing by infilling lets the model change parts of existing speech.

Architecture and training

Drama 3 is built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. In this style of model, the total parameter count is large, but only a subset is used for each token. The system pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder. This decoder predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. Acoustic codebooks are compact representations of sound.

A Technical Report on Fish Audio Drama 3

Drama 3 is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning. The reinforcement learning step uses a single learned model of human preference as its only reward.

Speed, access, and next steps

The inference engine decodes a single stream at 271 frames per second on one H200, compared with 177 frames per second for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The comparison places speed against size: Drama 3 is larger, but the engine reaches a higher single-stream frame rate.

The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3). Inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). Readers who want to try directing performances can visit https://fish.audio.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.