Welcome Beam is a 501 billion parameter open model you can own, customize, and scale.
Reflection's Beam is a 501 Billion Apache-2.0 open LLM with 1 Million token context for coding, agents, and GPU deployment.
Key Takeaways
- Beam is a 501B-parameter sparse Mixture-of-Experts model with 23B active parameters per token, designed for frontier-scale coding and agentic workloads.
- It supports a 1M-token context window using hybrid sliding-window and global attention, which reduces KV-cache growth to roughly one-sixth of a fully global design.
- The model is released under Apache 2.0 with day-0 support in transformers, vLLM, and SGLang, plus BF16, FP8, and NVFP4 checkpoints for different deployment scenarios.
- Practical serving requires substantial memory: the ~280 GB NVFP4 checkpoint is recommended for four B200 GPUs, while Hopper GPUs such as H200 should use FP8 checkpoints.
- Reflection is supporting adoption with an open-source Rust-based harness and $10,000 in Hub credits for standout community projects built on Beam.
Reflection has released Beam, a 501B-parameter sparse Mixture-of-Experts language model with 23B active parameters per token, aimed at teams that want frontier-scale coding and agentic performance on their own hardware or cloud. The model ships under Apache 2.0 with day-0 support in transformers, SGLang and vLLM, plus a dedicated open source Rust-based harness, and Reflection is backing adoption with $10,000 in Hub credits for the best community projects built on it.
What Beam actually is
Beam is a text-only model built for long-context reasoning and agentic work, with a 1M-token context window. It is sparse: each token is computed by a small fraction of the network, so it runs at the speed of a much smaller model while keeping the capacity of a large one. The model has 52 layers and a hidden size of 5,120, with 256 experts in each MoE layer and 8 routed per token.
Checkpoints are published in BF16, FP8 and NVFP4. The BF16 checkpoint is about 1 TB in safetensors format. NVFP4 stores most parameters in 4-bit precision for NVIDIA Blackwell GPUs while keeping some parameters in BF16 for accuracy, shrinking the footprint to roughly 280 GB; FP8 checkpoints are positioned for Hopper and Ampere chips. A speculative decoding drafter model using DFlash2 is also part of the release.
Architecture choices that matter for deployment
Most of Beam's layers use sliding-window attention over the previous 512 tokens: 43 sliding-window layers and 9 global layers, with every sixth layer and the last one attending to the full context. Sliding-window layers use RoPE; global layers use no positional encoding and instead apply a scalable-softmax query scaling that grows slowly with the number of visible tokens, keeping softmax precision as context lengthens.
Attention uses grouped-query attention with 48 query heads and 8 key-value heads, RMSNorm on queries and keys, and a learned sigmoid gate that scales every channel of every head before the output projection.
The first 3 layers use a dense SwiGLU MLP; the other 49 replace it with a Mixture-of-Experts block of 256 experts, each a SwiGLU MLP with an intermediate size of 2,560, or about 39M parameters. The experts hold 98.4% of the model's parameters. The router follows DeepSeek-V3's group-limited scheme: it scores all 256 experts with a sigmoid, splits them into 8 groups of 32, keeps the 4 best groups ranked by the sum of each group's top two scores, then picks the top 8 experts inside them. Beam uses the same group sizes as DeepSeek-V3 but no shared expert; expert outputs are combined with normalized router weights and scaled by 2.5.
The residual stream is the most distinctive part. Beam counts the input embedding as step 1, and at step k adds the sublayer output with weight 1/sqrt(k) while keeping the existing stream with weight sqrt((k-1)/k). The squares of the two weights sum to one, so the stream stays at a constant scale from the first layer to the last, and after all 104 additions every sublayer output and the input embedding carry the same weight in the final hidden state. Every sublayer is normalized before and after, and the final logits pass through a tanh softcap at ±30 computed in FP32.
The long-context math
Hybrid attention pays off in KV cache size. Each layer caches 2 x 8 KV heads x 128 dims = 2,048 values per token. If all 52 layers kept every token, the cache would grow by 208 KiB per token. Because only the 9 global layers keep the whole sequence and the 43 sliding-window layers hold a fixed 512-token window, the cache grows by 36 KiB per token — roughly a sixth of the naive rate. These figures are per sequence, in BF16, with the transformers dynamic cache.
Hardware: plan for four B200s, not two
The most practical way to serve Beam is the NVFP4 checkpoint, which retains most of the quality of the BF16 checkpoint. A B200 has 180 GB of HBM3e, so two of them (360 GB) can hold the weights with little room left for activations and the KV cache. Reflection recommends planning for four B200s (720 GB), which leaves room for long contexts and concurrent requests. Alternatives include a full 8xB200 node, two B300s at 288 GB each, or 8xRTX Pro 6000. GPUs older than Blackwell, such as the H200, cannot run NVFP4 natively and should use the FP8 checkpoints instead. Teams without Blackwell hardware can test the model through Inference Providers.
Running it: three supported paths
In transformers, the text-generation pipeline returns the whole chat, with the last message already parsed by the response template. At a lower level, `AutoModelForCausalLM` and `AutoTokenizer` work as usual, and the tokenizer carries a response template: `parse_response` turns raw output into a message dict with reasoning, answer and tool calls in separate fields, avoiding the usual regex splitting. Because Beam writes tool-call arguments inline in its own XML-style grammar, passing tool schemas to `parse_response` lets the parser cast each argument to its declared type.
For multi-device loading, Beam's config defines an expert-parallel plan: each expert goes on one device, while attention, dense layers, embeddings and the LM head are replicated on every device — 8B parameters, or 16 GB in BF16. Reflection's example launches across 16 H200 GPUs with torchrun, so each holds 78 GB of weights; with 8 GPUs each would hold 139 GB, leaving no room on a 141 GB H200. The device count must divide the number of experts, 256.
In vLLM, the BF16 checkpoint can be served with tensor parallelism across 16 GPUs (two 8x H200 nodes) behind an OpenAI-compatible API. Adding DFlash2 speculative decoding lets a small drafter propose up to 15 tokens ahead, which Beam checks in one forward pass, so each step can accept several tokens at the cost of a little extra memory for the drafter. SGLang support is also day-0, and Reflection recommends serving with DFlash2 there too: on Blackwell the NVFP4 checkpoint fits on 4 B300s, while on Hopper the FP8 checkpoint runs on 8 H200s with three flags changed — the checkpoint, the tensor-parallel size (8 instead of 4) and the attention backend (fa3 instead of trtllm_mha, which needs Blackwell).
Beam is also available on Hugging Face Inference Providers, powered by Baseten, Fireworks and Together, with more to follow, and there is a one-click Inference Endpoints recipe in FP8 or NVFP4 or via the hf CLI.
What Reflection tested, and what it admits
Reflection shared "vibe evals" of Beam-501B-A23B on five tasks, run once each at low and high reasoning effort through the Reflection platform. The company is explicit that these are vibe evals, not benchmarks, and positions them as a way to see where the model wins and loses on specific use cases. The complete inputs, outputs, API usage, task fixtures, graders and conversation traces are in a public Hugging Face bucket.
Two caveats are worth noting from the results. At low reasoning effort, the long-context response got all code behavior and all six source citations correct, but the response did not match the parser's format — which Reflection attributes partly to prompt ambiguity. On another task, low returned the correct answer but omitted the requested justification, while high supplied a correct roots-of-unity proof that was independently reviewed.
Fine-tuning: LoRA SFT on a single node
For teams adapting Beam to their own data, Reflection shares examples in TRL and Miles. The quickest route is supervised fine-tuning with TRL's SFTTrainer using LoRA adapters via PEFT rather than updating all 501B parameters. Reflection's example trains Beam to keep its reasoning in the language requested by the user, fine-tuning on Multilingual-Thinking with LoRA — a small set of adapter weights while the base model stays frozen.
The setup combines model sharding at load time, expert parallelism, FlashAttention 4 and sequence packing. LoRA SFT fits on a single node with 8 B300 GPUs, reaching around 19,000 tokens per second; the corresponding H100 setup uses four nodes. Full-parameter reinforcement learning is supported through Miles, and Reflection worked with the Reflection team's recipes for inference, SFT and RL.
The $10,000 build challenge
Reflection is giving out $10,000 in Hub credits to the best projects that use Beam. The entry bar is deliberately low: build something and share it on the Hub. Suggested directions include a demo that runs Beam on Spaces, a quantization for your favorite hardware, a synthetic dataset for a problem you care about, an RL environment built with or for Beam, or benchmark results. Entries are collected from links shared on the model's repo page, and credits go to the winners' Hub accounts.
What to watch
Beam's pitch is narrower than a general-purpose frontier model: coding and agentic workloads, long context, and deployment on hardware you control. The open questions are how the vibe-eval picture holds up under independent benchmarking, whether the NVFP4 path delivers on its quality claims in production, and how quickly the Inference Providers lineup expands beyond Baseten, Fireworks and Together. For teams already running H200 or Blackwell capacity, the more immediate question is whether a 501B model at 23B active parameters changes the economics of self-hosting compared with API access — and the published recipes make that test relatively cheap to run.

Comments (0)