Cross-layer latent vector quantization enables extreme compression of embeddings from each layer.
OPTIC compresses Gemma 4 embeddings about 100 times with shared codes, retaining 97.8% bfloat16 performance and reducing GPU memory 29.7%.
Key Takeaways
- OPTIC compresses Gemma 4 per-layer embeddings by about 100× while retaining 97.8% of BF16 performance across benchmarks.
- It assigns each token a single shared quantized code across layers, with layer-specific affine decoders reconstructing the needed embeddings.
- The method optimizes compression against downstream model effects, such as after the gate and output projection, rather than only reconstruction error.
- Serving BF16 E4B from compact codes reduces GPU memory by 29.7%, with a small latency increase of at most 0.40% on an RTX 4090.
- OPTIC can also work together with backbone weight quantization, making large embedding tables more practical for memory-limited deployment.
Meta description: OPTIC compresses Gemma 4 per-layer embeddings about 100×, retaining 97.8% BF16 performance and cutting GPU memory 29.7% with little added latency cost.
Tags: Gemma 4, embedding compression, OPTIC, language model serving, BF16
Featured image: abstract friendly illustration of compact shared codes expanding into layered embedding vectors, no real people, no logos, no text
For readers tracking language model scaling, a new method called OPTIC offers a practical way to handle embedding compression in Gemma 4. It targets per-layer embeddings (PLE), which account for 35.2% of E4B's parameters, and shows that extreme compression can preserve most measured performance while making serving more memory-friendly.
Why embedding compression matters
Recent work points to a useful scaling direction: increasing embedding capacity can improve language models. The catch is deployment memory. As more capacity moves into embedding tables, the model becomes harder to serve in constrained environments. That makes extreme compression increasingly important, because the goal is to keep the benefits of larger embeddings while fitting the model into limited memory.
In Gemma 4, the PLE component is a major part of the model. At 100× compression, only about five bytes remain for each token-layer vector. If every vector is encoded independently, each one receives a tiny code, but that approach misses a key opportunity: different layers may be able to share storage rather than repeat similar information.
How OPTIC shares one code across layers
OPTIC changes that pattern by giving each token a single quantized code that is shared across all layers. Then, layer-specific affine decoders reconstruct the different PLE vectors from that shared code. In other words, the method stores one compact representation per token and lets each layer recover the version it needs.
The approach also adjusts the optimization target to match how the model actually uses embeddings. Instead of compressing only for reconstruction error, OPTIC fits the representation to its effect after the gate and output projection. This makes the compression objective closer to real model behavior.
What the 100× compression keeps
At about 100× PLE compression, OPTIC retains 97.8% of BF16 E4B performance across seven benchmarks. That is a large gap compared with product quantization, which retains 66.3% under the same style of evaluation.
The serving results are also practical. Serving BF16 E4B directly from the compact codes reduces GPU memory by 29.7%. The time cost is small: on an RTX 4090 with the same 1 GiB key-value cache, wall time increases by at most 0.40%.
OPTIC is not limited to embedding tables alone. It also works alongside backbone weight quantization, meaning the compression can be combined with quantization of the rest of the model weights.
What readers can check next
The release includes a paper, a code and method walkthrough, and a released 100× PLE artifact. For readers, the important takeaway is that shared-code compression can make large embedding tables more usable in memory-limited serving setups without giving up most benchmark performance.

Comments (0)