An explanation of speculative decoding shows how any LLM can achieve three times faster inference.
Speculative decoding speeds up LLM inference by drafting and verifying tokens in parallel for 2 to 3 times faster output while preserving quality.
Speculative Decoding
There Are Three Ways to Enhance Mixture-of-Experts Inference on NVIDIA DGX Spark
Three optimizations speed Ling Mixture of Experts inference on DGX Spark by 42.5%, cut latency, and keep accuracy if wired correctly.