Last Updated on 2025-07-01 by Clay Currently, most of the time spent during LLM inference is bottlenecked by the need to generate tokens sequentially. This highlights a limitation imposed by GPU memory bandwidth — for every single token decoded, the model’s entire weight must be loaded, even though the actual floating-point computation is minimal. This … Continue reading [Paper Reading] Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding