Last Updated on 2025-07-01 by Clay Introduction Speculative Decoding and KV Cache are both acceleration techniques applicable to Transformer models. The former uses a faster draft model to speculatively generate several subsequent tokens, which are then validated in a batch by the target model to reduce the cost of autoregressive decoding. The latter leverages the … Continue reading Implementation Notes on Integrating Speculative Decoding with KV Cache