Last Updated on 2024-11-01 by Clay During the decoding process of large language models, especially in Auto-regressive models, decoding must be performed step-by-step until the entire sequence is generated. Within this process, there are caching techniques that can help reduce computation and improve decoding speed; one such technique is known as the KV Cache. This … Continue reading KV Cache: A Caching Mechanism To Accelerate Transformer Generation