Kangaroo is an implementation of Self-Speculative Decoding that introduces a trainable adapter layer. Over the past few weeks, I have been working on fine-tuning its adapter layer and have achieved some preliminary results, which I am documenting here.
Read More »Kangaroo: Inference Acceleration Architecture Implementation
Recently, while implementing the paper Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting, I encountered a question about its use of Cross Entropy Loss to align the probability distributions of the draft model and the target model. Why not use KL Divergence instead?
Read More »Differences and Comparison Between KL Divergence and Cross Entropy
Recently, I have been reviewing notes on distributed systems to reflect on the systems I built over the past year and examine potential areas for improvement. Someone recommended studying the CAP theorem, and after reading it, I found it quite intuitive, so I decided to document it here.
Read More »A Note Of Brewer's/CAP Theorem
Many of the inference acceleration techniques I have studied, such as Speculative Decoding, predominantly use a threshold for the confidence scores of the draft model. This threshold determines how many draft tokens should be decoded before passing them to the target model for verification, thereby reducing the extra computational cost when the draft model operates with low confidence.
Read More »Using The Target Model's Confidence Threshold To Decide Whether To Enable Speculative Decoding
Recently, I attempted to implement various speculative decoding acceleration methods. HuggingFace's transformers library also provides a corresponding acceleration feature called assistant_model. Today, let me take this opportunity to document it.
Over the past week, I dedicated some time to reproducing the Self-Speculative Decoding mechanism based on the ideas from the paper Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding, implementing the following modules:
In self-speculative decoding, since our draft model is derived from part of the target model’s network, finding an optimal 'Layer Skip Strategy' is crucial. We need to skip enough layers to achieve meaningful speedup while ensuring the draft model’s speculative decoding is good enough to avoid frequent rejection by the target model.
Today’s implementation focuses on optimizing my previously implemented LayerSkip model using the Bayesian optimization framework Optuna, to determine which layers to skip.
Read More »Optimizing LayerSkip Models with Bayesian Search for an Effective Layer Skipping Strategy
Self-Speculative Decoding is a variant of Speculative Decoding. The original Speculative Decoding method uses a draft model to optimize the inference of the target model. The draft model, which is typically distilled from the target model, offers similar output quality but with several times faster inference speed.
Read More »Self-Speculative Decoding Implementation: LayerSkip Transformer
Recently, I've been trying to organize the papers on accelerated reasoning techniques I've read over the past year into notes. During this process, I came across Bayesian optimization techniques that utilize Bayes' theorem, so I decided to write a note to record the essence of Bayes' theorem.
In simple terms, Bayes' theorem is a frequently encountered theorem in probability theory that describes the probability of a random event occurring under specific conditions.
Read More »A Note of Bayes' Theorem
