In auto-regressive model decoding, if we need to decode K tokens, we must go through the process K times, which is the current bottleneck in the inference time of large language models.
Read More »[Paper Reading] Fast Inference from Transformers via Speculative Decoding
During the decoding process of large language models, especially in Auto-regressive models, decoding must be performed step-by-step until the entire sequence is generated. Within this process, there are caching techniques that can help reduce computation and improve decoding speed; one such technique is known as the KV Cache.
Read More »KV Cache: A Caching Mechanism To Accelerate Transformer Generation
When implementing various services through LLMs, do you worry about uncontrolled language generation? Recently, at a critical juncture in wrapping up a project, I used tools like Outlines to constrain LLM decoding, which effectively controlled the model's output to follow the desired patterns. However, a colleague posed a deep question: What if I want it not to generate specific words?
Read More »Using Finite State Machine (FSM) and Rollback Mechanism to Restrict LLM from Generating Banned Words
I've always used rough formulas to estimate the relationship between the scale of my models and the GPU VRAM consumption; after all, there are too many variables involved—model architecture, number of layers, attention mechanism implementation, sequence length, batch size, data precision used in training or inference... all of these affect our final calculation results.
Read More »Note on Calculating VRAM Consumption for Training and Inference of AI Models
I've been intermittently reading about a fine-tuning method called Kahneman-Tversky Optimization (KTO) from various sources like HuggingFace's official documents and other online materials. It's similar to DPO as a way to align models with human values, but KTO's data preparation format is much more convenient, so I'm quickly applying it to my current tasks before making time to study the detailed content in the related papers.
Read More »Note Of KTOTrainer (Kahneman-Tversky Optimization Trainer)
The following are some points in this paper:
Read More »[Paper Reading] ENTP: ENCODER-ONLY NEXT TOKEN PREDICTION
In machine learning, we often encounter the term KL Divergence (also known as Kullback-Leibler Divergence). KL Divergence is a metric used to evaluate the difference between two probability distributions P and Q.
Read More »[Machine Learning] Note Of Kullback-Leibler Divergence
A multi-modal large language model (Multi-Modal Large Language Model) isn’t limited to text only. I know this might sound contradictory, but this is a term that has become widely accepted. What I want to document today is how to fine-tune a multi-modal model using a script.
Read More »Notes on Fine-Tuning a Multi-Modal Large Language Model Using SFTTrainer (Taking LLaVa-1.5 as an Example)
The first time I heard about Vector Quantization (VQ) was from a friend who was working on audio processing, which gave me a vague understanding that VQ is a technique used for data feature compression and representation. At that time, I still wasn't clear on how it differed from dimensionality reduction techniques like PCA.
Read More »[Machine Learning] Vector Quantization (VQ) Notes
In the process of training and fine-tuning deep neural networks, the most important and scarce resource is undoubtedly the GPU's VRAM. Therefore, making every bit perform at its best is a critical task.
Read More »Differences in Precision Representations in Deep Learning: Float32, Float16, Float8, and BFloat16
