vLLM is a large language model (LLM) acceleration framework that developed by the research team of University of California, Berkeley. It is used PagedAttention to increase the usage rate of GPU VRAM, and this method do not change the model architecture.
Read More »Using vLLM To Accelerate The Decoding Of Large Language Model
Last night I tried to improve some code about merge two models. Because my poor device, I cannot merge all layers totally, but need to merge them layer by layer for reducing my memory cost.
Read More »[PyTorch] Delete Model And Free Memory (GPU / CPU)
Today I tried to install Ubuntu on a new MSI notebook, but encountered an error when I need to choose the partition: system message tell me, the current device cannot found any disk.
Read More »[Solved][Linux] Cannot Found The Disk On MSI Notebook - Abnormal Of RAID (Intel RST)
Given the root of a binary tree, construct a string consisting of parenthesis and integers from a binary tree with the preorder traversal way, and return it.
You are given a string num, representing a large integer. Return the largest-valued odd integer (as a string) that is a non-empty substring of num, or an empty string "" if no odd integer exists.
Although I reach AI and research it all day, but there are too many new architectures, breakout, theories... many things I just know a rough concepts and put it away. The almost deeper understand need to until I must to be adjust the model architecture.
Read More »[Machine Learning] A LLM (Large Language Model) Visualization Website
The sticky bit is an interesting application in Unix and Unix-like systems. It represents a special permission setting that offers more nuanced operations for files and directories. Most are familiar with file permissions, categorized into three distinct types: rwx (read, write, execute). The sticky bit goes a step further by regulating the rights to "delete" and "rename" files.
Read More »[Linux] Explanation Of Sticky Bit (chmod 1777) In early 2023, The pytorch 2.0 version added a new function that is torch.compile(), It cloud be accelerate the speed of model training or inference. By using precision training with compiling, it always make my training time reduce to half.
Today I tried to load the model weights file after torch.compile(), and then I found the weights is stored by OrderedDict data structure. This data structure is an ordered sequence, in other word its content needs to follow the order.
