Machine Learning

Using AutoModel.from_pretrained() In Transformers To Load Customized Model Architecture

Clay
2024-08-222024-08-22
Machine Learning

To this day, many AI applications and open-source projects are developed based on the HuggingFace transformers package. A large number of models and packages are written to be compatible with the transformers format, and even share the same functions and methods, which makes them more widely accepted.

Under this premise, I came across an open-source training framework that conveniently wraps the automatic reading of Transformer architectures. However, one unavoidable problem is I want to use my custom model for experiments. I tried several solutions, hoping that when using AutoModel.from_pretrained(), by simply providing the local path to my model, I could successfully use my custom model architecture. This article records the method that worked.

[Solved] RuntimeError: view size is not compatible with input tensor’s size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(…) instead.

Clay
2024-08-212024-08-21
Machine Learning, PyTorch

Problem Description

When building deep learning models in PyTorch, adjusting the shapes of layers and input/output dimensions is something every AI engineer has to deal with. However, there is a small but interesting pitfall in the view() method of PyTorch:

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

[Machine Learning] Note of LayerNorm

Clay
2024-08-192024-08-19
Machine Learning, PyTorch

The working principle of LayerNorm is as follows:

Calculate mean and variance

$mean = \mu =\frac{\sum_{i=1}^{N}x_i}{N} \newline variance = \sigma^2 = \frac{\sum_{i=1}^{N}(x_{i}-\mu)^2}{N}$ Read More »

[Machine Learning] Note of Cross Entropy Loss

Clay
2024-08-182024-08-18
Machine Learning, PyTorch

Introduction to Cross Entropy

Cross entropy is a very common loss function in Machine Learning, as it is able to quantify the difference between a model’s classification predictions and the actual class labels, particularly in ‘classification tasks’.

[Machine Learning] Note of Activation Function GELU

Clay
2024-08-182024-08-18
Machine Learning, PyTorch

Gaussian Error Linear Unit (GELU) is an activation function used in machine learning. While it resembles the classic ReLU (Rectified Linear Unit), there are some key differences.

[Machine Learning] Note of RMSNorm

Clay
2024-08-172024-08-17
Machine Learning, PyTorch

Introduction to RMSNorm

RMSNorm is an improvement over LayerNorm, often used in the Transformer self-attention mechanism. It aims to mitigate the issues of vanishing and exploding gradients, helping the model converge faster and improve performance.

[Machine Learning] Note of Rotary Position Embedding (RoPE)

Clay
2024-08-162024-08-17
Machine Learning, PyTorch

Introduction

(Note: Since this article is imported from my personal Hackmd, some symbols and formatting might not display properly in WordPress. I appreciate your understanding, sorry for any inconvenience.)

RoPE is a method for introducing relative position information into the self-attention mechanism through absolute positional encoding.

[PyTorch] Using SDPA in 2.0+ to Improve the Computation Speed of Transformer’s Self-Attention Mechanism

Clay
2024-08-152024-08-16
Machine Learning, PyTorch

SDPA Introduction

Scaled Dot-Product Attention (SDPA) might immediately pop into the minds of those familiar with the Transformer self-attention mechanism:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$

[Paper Reading] Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Clay
2024-08-132024-08-19
AI, Machine Learning

Cross-lingual Modular (X-Mod) is an interesting language model architecture that modularizes the parameters for different languages as Module Units, allowing the model to use separate parameters when fine-tuning for a new language, thereby (comparatively) avoiding the problem of catastrophic forgetting.

[Paper Reading] RAGAS: Automated Evaluation of Retrieval Augmented Generation

Clay
2024-08-102024-08-10
Machine Learning, PyTorch

Introduction

The year 2023 witnessed an explosion of generative AI technologies, with a myriad of applications emerging across various domains. In the field of Natural Language Processing (NLP), Large Language Models (LLMs) stand out as one of the most significant advancements. By training LLMs effectively and reducing hallucinations, they can significantly reduce human effort across a wide range of tasks.

« Previous
1
2
3
4
5
6
…
15
Next »