August 2024

[Flutter] Using bitsdojo_window to Customize Window Title Bar

Clay
2024-08-202024-08-20
Flutter, Linux

Last Updated on 2024-08-20 by Clay

Recently, I started experimenting with Flutter to develop desktop applications. Since I’m using the Linux Gnome desktop, its default system window bar (the top strip) is completely black, which doesn't match the light and playful tone of the app I’m developing. So, I found a great tool: bitsdojo_window.

[Machine Learning] Note of LayerNorm

Clay
2024-08-192024-08-19
Machine Learning, PyTorch

Last Updated on 2024-08-19 by Clay

The working principle of LayerNorm is as follows:

Calculate mean and variance

$mean = \mu =\frac{\sum_{i=1}^{N}x_i}{N} \newline variance = \sigma^2 = \frac{\sum_{i=1}^{N}(x_{i}-\mu)^2}{N}$ Read More »

[Machine Learning] Note of Cross Entropy Loss

Clay
2024-08-182024-08-18
Machine Learning, PyTorch

Last Updated on 2024-08-18 by Clay

Introduction to Cross Entropy

Cross entropy is a very common loss function in Machine Learning, as it is able to quantify the difference between a model's classification predictions and the actual class labels, particularly in 'classification tasks'.

[Machine Learning] Note of Activation Function GELU

Clay
2024-08-182024-08-18
Machine Learning, PyTorch

Last Updated on 2024-08-18 by Clay

Gaussian Error Linear Unit (GELU) is an activation function used in machine learning. While it resembles the classic ReLU (Rectified Linear Unit), there are some key differences.

[Machine Learning] Note of RMSNorm

Clay
2024-08-172024-08-17
Machine Learning, PyTorch

Last Updated on 2024-08-17 by Clay

Introduction to RMSNorm

RMSNorm is an improvement over LayerNorm, often used in the Transformer self-attention mechanism. It aims to mitigate the issues of vanishing and exploding gradients, helping the model converge faster and improve performance.

[Machine Learning] Note of Rotary Position Embedding (RoPE)

Clay
2024-08-162024-08-17
Machine Learning, PyTorch

Last Updated on 2024-08-17 by Clay

Introduction

(Note: Since this article is imported from my personal Hackmd, some symbols and formatting might not display properly in WordPress. I appreciate your understanding, sorry for any inconvenience.)

RoPE is a method for introducing relative position information into the self-attention mechanism through absolute positional encoding.

[PyTorch] Using SDPA in 2.0+ to Improve the Computation Speed of Transformer’s Self-Attention Mechanism

Clay
2024-08-152024-08-16
Machine Learning, PyTorch

Last Updated on 2024-08-16 by Clay

SDPA Introduction

Scaled Dot-Product Attention (SDPA) might immediately pop into the minds of those familiar with the Transformer self-attention mechanism:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$

[Linux] tty-clock: A Tool For Displaying Electronic Clock Time On Terminal

Clay
2024-08-142024-08-14
Linux

Last Updated on 2024-08-14 by Clay

Introduction

I often use my computer to watch full-screen videos and need to promptly return to work mode when the time comes — but embarrassingly, I don't like constantly picking up my phone to check the current time, as it's quite inconvenient.

[Paper Reading] Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Clay
2024-08-132024-08-19
AI, Machine Learning

Last Updated on 2024-08-19 by Clay

Cross-lingual Modular (X-Mod) is an interesting language model architecture that modularizes the parameters for different languages as Module Units, allowing the model to use separate parameters when fine-tuning for a new language, thereby (comparatively) avoiding the problem of catastrophic forgetting.

Stable Diffusion ComfyUI Note 03 - How To Download SD Models

Clay
2024-08-122024-08-12
AI

Last Updated on 2024-08-12 by Clay

When using ComfyUI to generate images, we need to leverage the capabilities of various models to ultimately form a complete workflow. In other words, these so-called various models together constitute what we call Stable Diffusion. Today, I will introduce where to download these models.