Skip to content

Blog

Using the Integrated Outlines Tool for Decoding Constraints in the vLLM Inference Acceleration Framework

Recently, I integrated several applications of Outlines into my current workflow. Among them, the one I use most frequently is with vLLM. However, for some reason, its documentation has not been merged into the vLLM GitHub repository, so while designing the process, I had to constantly refer to the source code of a rejected PR for guidance XD

Read More »Using the Integrated Outlines Tool for Decoding Constraints in the vLLM Inference Acceleration Framework

Implementation of Using Finite-State Machine to Constrain Large Language Model Decoding

This is a simple Python implementation, used to test Finite-State Machine (FSM) constraints for a Large Language Model (LLM) to decode responses in a specific format. It also serves as an introduction to the concept behind the Outlines tool. Of course, my implementation is far simpler compared to the actual Outlines tool.

Read More »Implementation of Using Finite-State Machine to Constrain Large Language Model Decoding

Implementing Streamed Output Token Generation Using TextStreamer and TextIteratorStreamer in HuggingFace Transformers

Introduction

Generative models are becoming increasingly powerful, and independent researchers are deploying one open-source large language model (LLMs) after another. However, when using LLMs for inference or generating responses, waiting for a longer output can be quite time-consuming.

Read More »Implementing Streamed Output Token Generation Using TextStreamer and TextIteratorStreamer in HuggingFace Transformers

Evaluating LLM Defense Capabilities Using the Microsoft BIPIA Framework

Currently, LLM services cover a wide range of fields, and Prompt Injection and Jailbreak threats to LLMs are growing by the day. A few months ago, a customer service LLM even provided incorrect information, leading to a loss of customer rights (although that wasn’t caused by a prompt attack).

Microsoft’s open-source BIPIA (Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models) evaluation method, although tested six months ago without significant updates since, remains a simple and convenient testing method for the tasks I have at hand.

Read More »Evaluating LLM Defense Capabilities Using the Microsoft BIPIA Framework

Note Of Newton Polynomial

Newton’s interpolation is a polynomial interpolation method that constructs a set of polynomial functions using multiple data points. A major advantage is that with the addition of new data, Newton’s interpolation method does not require recalculations from scratch but can instead expand on the existing function.

Read More »Note Of Newton Polynomial