llmlingua-2-js

JavaScript/TypeScript Implementation of LLMLingua-2 (Experimental)

LLMLingua-2, Originally developed and implemented in Python by Microsoft, is a small-size yet powerful prompt compression method.

• Efficient: Compresses context prompts with BERT sized models.
• Accurate: Achieves high accuracy then other methods while requiring less computational resources.

llmlingua-2-js, ported by atjsh, is a pure JavaScript/TypeScript implementation of LLMLingua-2, designed to run in web browsers and Node.js environments.

• Performance: Everything can be done in the browser. If your environment supports WebGPU, you can use it. Server-side processing is not required by default.
• Correctness: The original logic will be ported to TypeScript as accurately as possible.

Try Demo Online (No Installation Required)

You can try it on the GitHub Pages without any installation.

The source code for demo is available in the examples directory. You can run it locally using the following command:

cd examples/react-vite-webgpu
yarn install

Learn More: Demo source code

Getting Started

This implementation depends on the following libraries:

• @huggingface/transformers
• js-tiktoken

Especially, the @huggingface/transformers library utilizes various computational optimizations to achieve high performance. Please consult if the running environment supports the minimum requirements from these libraries.

Installation

You can use the library by downloading the library from npm and importing it into your JavaScript/TypeScript project.

First, install the dependencies:

npm install @huggingface/transformers js-tiktoken

Then, install the library:

npm install @atjsh/llmlingua-2

Runtime Backend Selection

Pass an explicit transformerJSConfig to the factory. These portable configurations favor predictable output across supported runtimes:

const nodeTransformerJSConfig = {
  device: "cpu",
  dtype: "fp32",
} as const;

const browserTransformerJSConfig = {
  device: "wasm",
  dtype: "fp32",
} as const;

Hardware acceleration is opt-in and should be tested on each target platform:

Runtime	device	dtype
Windows Node.js with DirectML	"dml"	"fp32"
Linux Node.js with NVIDIA CUDA	"cuda"	"fp32"
Compatible browsers or macOS Node.js	"webgpu"	"fp32"

The library passes the configuration to Transformers.js without detecting a backend or retrying with a fallback. For an XLM-RoBERTa model stored with one external data file, also pass:

modelSpecificOptions: {
  use_external_data_format: { "model.onnx": 1 },
}

Model Selection

You can choose between models based on your needs.

Model	Size	Pros	Cons	Public Model
TinyBERT	57.1 MB	Very small, fast	Lower accuracy than larger models	atjsh/llmlingua-2-js-tinybert-meetingbank
MobileBERT	99.2 MB	Small, optimized for mobile	Moderate accuracy, tradeoff in depth	atjsh/llmlingua-2-js-mobilebert-meetingbank
BERT	710 MB	Faster, smaller size	Lower accuracy than XLM-RoBERTa	Arcoldd/llmlingua4j-bert-base-onnx
XLM-RoBERTa	2240 MB	High accuracy	Slower, slightly larger in size	atjsh/llmlingua-2-js-xlm-roberta-large-meetingbank

Learn More about the performance of each model (actual performance may vary).

The model files will be downloaded automatically by default.

API Reference

For more details on how to use the library, please refer to the API reference documentation.

Testing

Unit tests are not available at the moment.

E2E tests are partially available in following directories:

• src/e2e
• examples/**

License

See LICENSE for details.

Credits

This software includes other software related under the following licenses:

• LLMLingua (https://github.com/microsoft/LLMLingua), Copyright © Microsoft Corporation. (The original logic and implementation was licensed under the MIT license. For licensing, see: https://github.com/microsoft/LLMLingua/blob/main/LICENSE )