<div style="display: flex; align-items: center;">
<div style="width: 100px; margin-right: 10px; height:auto;" align="left">
<img src="images/LLMLingua_logo.png" alt="LLMLingua" width="100" align="left">
</div>
<div style="flex-grow: 1;" align="center">
<h2 align="center">LLMLingua Series | Effectively Deliver Information to LLMs via Prompt Compression</h2>
</div>
</div>
<p align="center">
| <a href="https://llmlingua.com/"><b>Project Page</b></a> |
<a href="https://aclanthology.org/2023.emnlp-main.825/"><b>LLMLingua</b></a> |
<a href="https://arxiv.org/abs/2310.06839"><b>LongLLMLingua</b></a> |
<a href="https://arxiv.org/abs/2403.12968"><b>LLMLingua-2</b></a> |
<a href="https://huggingface.co/spaces/microsoft/LLMLingua"><b>LLMLingua Demo</b></a> |
<a href="https://huggingface.co/spaces/microsoft/LLMLingua-2"><b>LLMLingua-2 Demo</b></a> |
</p>
https://github.com/microsoft/LLMLingua/assets/30883354/eb0ea70d-6d4c-4aa7-8977-61f94bb87438
## News
- 🦚 We're excited to announce the release of **LLMLingua-2**, boasting a 3x-6x speed improvement over LLMLingua! For more information, check out our [paper](https://arxiv.org/abs/2403.12968), visit the [project page](https://llmlingua.com/llmlingua2.html), and explore our [demo](https://huggingface.co/spaces/microsoft/LLMLingua-2).
- 👾 LLMLingua has been integrated into [LangChain](https://github.com/langchain-ai/langchain/blob/master/docs/docs/integrations/retrievers/llmlingua.ipynb) and [LlamaIndex](https://github.com/run-llama/llama_index/blob/main/docs/examples/node_postprocessor/LongLLMLingua.ipynb), two widely-used RAG frameworks.
- 🤳 Talk slides are available in [AI Time Jan, 24](https://drive.google.com/file/d/1fzK3wOvy2boF7XzaYuq2bQ3jFeP1WMk3/view?usp=sharing).
- 🖥 EMNLP'23 slides are available in [Session 5](https://drive.google.com/file/d/1GxQLAEN8bBB2yiEdQdW4UKoJzZc0es9t/view) and [BoF-6](https://drive.google.com/file/d/1LJBUfJrKxbpdkwo13SgPOqugk-UjLVIF/view).
- 📚 Check out our new [blog post](https://medium.com/@iofu728/longllmlingua-bye-bye-to-middle-loss-and-save-on-your-rag-costs-via-prompt-compression-54b559b9ddf7) discussing RAG benefits and cost savings through prompt compression. See the script example [here](https://github.com/microsoft/LLMLingua/blob/main/examples/Retrieval.ipynb).
- 🎈 Visit our [project page](https://llmlingua.com/) for real-world case studies in RAG, Online Meetings, CoT, and Code.
- 👨🦯 Explore our ['./examples'](./examples) directory for practical applications, including [LLMLingua-2](./examples/LLMLingua2.ipynb), [RAG](./examples/RAG.ipynb), [Online Meeting](./examples/OnlineMeeting.ipynb), [CoT](./examples/CoT.ipynb), [Code](./examples/Code.ipynb), and [RAG using LlamaIndex](./examples/RAGLlamaIndex.ipynb).
## TL;DR
LLMLingua utilizes a compact, well-trained language model (e.g., GPT2-small, LLaMA-7B) to identify and remove non-essential tokens in prompts. This approach enables efficient inference with large language models (LLMs), achieving up to 20x compression with minimal performance loss.
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://aclanthology.org/2023.emnlp-main.825/) (EMNLP 2023)<br>
_Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang and Lili Qiu_
LongLLMLingua mitigates the 'lost in the middle' issue in LLMs, enhancing long-context information processing. It reduces costs and boosts efficiency with prompt compression, improving RAG performance by up to 21.4% using only 1/4 of the tokens.
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://arxiv.org/abs/2310.06839) (ICLR ME-FoMo 2024)<br>
_Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang and Lili Qiu_
LLMLingua-2, a small-size yet powerful prompt compression method trained via data distillation from GPT-4 for token classification with a BERT-level encoder, excels in task-agnostic compression. It surpasses LLMLingua in handling out-of-domain data, offering 3x-6x faster performance.
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://arxiv.org/abs/2403.12968) (Under Review)<br>
_Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang_
## 🎥 Overview
- Ever encountered the token limit when asking ChatGPT to summarize lengthy texts?
- Frustrated with ChatGPT forgetting previous instructions after extensive fine-tuning?
- Experienced high costs using GPT3.5/4 API for experiments despite excellent results?
While Large Language Models like ChatGPT and GPT-4 excel in generalization and reasoning, they often face challenges like prompt length limits and prompt-based pricing schemes.
Now you can use **LLMLingua**, **LongLLMLingua**, and **LLMLingua-2**!
These tools offer an efficient solution to compress prompts by up to **20x**, enhancing the utility of LLMs.
- 💰 **Cost Savings**: Reduces both prompt and generation lengths with minimal overhead.
- 📝 **Extended Context Support**: Enhances support for longer contexts, mitigates the "lost in the middle" issue, and boosts overall performance.
- ⚖️ **Robustness**: No additional training needed for LLMs.
- 🕵️ **Knowledge Retention**: Maintains original prompt information like ICL and reasoning.
- 📜 **KV-Cache Compression**: Accelerates inference process.
- 🪃 **Comprehensive Recovery**: GPT-4 can recover all key information from compressed prompts.
PS: This demo is based on the [alt-gpt](https://github.com/feedox/alt-gpt) project. Special thanks to @Livshitz for their valuable contribution.
If you find this repo helpful, please cite the following papers:
```bibtex
@inproceedings{jiang-etal-2023-llmlingua,
title = "{LLML}ingua: Compressing Prompts for Accelerated Inference of Large Language Models",
author = "Huiqiang Jiang and Qianhui Wu and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.emnlp-main.825",
doi = "10.18653/v1/2023.emnlp-main.825",
pages = "13358--13376",
}
```
```bibtex
@article{jiang-etal-2023-longllmlingua,
title = "{L}ong{LLML}ingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression",
author = "Huiqiang Jiang and Qianhui Wu and and Xufang Luo and Dongsheng Li and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
url = "https://arxiv.org/abs/2310.06839",
journal = "ArXiv preprint",
volume = "abs/2310.06839",
year = "2023",
}
```
```bibtex
@article{wu2024llmlingua2,
title = "{LLML}ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression",
author = "Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Menglin Xia and Xufang Luo and Jue Zhang and Qingwei Lin and Victor Ruhle and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Dongmei Zhang",
url = "https://arxiv.org/abs/2403.12968",
journal = "ArXiv preprint",
volume = "abs/2403.12968",
year = "2024",
}
```
## 🎯 Quick Start
#### 1. **Installing LLMLingua:**
To get started with LLMLingua, simply install it using pip:
```bash
pip install llmlingua
```
#### 2. **Using LLMLingua Series Methods for Prompt Compression:**
With **LLMLingua**, you can easily compress your prompts. Here’s how you can do it:
```python
from llmlingua import PromptCompressor
llm_lingua = PromptCompressor()
compressed_prompt = llm_lingua.compress_prompt(prompt, instruction="", question="", target_token=200)
# > {'compressed_prompt': 'Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He reanged five of boxes into packages of sixlters each and sold them $3 per. He sold the rest theters separately at the of three pens $2. How much did make in total, dollars?\nLets think step step\nSam bought 1 boxes x00 oflters.\nHe bought 12 * 300ters in total\nSam then took 5 boxes 6ters0ters.\nHe sold these boxes for 5 *5\nAfterelling these boxes there were 3030 highlighters remaining.\nThese form 330 / 3 = 110 groups of three pens.\nHe sold each of these groups for $2 each, so made 110 * 2 = $220 from them.\nIn total, then, he earned $220 + $15 = $235.\nSince his original cost was $120, he earned $235 - $120 = $115 in profit.\nThe answer is 115',
# 'origin_tokens': 2365,
# 'compressed_tokens': 211,
# 'ratio': '11.2x',
# 'saving': ', Saving $0.1 in GPT-4.'}
## Or use the phi-2 model,
llm_lingua = PromptCompressor("microsoft/phi-2")
## Or use the quantation model, like TheBloke/Llama-2-7b-Chat-GPTQ, only need <8GB GPU memory.
## Before that, you need to pip install optimum auto-gptq
llm_lingua = PromptCompressor("TheBloke/Llama-2-7b-Chat-GPTQ", model_config={"revision": "main"})
```
To try **LongLLMLingua** in your scenorias, you can use
```python
from llmlingua import PromptCompressor
llm_lingua = PromptCompressor()
compressed_prompt = llm_lingua.compress_prompt(
prompt_list,
question=question,
ratio=0.55,
# Set the special parameter for LongLLMLingua
condition_in_question="after_condition",
reorder_context="sort",
dynamic_context_compression_ratio=0.3, # or 0.4
condition_compare=True,
context_budget="+100",
rank_method="longllmlingua",
)
```
To try **LLMLingua-2** in your scenorias, you can use
```python
from llmlingua import PromptCompressor
llm_lingua = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True, # Whether to use llmlingua-2
)
compressed_prompt = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens = ['\n', '?'])
## Or use LLMLingua-2-small model
llm_lingua = PromptCompressor(
model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
use_llmlingua2=True, # Whether to use llmlingua-2
)
```
#### 3. **Advanced usage - Structured Prompt Compression:**
Split text into sections, decide on whether to compress and its rate. Use `<llmlingua></llmlingua>` tags for context segmentation, with optional rate and compress parameters.
```python
structured_prompt = """<llmlingua, compress=False>Speaker 4:</llmlingua><llmlingua, rate=0.4> Thank you. And can we do the functions for content? Items I believe are 11, three, 14, 16 and 28, I believe.</llmlingua><llmlingua, compress=False>
Speaker 0:</llmlingua><llmlingua, rate=0.4> Item 11 is a communication from Council on Price recommendation to increase appropriation in the general fund group in the City Manager Department by $200 to provide a contribution to the Friends of the Long Beach Public Library. Item 12 is communication from Councilman Super Now. Recommendation to increase appropriation in the special advertising and promotion fund group and the city manager's department by $10,000 to provide support for the end of summer celebration. Item 13 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the city manager department by $500 to provide a donation to the Jazz Angels . Item 14 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.</llmlingua><llmlingua, compress=False>
Speaker 4:</llmlingua><llmlingua, rate=0.6> We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.</llmlingua>"""
compressed_prompt = llm_lingua.structured_compress_prompt(structured_prompt, instruction="", question="", rate=0.5)
print(compressed_prompt['compressed_prompt'])
# > Speaker 4:. And can we do the functions for content? Items I believe are11,,116 28,.
# Speaker 0: a from Council on Price to increase the fund group the Manager0 provide a the the1 is Councilman Super Now. the special group the provide the summerman a the Jazzels a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.
# Speaker 4: We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.
```
#### 4. **Learning More:**
To understand how to apply LLMLingua and LongLLMLingua in real-world scenarios like RAG, Online Meetings, CoT, and Code, please refer to our [**examples**](./examples). For detailed guidance, the [**documentation**](./DOCUMENT.md) provides extensive recommendations on effectively utilizing LLMLingua.
#### 5. **Data collection and model training of LLMLingua-2:**
To train the compressor on your custom data, please refer to our [**data_collection**](./experiments/llmlingua2/data_collection) and [**model_training**](./experiments/llmlingua2/model_training).
## Frequently Asked Questions
For more insights and answers, visit our [FAQ section](./Transparency_FAQ.md).
## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
Raw data
{
"_id": null,
"home_page": "https://github.com/microsoft/LLMLingua",
"name": "llmlingua",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": null,
"keywords": "Prompt Compression, LLMs, Inference Acceleration, Black-box LLMs, Efficient LLMs",
"author": "The LLMLingua team",
"author_email": "hjiang@microsoft.com",
"download_url": "https://files.pythonhosted.org/packages/fd/02/7ed8ad9cec4b2f0a8a45c5b50a2435b559d2c764b5865e896d1c511080b1/llmlingua-0.2.2.tar.gz",
"platform": null,
"description": "<div style=\"display: flex; align-items: center;\">\n <div style=\"width: 100px; margin-right: 10px; height:auto;\" align=\"left\">\n <img src=\"images/LLMLingua_logo.png\" alt=\"LLMLingua\" width=\"100\" align=\"left\">\n </div>\n <div style=\"flex-grow: 1;\" align=\"center\">\n <h2 align=\"center\">LLMLingua Series | Effectively Deliver Information to LLMs via Prompt Compression</h2>\n </div>\n</div>\n\n<p align=\"center\">\n | <a href=\"https://llmlingua.com/\"><b>Project Page</b></a> |\n <a href=\"https://aclanthology.org/2023.emnlp-main.825/\"><b>LLMLingua</b></a> |\n <a href=\"https://arxiv.org/abs/2310.06839\"><b>LongLLMLingua</b></a> |\n <a href=\"https://arxiv.org/abs/2403.12968\"><b>LLMLingua-2</b></a> |\n <a href=\"https://huggingface.co/spaces/microsoft/LLMLingua\"><b>LLMLingua Demo</b></a> |\n <a href=\"https://huggingface.co/spaces/microsoft/LLMLingua-2\"><b>LLMLingua-2 Demo</b></a> |\n</p>\n\nhttps://github.com/microsoft/LLMLingua/assets/30883354/eb0ea70d-6d4c-4aa7-8977-61f94bb87438\n\n## News\n\n- \ud83e\udd9a We're excited to announce the release of **LLMLingua-2**, boasting a 3x-6x speed improvement over LLMLingua! For more information, check out our [paper](https://arxiv.org/abs/2403.12968), visit the [project page](https://llmlingua.com/llmlingua2.html), and explore our [demo](https://huggingface.co/spaces/microsoft/LLMLingua-2).\n- \ud83d\udc7e LLMLingua has been integrated into [LangChain](https://github.com/langchain-ai/langchain/blob/master/docs/docs/integrations/retrievers/llmlingua.ipynb) and [LlamaIndex](https://github.com/run-llama/llama_index/blob/main/docs/examples/node_postprocessor/LongLLMLingua.ipynb), two widely-used RAG frameworks.\n- \ud83e\udd33 Talk slides are available in [AI Time Jan, 24](https://drive.google.com/file/d/1fzK3wOvy2boF7XzaYuq2bQ3jFeP1WMk3/view?usp=sharing).\n- \ud83d\udda5 EMNLP'23 slides are available in [Session 5](https://drive.google.com/file/d/1GxQLAEN8bBB2yiEdQdW4UKoJzZc0es9t/view) and [BoF-6](https://drive.google.com/file/d/1LJBUfJrKxbpdkwo13SgPOqugk-UjLVIF/view).\n- \ud83d\udcda Check out our new [blog post](https://medium.com/@iofu728/longllmlingua-bye-bye-to-middle-loss-and-save-on-your-rag-costs-via-prompt-compression-54b559b9ddf7) discussing RAG benefits and cost savings through prompt compression. See the script example [here](https://github.com/microsoft/LLMLingua/blob/main/examples/Retrieval.ipynb).\n- \ud83c\udf88 Visit our [project page](https://llmlingua.com/) for real-world case studies in RAG, Online Meetings, CoT, and Code.\n- \ud83d\udc68\u200d\ud83e\uddaf Explore our ['./examples'](./examples) directory for practical applications, including [LLMLingua-2](./examples/LLMLingua2.ipynb), [RAG](./examples/RAG.ipynb), [Online Meeting](./examples/OnlineMeeting.ipynb), [CoT](./examples/CoT.ipynb), [Code](./examples/Code.ipynb), and [RAG using LlamaIndex](./examples/RAGLlamaIndex.ipynb).\n\n## TL;DR\n\nLLMLingua utilizes a compact, well-trained language model (e.g., GPT2-small, LLaMA-7B) to identify and remove non-essential tokens in prompts. This approach enables efficient inference with large language models (LLMs), achieving up to 20x compression with minimal performance loss.\n\n- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://aclanthology.org/2023.emnlp-main.825/) (EMNLP 2023)<br>\n _Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang and Lili Qiu_\n\nLongLLMLingua mitigates the 'lost in the middle' issue in LLMs, enhancing long-context information processing. It reduces costs and boosts efficiency with prompt compression, improving RAG performance by up to 21.4% using only 1/4 of the tokens.\n\n- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://arxiv.org/abs/2310.06839) (ICLR ME-FoMo 2024)<br>\n _Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang and Lili Qiu_\n\nLLMLingua-2, a small-size yet powerful prompt compression method trained via data distillation from GPT-4 for token classification with a BERT-level encoder, excels in task-agnostic compression. It surpasses LLMLingua in handling out-of-domain data, offering 3x-6x faster performance.\n\n- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://arxiv.org/abs/2403.12968) (Under Review)<br>\n _Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang_\n\n## \ud83c\udfa5 Overview\n\n\n\n- Ever encountered the token limit when asking ChatGPT to summarize lengthy texts?\n- Frustrated with ChatGPT forgetting previous instructions after extensive fine-tuning?\n- Experienced high costs using GPT3.5/4 API for experiments despite excellent results?\n\nWhile Large Language Models like ChatGPT and GPT-4 excel in generalization and reasoning, they often face challenges like prompt length limits and prompt-based pricing schemes.\n\n\n\nNow you can use **LLMLingua**, **LongLLMLingua**, and **LLMLingua-2**!\n\nThese tools offer an efficient solution to compress prompts by up to **20x**, enhancing the utility of LLMs.\n\n- \ud83d\udcb0 **Cost Savings**: Reduces both prompt and generation lengths with minimal overhead.\n- \ud83d\udcdd **Extended Context Support**: Enhances support for longer contexts, mitigates the \"lost in the middle\" issue, and boosts overall performance.\n- \u2696\ufe0f **Robustness**: No additional training needed for LLMs.\n- \ud83d\udd75\ufe0f **Knowledge Retention**: Maintains original prompt information like ICL and reasoning.\n- \ud83d\udcdc **KV-Cache Compression**: Accelerates inference process.\n- \ud83e\ude83 **Comprehensive Recovery**: GPT-4 can recover all key information from compressed prompts.\n\n\n\n\n\n\n\nPS: This demo is based on the [alt-gpt](https://github.com/feedox/alt-gpt) project. Special thanks to @Livshitz for their valuable contribution.\n\nIf you find this repo helpful, please cite the following papers:\n\n```bibtex\n@inproceedings{jiang-etal-2023-llmlingua,\n title = \"{LLML}ingua: Compressing Prompts for Accelerated Inference of Large Language Models\",\n author = \"Huiqiang Jiang and Qianhui Wu and Chin-Yew Lin and Yuqing Yang and Lili Qiu\",\n booktitle = \"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing\",\n month = dec,\n year = \"2023\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2023.emnlp-main.825\",\n doi = \"10.18653/v1/2023.emnlp-main.825\",\n pages = \"13358--13376\",\n}\n```\n\n```bibtex\n@article{jiang-etal-2023-longllmlingua,\n title = \"{L}ong{LLML}ingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression\",\n author = \"Huiqiang Jiang and Qianhui Wu and and Xufang Luo and Dongsheng Li and Chin-Yew Lin and Yuqing Yang and Lili Qiu\",\n url = \"https://arxiv.org/abs/2310.06839\",\n journal = \"ArXiv preprint\",\n volume = \"abs/2310.06839\",\n year = \"2023\",\n}\n```\n\n```bibtex\n@article{wu2024llmlingua2,\n title = \"{LLML}ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression\",\n author = \"Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Menglin Xia and Xufang Luo and Jue Zhang and Qingwei Lin and Victor Ruhle and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Dongmei Zhang\",\n url = \"https://arxiv.org/abs/2403.12968\",\n journal = \"ArXiv preprint\",\n volume = \"abs/2403.12968\",\n year = \"2024\",\n}\n```\n\n## \ud83c\udfaf Quick Start\n\n#### 1. **Installing LLMLingua:**\n\nTo get started with LLMLingua, simply install it using pip:\n\n```bash\npip install llmlingua\n```\n\n#### 2. **Using LLMLingua Series Methods for Prompt Compression:**\n\nWith **LLMLingua**, you can easily compress your prompts. Here\u2019s how you can do it:\n\n```python\nfrom llmlingua import PromptCompressor\n\nllm_lingua = PromptCompressor()\ncompressed_prompt = llm_lingua.compress_prompt(prompt, instruction=\"\", question=\"\", target_token=200)\n\n# > {'compressed_prompt': 'Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He reanged five of boxes into packages of sixlters each and sold them $3 per. He sold the rest theters separately at the of three pens $2. How much did make in total, dollars?\\nLets think step step\\nSam bought 1 boxes x00 oflters.\\nHe bought 12 * 300ters in total\\nSam then took 5 boxes 6ters0ters.\\nHe sold these boxes for 5 *5\\nAfterelling these boxes there were 3030 highlighters remaining.\\nThese form 330 / 3 = 110 groups of three pens.\\nHe sold each of these groups for $2 each, so made 110 * 2 = $220 from them.\\nIn total, then, he earned $220 + $15 = $235.\\nSince his original cost was $120, he earned $235 - $120 = $115 in profit.\\nThe answer is 115',\n# 'origin_tokens': 2365,\n# 'compressed_tokens': 211,\n# 'ratio': '11.2x',\n# 'saving': ', Saving $0.1 in GPT-4.'}\n\n## Or use the phi-2 model,\nllm_lingua = PromptCompressor(\"microsoft/phi-2\")\n\n## Or use the quantation model, like TheBloke/Llama-2-7b-Chat-GPTQ, only need <8GB GPU memory.\n## Before that, you need to pip install optimum auto-gptq\nllm_lingua = PromptCompressor(\"TheBloke/Llama-2-7b-Chat-GPTQ\", model_config={\"revision\": \"main\"})\n```\n\nTo try **LongLLMLingua** in your scenorias, you can use\n\n```python\nfrom llmlingua import PromptCompressor\n\nllm_lingua = PromptCompressor()\ncompressed_prompt = llm_lingua.compress_prompt(\n prompt_list,\n question=question,\n ratio=0.55,\n # Set the special parameter for LongLLMLingua\n condition_in_question=\"after_condition\",\n reorder_context=\"sort\",\n dynamic_context_compression_ratio=0.3, # or 0.4\n condition_compare=True,\n context_budget=\"+100\",\n rank_method=\"longllmlingua\",\n)\n```\n\nTo try **LLMLingua-2** in your scenorias, you can use\n\n```python\nfrom llmlingua import PromptCompressor\n\nllm_lingua = PromptCompressor(\n model_name=\"microsoft/llmlingua-2-xlm-roberta-large-meetingbank\",\n use_llmlingua2=True, # Whether to use llmlingua-2\n)\ncompressed_prompt = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens = ['\\n', '?'])\n\n## Or use LLMLingua-2-small model\nllm_lingua = PromptCompressor(\n model_name=\"microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank\",\n use_llmlingua2=True, # Whether to use llmlingua-2\n)\n```\n\n#### 3. **Advanced usage - Structured Prompt Compression:**\n\nSplit text into sections, decide on whether to compress and its rate. Use `<llmlingua></llmlingua>` tags for context segmentation, with optional rate and compress parameters.\n\n```python\nstructured_prompt = \"\"\"<llmlingua, compress=False>Speaker 4:</llmlingua><llmlingua, rate=0.4> Thank you. And can we do the functions for content? Items I believe are 11, three, 14, 16 and 28, I believe.</llmlingua><llmlingua, compress=False>\nSpeaker 0:</llmlingua><llmlingua, rate=0.4> Item 11 is a communication from Council on Price recommendation to increase appropriation in the general fund group in the City Manager Department by $200 to provide a contribution to the Friends of the Long Beach Public Library. Item 12 is communication from Councilman Super Now. Recommendation to increase appropriation in the special advertising and promotion fund group and the city manager's department by $10,000 to provide support for the end of summer celebration. Item 13 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the city manager department by $500 to provide a donation to the Jazz Angels . Item 14 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.</llmlingua><llmlingua, compress=False>\nSpeaker 4:</llmlingua><llmlingua, rate=0.6> We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.</llmlingua>\"\"\"\ncompressed_prompt = llm_lingua.structured_compress_prompt(structured_prompt, instruction=\"\", question=\"\", rate=0.5)\nprint(compressed_prompt['compressed_prompt'])\n\n# > Speaker 4:. And can we do the functions for content? Items I believe are11,,116 28,.\n# Speaker 0: a from Council on Price to increase the fund group the Manager0 provide a the the1 is Councilman Super Now. the special group the provide the summerman a the Jazzels a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.\n# Speaker 4: We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.\n```\n\n#### 4. **Learning More:**\n\nTo understand how to apply LLMLingua and LongLLMLingua in real-world scenarios like RAG, Online Meetings, CoT, and Code, please refer to our [**examples**](./examples). For detailed guidance, the [**documentation**](./DOCUMENT.md) provides extensive recommendations on effectively utilizing LLMLingua.\n\n#### 5. **Data collection and model training of LLMLingua-2:**\n\nTo train the compressor on your custom data, please refer to our [**data_collection**](./experiments/llmlingua2/data_collection) and [**model_training**](./experiments/llmlingua2/model_training).\n\n## Frequently Asked Questions\n\nFor more insights and answers, visit our [FAQ section](./Transparency_FAQ.md).\n\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft\ntrademarks or logos is subject to and must follow\n[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.",
"version": "0.2.2",
"project_urls": {
"Homepage": "https://github.com/microsoft/LLMLingua"
},
"split_keywords": [
"prompt compression",
" llms",
" inference acceleration",
" black-box llms",
" efficient llms"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6e3e221fe46a3338f2babdb2082ee42df88fcaa8ea0e639e832cbb1b93c5923a",
"md5": "6c0efb180fc9d7f7cab51d073d419600",
"sha256": "da55137efe0db78063b3395396efe8a0dcfe4ae5a09aea0d503c34b7bf1d800c"
},
"downloads": -1,
"filename": "llmlingua-0.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6c0efb180fc9d7f7cab51d073d419600",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 30536,
"upload_time": "2024-04-09T08:21:55",
"upload_time_iso_8601": "2024-04-09T08:21:55.428185Z",
"url": "https://files.pythonhosted.org/packages/6e/3e/221fe46a3338f2babdb2082ee42df88fcaa8ea0e639e832cbb1b93c5923a/llmlingua-0.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fd027ed8ad9cec4b2f0a8a45c5b50a2435b559d2c764b5865e896d1c511080b1",
"md5": "131f7f6f7e1cd495dcf8cb7be2d057ee",
"sha256": "1a0caedd8d5a65512a85dadb6bfda6f5b3c4b45e5cb9e7b1c6009573f9058572"
},
"downloads": -1,
"filename": "llmlingua-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "131f7f6f7e1cd495dcf8cb7be2d057ee",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 59753,
"upload_time": "2024-04-09T08:21:56",
"upload_time_iso_8601": "2024-04-09T08:21:56.880658Z",
"url": "https://files.pythonhosted.org/packages/fd/02/7ed8ad9cec4b2f0a8a45c5b50a2435b559d2c764b5865e896d1c511080b1/llmlingua-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-09 08:21:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "microsoft",
"github_project": "LLMLingua",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llmlingua"
}
JSON
