# *LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders*
[](https://arxiv.org/abs/2404.05961)
[](https://pypi.org/project/llm2vec/)
[](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34)
[](https://github.com/McGill-NLP/llm2vec/blob/main/LICENSE)
[](https://pepy.tech/project/llm2vec)
LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) training with masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.
<p align="center">
<img src="https://github.com/McGill-NLP/llm2vec/assets/12207571/48efd48a-431b-4625-8e0f-248a442e3839" width="75%" alt="LLM2Vec_figure1"/>
</p>
**************************** **Updates** ****************************
* 04/07: Added support for Gemma and Qwen-2 models, huge thanks to [@bzantium](https://github.com/bzantium) for the contribution.
* 30/04: We release LLM2Vec transformed Meta-Llama-3 checkpoints. See our [HuggingFace collection](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34) for both [supervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised) and [unsupervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse) variants.
## Installation
To use LLM2Vec, first install the llm2vec package from PyPI, followed by installing flash-attention:
```bash
pip install llm2vec
pip install flash-attn --no-build-isolation
```
You can also directly install the latest version of llm2vec by cloning the repository:
```bash
pip install -e .
pip install flash-attn --no-build-isolation
```
## Getting Started
LLM2Vec class is a wrapper on top of HuggingFace models to support enabling bidirectionality in decoder-only LLMs, sequence encoding and pooling operations. The steps below showcase an example on how to use the library.
### Preparing the model
Initializing LLM2Vec model using pretrained LLMs is straightforward. The `from_pretrained` method of LLM2Vec takes a base model identifier/path and an optional PEFT model identifier/path. All HuggingFace model loading arguments can be passed to `from_pretrained` method. By default, the models are loaded with bidirectional connections enabled. This can be turned off by passing `enable_bidirectional=False` to the `from_pretrained` method.
Here, we first initialize the Llama-3 MNTP base model and load the unsupervised-trained LoRA weights (trained with SimCSE objective and wiki corpus).
```python
import torch
from llm2vec import LLM2Vec
l2v = LLM2Vec.from_pretrained(
"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse",
device_map="cuda" if torch.cuda.is_available() else "cpu",
torch_dtype=torch.bfloat16,
)
```
We can also load the model with supervised-trained LoRA weights (trained with contrastive learning and public E5 data) by changing the `peft_model_name_or_path`.
```python
import torch
from llm2vec import LLM2Vec
l2v = LLM2Vec.from_pretrained(
"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
device_map="cuda" if torch.cuda.is_available() else "cpu",
torch_dtype=torch.bfloat16,
)
```
By default the LLM2Vec model uses the `mean` pooling strategy. You can change the pooling strategy by passing the `pooling_mode` argument to the `from_pretrained` method. Similarly, you can change the maximum sequence length by passing the `max_length` argument (default is 512).
### Inference
This model now returns the text embedding for any input in the form of `[[instruction1, text1], [instruction2, text2]]` or `[text1, text2]`. While training, we provide instructions for both sentences in symmetric tasks, and only for for queries in asymmetric tasks.
```python
# Encoding queries using instructions
instruction = (
"Given a web search query, retrieve relevant passages that answer the query:"
)
queries = [
[instruction, "how much protein should a female eat"],
[instruction, "summit define"],
]
q_reps = l2v.encode(queries)
# Encoding documents. Instruction are not required for documents
documents = [
"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.",
]
d_reps = l2v.encode(documents)
# Compute cosine similarity
q_reps_norm = torch.nn.functional.normalize(q_reps, p=2, dim=1)
d_reps_norm = torch.nn.functional.normalize(d_reps, p=2, dim=1)
cos_sim = torch.mm(q_reps_norm, d_reps_norm.transpose(0, 1))
print(cos_sim)
"""
tensor([[0.6470, 0.1619],
[0.0786, 0.5844]])
"""
```
More examples of classification, clustering, sentence similarity etc are present in [examples](examples) directory.
## Model List
| | Meta-Llama-3-8B | Mistral-7B | Llama-2-7B | Sheared-Llama-1.3B |
|----|------------------|------------|------------|---------------------|
| Bi + MNTP | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp) |
| Bi + MNTP + SimCSE | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-unsup-simcse)** | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-unsup-simcse) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-unsup-simcse) |
| Bi + MNTP + Supervised | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised)* | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp-supervised) |
\* State-of-the-art on MTEB among models trained on public data
\*\* Unsupervised state-of-the-art on MTEB
## Training
### MNTP training
To train the model with Masked Next Token Prediction (MNTP), you can use the `experiments/run_mntp.py` script. It is adapted from HuggingFace Masked Language Modeling (MLM) [script](https://github.com/huggingface/transformers/blob/51bcadc10a569847b93a30dbe3a077037ae63bad/examples/pytorch/language-modeling/run_mlm.py). To train the Meta-Llama-3-8B model with MNTP, run the following command:
```bash
python experiments/run_mntp.py train_configs/mntp/MetaLlama3.json
```
The Meta-Llama-3-8B training configuration [file](train_configs/mntp/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper.
```json
{
"model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
"dataset_name": "wikitext",
"dataset_config_name": "wikitext-103-raw-v1",
"mask_token_type": "blank",
"data_collator_type": "default",
"mlm_probability": 0.2,
"lora_r": 16,
"gradient_checkpointing": true,
"torch_dtype": "bfloat16",
"attn_implementation": "flash_attention_2"
// ....
}
```
Similar configurations are also available for[Mistral-7B](train_configs/mntp/Mistral.json), [Llama-2-7B](train_configs/mntp/Llama2.json), and [Sheared-Llama-1.3B](train_configs/mntp/Sheared-Llama.json) models.
### Unsupervised contrastive training (SimCSE)
For SimCSE training, we replicated the training procedure from [SimCSE](https://arxiv.org/abs/2104.08821) paper. For training, we use the dataset 1 million sentences from English Wikipedia released by the authors. It can be downloaded using the following command:
```bash
wget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt
```
To use the training script with pre-set configurations, the downloaded file should be placed in the `cache` directory. The directory layout should be as follows:
```
cache
└── wiki1m_for_simcse.txt
```
If the dataset is placed in a different directory, please change the dataset_file_path in the training configuration accordingly.
To train the Meta-Llama-3-8B model with SimCSE, run the following command:
```bash
python experiments/run_simcse.py train_configs/simcse/MetaLlama3.json
```
The Meta-Llama-3-8B training configuration [file](train_configs/simcse/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper.
```json
{
"model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
"peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
"simcse_dropout": 0.3,
"bidirectional": true,
"pooling_mode": "mean",
"dataset_name": "Wiki1M",
"dataset_file_path": "cache/wiki1m_for_simcse.txt",
"learning_rate": 3e-5,
"loss_scale": 20,
"per_device_train_batch_size": 128,
"max_seq_length": 128,
"stop_after_n_steps": 1000,
"lora_r": 16,
"gradient_checkpointing": true,
"torch_dtype": "bfloat16",
"attn_implementation": "flash_attention_2",
// ....
}
```
Similar configurations are also available for [Mistral](train_configs/simcse/Mistral.json), [Llama-2-7B](train_configs/simcse/Llama2.json), and [Sheared-Llama-1.3B](train_configs/simcse/Sheared-Llama.json) models.
### Supervised contrastive training
For supervised contrastive training, we use the public portion of dataset used in [Improving Text Embeddings with Large Language Models](https://arxiv.org/abs/2401.00368), curated by authors of [Repetition Improves Language Model Embeddings](https://arxiv.org/abs/2402.15449). The dataset can be downloaded from the [GitHub page of Echo embeddings repository](https://github.com/jakespringer/echo-embeddings#training). To use the training script, the downloaded dataset should be placed in the `cache` directory. The directory layout should be as follows:
```
cache
|── wiki1m_for_simcse.txt
└── echo-data
├── allnli_split1.jsonl
├── allnli_split2.jsonl
├── allnli.jsonl
├── dureader.jsonl
...
```
If the dataset is placed in a different directory, please change the `dataset_file_path` in the training configuration accordingly.
To train the Meta-Llama-3-8B model with supervised contrastive learning, run the following command:
```bash
torchrun --nproc_per_node=8 experiments/run_supervised.py train_configs/supervised/MetaLlama3.json
```
The number of GPUs can be changed by modifying the `--nproc_per_node` argument.
The Meta-Llama-3-8B training configuration [file](train_configs/supervised/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper.
```json
{
"model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
"peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
"bidirectional": true,
"pooling_mode": "mean",
"dataset_name": "E5",
"dataset_file_path": "cache/echo-data",
"learning_rate": 2e-4,
"num_train_epochs": 3,
"warmup_steps": 300,
"per_device_train_batch_size": 64,
"lora_r": 16,
"gradient_checkpointing": true,
"torch_dtype": "bfloat16",
"attn_implementation": "flash_attention_2"
// ....
}
```
Similar configurations are also available for [Mistral](train_configs/supervised/Mistral.json), [Llama-2-7B](train_configs/supervised/Llama2.json), and [Sheared-Llama-1.3B](train_configs/supervised/Sheared-Llama.json) models.
### Word-level tasks training
To tune the model for word-level tasks, we define a classifier on top of the models, and only train the classifier weights. The code is adapted from HuggingFace token classification [example](https://huggingface.co/docs/transformers/en/tasks/token_classification). To train and test the classifier for Llama-2-7B MNTP model on `pos_tags` task, run the following command:
```bash
python experiments/run_word_task.py train_configs/word-task/Llama2-bi-mntp.json
python experiments/test_word_task.py --config_file test_configs/word-task/Llama2-bi-mntp.json
```
The config files contain all the parameters and configurations used in our paper. For instance, `Llama2-bi-mntp.json` includes:
```json
{
"model_name_or_path": "meta-llama/Llama-2-7b-chat-hf",
"peft_addr": "McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp", // or any local directory containing `adapter_model` files.
"model_class": "custom",
"bidirectional": true,
"classifier_dropout": 0.1,
"merge_subwords": true,
"retroactive_labels": "next_token",
"output_dir": "output/word-task/pos_tags/Llama2/bi-mntp",
"dataset_name": "conll2003",
"task": "pos_tags", // or ner_tags, or chunk_tags
// ....
}
```
[train_configs/word-task](train_configs/word-task) and [test_configs/word-task](train_configs/word-task) contain similar configurations for Llama-2-7B, Mistral-7B, and Sheared-Llama-1.3B for all Uni, Bi, Bi-MNTP, and Bi-MNTP-SimCSE (LLM2Vec) variants.
## Evaluation
### MTEB Evaluation
To evaluate the model on the MTEB benchmark, we use the `experiments/mteb_eval.py` script. The script requires `mteb>=1.12.60`, amongst other dependencies, which can be installed with the following command.
```bash
pip install llm2vec[evaluation]
```
The evaluation utilizes instructions for each task which are provided in the `test_configs/mteb/task_to_instructions.json` file.
To evaluate the supervised trained Meta-Llama-3-8B model on the `STS16` task, run the following command:
```bash
python experiments/mteb_eval.py --model_name McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised \
--task_name STS16 \
--task_to_instructions_fp test_configs/mteb/task_to_instructions.json \
--output_dir results
```
The evaluation script supports all the models available in the [HuggingFace collection](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34).
## Citation
If you find our work helpful, please cite us:
```bibtex
@article{llm2vec,
title={{LLM2Vec}: {L}arge Language Models Are Secretly Powerful Text Encoders},
author={Parishad BehnamGhader and Vaibhav Adlakha and Marius Mosbach and Dzmitry Bahdanau and Nicolas Chapados and Siva Reddy},
year={2024},
journal={arXiv preprint},
url={https://arxiv.org/abs/2404.05961}
}
```
## Bugs or questions?
If you have any questions about the code, feel free to open an issue on the GitHub repository.
Raw data
{
"_id": null,
"home_page": "https://github.com/McGill-NLP/llm2vec",
"name": "llm2vec",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "McGill NLP",
"author_email": "parishad.behnamghader@mila.quebec",
"download_url": "https://files.pythonhosted.org/packages/64/8a/a4101e6d1c3284d7c0ea4d5791a3e28de8eba69ecaf98e48fd791d940074/llm2vec-0.2.2.tar.gz",
"platform": null,
"description": "\n# *LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders* \n\n\n[](https://arxiv.org/abs/2404.05961)\n[](https://pypi.org/project/llm2vec/)\n[](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34)\n[](https://github.com/McGill-NLP/llm2vec/blob/main/LICENSE)\n[](https://pepy.tech/project/llm2vec)\n\nLLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) training with masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.\n\n<p align=\"center\">\n <img src=\"https://github.com/McGill-NLP/llm2vec/assets/12207571/48efd48a-431b-4625-8e0f-248a442e3839\" width=\"75%\" alt=\"LLM2Vec_figure1\"/>\n</p>\n\n**************************** **Updates** ****************************\n\n* 04/07: Added support for Gemma and Qwen-2 models, huge thanks to [@bzantium](https://github.com/bzantium) for the contribution.\n\n* 30/04: We release LLM2Vec transformed Meta-Llama-3 checkpoints. See our [HuggingFace collection](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34) for both [supervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised) and [unsupervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse) variants.\n\n## Installation\nTo use LLM2Vec, first install the llm2vec package from PyPI, followed by installing flash-attention:\n\n```bash\npip install llm2vec\npip install flash-attn --no-build-isolation\n```\nYou can also directly install the latest version of llm2vec by cloning the repository: \n\n```bash\npip install -e .\npip install flash-attn --no-build-isolation\n```\n\n## Getting Started\nLLM2Vec class is a wrapper on top of HuggingFace models to support enabling bidirectionality in decoder-only LLMs, sequence encoding and pooling operations. The steps below showcase an example on how to use the library.\n\n### Preparing the model\nInitializing LLM2Vec model using pretrained LLMs is straightforward. The `from_pretrained` method of LLM2Vec takes a base model identifier/path and an optional PEFT model identifier/path. All HuggingFace model loading arguments can be passed to `from_pretrained` method. By default, the models are loaded with bidirectional connections enabled. This can be turned off by passing `enable_bidirectional=False` to the `from_pretrained` method.\n\nHere, we first initialize the Llama-3 MNTP base model and load the unsupervised-trained LoRA weights (trained with SimCSE objective and wiki corpus).\n\n```python\nimport torch\nfrom llm2vec import LLM2Vec\n\nl2v = LLM2Vec.from_pretrained(\n \"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp\",\n peft_model_name_or_path=\"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse\",\n device_map=\"cuda\" if torch.cuda.is_available() else \"cpu\",\n torch_dtype=torch.bfloat16,\n)\n```\n\nWe can also load the model with supervised-trained LoRA weights (trained with contrastive learning and public E5 data) by changing the `peft_model_name_or_path`.\n\n```python\nimport torch\nfrom llm2vec import LLM2Vec\n\nl2v = LLM2Vec.from_pretrained(\n \"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp\",\n peft_model_name_or_path=\"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised\",\n device_map=\"cuda\" if torch.cuda.is_available() else \"cpu\",\n torch_dtype=torch.bfloat16,\n)\n```\n\nBy default the LLM2Vec model uses the `mean` pooling strategy. You can change the pooling strategy by passing the `pooling_mode` argument to the `from_pretrained` method. Similarly, you can change the maximum sequence length by passing the `max_length` argument (default is 512).\n\n### Inference\nThis model now returns the text embedding for any input in the form of `[[instruction1, text1], [instruction2, text2]]` or `[text1, text2]`. While training, we provide instructions for both sentences in symmetric tasks, and only for for queries in asymmetric tasks.\n\n```python\n# Encoding queries using instructions\ninstruction = (\n \"Given a web search query, retrieve relevant passages that answer the query:\"\n)\nqueries = [\n [instruction, \"how much protein should a female eat\"],\n [instruction, \"summit define\"],\n]\nq_reps = l2v.encode(queries)\n\n# Encoding documents. Instruction are not required for documents\ndocuments = [\n \"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.\",\n \"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.\",\n]\nd_reps = l2v.encode(documents)\n\n# Compute cosine similarity\nq_reps_norm = torch.nn.functional.normalize(q_reps, p=2, dim=1)\nd_reps_norm = torch.nn.functional.normalize(d_reps, p=2, dim=1)\ncos_sim = torch.mm(q_reps_norm, d_reps_norm.transpose(0, 1))\n\nprint(cos_sim)\n\"\"\"\ntensor([[0.6470, 0.1619],\n [0.0786, 0.5844]])\n\"\"\"\n```\n\nMore examples of classification, clustering, sentence similarity etc are present in [examples](examples) directory.\n\n## Model List\n\n| | Meta-Llama-3-8B | Mistral-7B | Llama-2-7B | Sheared-Llama-1.3B |\n|----|------------------|------------|------------|---------------------|\n| Bi + MNTP | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp) |\n| Bi + MNTP + SimCSE | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-unsup-simcse)** | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-unsup-simcse) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-unsup-simcse) |\n| Bi + MNTP + Supervised | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised)* | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised) | [HF Link](https://huggingface.co/McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp-supervised) |\n\n\\* State-of-the-art on MTEB among models trained on public data\n\n\\*\\* Unsupervised state-of-the-art on MTEB\n\n## Training \n### MNTP training\nTo train the model with Masked Next Token Prediction (MNTP), you can use the `experiments/run_mntp.py` script. It is adapted from HuggingFace Masked Language Modeling (MLM) [script](https://github.com/huggingface/transformers/blob/51bcadc10a569847b93a30dbe3a077037ae63bad/examples/pytorch/language-modeling/run_mlm.py). To train the Meta-Llama-3-8B model with MNTP, run the following command:\n\n```bash\npython experiments/run_mntp.py train_configs/mntp/MetaLlama3.json\n```\n\nThe Meta-Llama-3-8B training configuration [file](train_configs/mntp/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper. \n```json\n{\n \"model_name_or_path\": \"meta-llama/Meta-Llama-3-8B-Instruct\",\n \"dataset_name\": \"wikitext\",\n \"dataset_config_name\": \"wikitext-103-raw-v1\",\n \"mask_token_type\": \"blank\",\n \"data_collator_type\": \"default\",\n \"mlm_probability\": 0.2,\n \"lora_r\": 16,\n \"gradient_checkpointing\": true,\n \"torch_dtype\": \"bfloat16\",\n \"attn_implementation\": \"flash_attention_2\"\n // ....\n}\n```\n\nSimilar configurations are also available for[Mistral-7B](train_configs/mntp/Mistral.json), [Llama-2-7B](train_configs/mntp/Llama2.json), and [Sheared-Llama-1.3B](train_configs/mntp/Sheared-Llama.json) models.\n\n### Unsupervised contrastive training (SimCSE)\nFor SimCSE training, we replicated the training procedure from [SimCSE](https://arxiv.org/abs/2104.08821) paper. For training, we use the dataset 1 million sentences from English Wikipedia released by the authors. It can be downloaded using the following command:\n\n```bash\nwget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt\n```\n\nTo use the training script with pre-set configurations, the downloaded file should be placed in the `cache` directory. The directory layout should be as follows:\n\n```\ncache\n\u2514\u2500\u2500 wiki1m_for_simcse.txt\n```\nIf the dataset is placed in a different directory, please change the dataset_file_path in the training configuration accordingly.\n\nTo train the Meta-Llama-3-8B model with SimCSE, run the following command:\n\n```bash\npython experiments/run_simcse.py train_configs/simcse/MetaLlama3.json\n```\n\nThe Meta-Llama-3-8B training configuration [file](train_configs/simcse/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper. \n```json\n{\n \"model_name_or_path\": \"meta-llama/Meta-Llama-3-8B-Instruct\",\n \"peft_model_name_or_path\": \"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp\",\n \"simcse_dropout\": 0.3,\n \"bidirectional\": true,\n \"pooling_mode\": \"mean\",\n \"dataset_name\": \"Wiki1M\",\n \"dataset_file_path\": \"cache/wiki1m_for_simcse.txt\",\n \"learning_rate\": 3e-5,\n \"loss_scale\": 20,\n \"per_device_train_batch_size\": 128,\n \"max_seq_length\": 128,\n \"stop_after_n_steps\": 1000,\n \"lora_r\": 16,\n \"gradient_checkpointing\": true,\n \"torch_dtype\": \"bfloat16\",\n \"attn_implementation\": \"flash_attention_2\",\n // ....\n}\n```\n\nSimilar configurations are also available for [Mistral](train_configs/simcse/Mistral.json), [Llama-2-7B](train_configs/simcse/Llama2.json), and [Sheared-Llama-1.3B](train_configs/simcse/Sheared-Llama.json) models.\n\n### Supervised contrastive training\nFor supervised contrastive training, we use the public portion of dataset used in [Improving Text Embeddings with Large Language Models](https://arxiv.org/abs/2401.00368), curated by authors of [Repetition Improves Language Model Embeddings](https://arxiv.org/abs/2402.15449). The dataset can be downloaded from the [GitHub page of Echo embeddings repository](https://github.com/jakespringer/echo-embeddings#training). To use the training script, the downloaded dataset should be placed in the `cache` directory. The directory layout should be as follows:\n\n```\ncache\n|\u2500\u2500 wiki1m_for_simcse.txt\n\u2514\u2500\u2500 echo-data\n \u251c\u2500\u2500 allnli_split1.jsonl\n \u251c\u2500\u2500 allnli_split2.jsonl\n \u251c\u2500\u2500 allnli.jsonl\n \u251c\u2500\u2500 dureader.jsonl\n ...\n```\nIf the dataset is placed in a different directory, please change the `dataset_file_path` in the training configuration accordingly. \n\nTo train the Meta-Llama-3-8B model with supervised contrastive learning, run the following command:\n\n```bash\ntorchrun --nproc_per_node=8 experiments/run_supervised.py train_configs/supervised/MetaLlama3.json\n```\nThe number of GPUs can be changed by modifying the `--nproc_per_node` argument.\n\nThe Meta-Llama-3-8B training configuration [file](train_configs/supervised/MetaLlama3.json) contains all the training hyperparameters and configurations used in our paper. \n```json\n{\n \"model_name_or_path\": \"meta-llama/Meta-Llama-3-8B-Instruct\",\n \"peft_model_name_or_path\": \"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp\",\n \"bidirectional\": true,\n \"pooling_mode\": \"mean\",\n \"dataset_name\": \"E5\",\n \"dataset_file_path\": \"cache/echo-data\",\n \"learning_rate\": 2e-4,\n \"num_train_epochs\": 3,\n \"warmup_steps\": 300,\n \"per_device_train_batch_size\": 64,\n \"lora_r\": 16,\n \"gradient_checkpointing\": true,\n \"torch_dtype\": \"bfloat16\",\n \"attn_implementation\": \"flash_attention_2\"\n // ....\n}\n```\nSimilar configurations are also available for [Mistral](train_configs/supervised/Mistral.json), [Llama-2-7B](train_configs/supervised/Llama2.json), and [Sheared-Llama-1.3B](train_configs/supervised/Sheared-Llama.json) models.\n\n\n### Word-level tasks training\n\nTo tune the model for word-level tasks, we define a classifier on top of the models, and only train the classifier weights. The code is adapted from HuggingFace token classification [example](https://huggingface.co/docs/transformers/en/tasks/token_classification). To train and test the classifier for Llama-2-7B MNTP model on `pos_tags` task, run the following command:\n```bash\npython experiments/run_word_task.py train_configs/word-task/Llama2-bi-mntp.json\npython experiments/test_word_task.py --config_file test_configs/word-task/Llama2-bi-mntp.json\n```\nThe config files contain all the parameters and configurations used in our paper. For instance, `Llama2-bi-mntp.json` includes:\n```json\n{\n \"model_name_or_path\": \"meta-llama/Llama-2-7b-chat-hf\",\n \"peft_addr\": \"McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp\", // or any local directory containing `adapter_model` files.\n \"model_class\": \"custom\",\n \"bidirectional\": true,\n \"classifier_dropout\": 0.1,\n \"merge_subwords\": true,\n \"retroactive_labels\": \"next_token\",\n \"output_dir\": \"output/word-task/pos_tags/Llama2/bi-mntp\",\n \"dataset_name\": \"conll2003\",\n \"task\": \"pos_tags\", // or ner_tags, or chunk_tags\n // ....\n}\n``` \n[train_configs/word-task](train_configs/word-task) and [test_configs/word-task](train_configs/word-task) contain similar configurations for Llama-2-7B, Mistral-7B, and Sheared-Llama-1.3B for all Uni, Bi, Bi-MNTP, and Bi-MNTP-SimCSE (LLM2Vec) variants. \n\n\n## Evaluation\n\n### MTEB Evaluation\nTo evaluate the model on the MTEB benchmark, we use the `experiments/mteb_eval.py` script. The script requires `mteb>=1.12.60`, amongst other dependencies, which can be installed with the following command.\n```bash\npip install llm2vec[evaluation]\n```\n\nThe evaluation utilizes instructions for each task which are provided in the `test_configs/mteb/task_to_instructions.json` file. \n\nTo evaluate the supervised trained Meta-Llama-3-8B model on the `STS16` task, run the following command:\n```bash\npython experiments/mteb_eval.py --model_name McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised \\\n--task_name STS16 \\\n--task_to_instructions_fp test_configs/mteb/task_to_instructions.json \\\n--output_dir results\n```\n\nThe evaluation script supports all the models available in the [HuggingFace collection](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34).\n\n\n## Citation\nIf you find our work helpful, please cite us:\n```bibtex\n@article{llm2vec,\n title={{LLM2Vec}: {L}arge Language Models Are Secretly Powerful Text Encoders}, \n author={Parishad BehnamGhader and Vaibhav Adlakha and Marius Mosbach and Dzmitry Bahdanau and Nicolas Chapados and Siva Reddy},\n year={2024},\n journal={arXiv preprint},\n url={https://arxiv.org/abs/2404.05961}\n}\n```\n\n## Bugs or questions?\nIf you have any questions about the code, feel free to open an issue on the GitHub repository.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "The official llm2vec library",
"version": "0.2.2",
"project_urls": {
"Homepage": "https://github.com/McGill-NLP/llm2vec"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6d3c595ebbd768cbab3e6d62642563bbd265b4ede2d6457fbdcd65e18424115c",
"md5": "8525dbf69627fea848a730fcf3f8e8eb",
"sha256": "86002770fed5e77b731ddd93962ec4cff58e6b21c69ebe4f536491019535e656"
},
"downloads": -1,
"filename": "llm2vec-0.2.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "8525dbf69627fea848a730fcf3f8e8eb",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 30521,
"upload_time": "2024-07-30T12:24:38",
"upload_time_iso_8601": "2024-07-30T12:24:38.756455Z",
"url": "https://files.pythonhosted.org/packages/6d/3c/595ebbd768cbab3e6d62642563bbd265b4ede2d6457fbdcd65e18424115c/llm2vec-0.2.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "648aa4101e6d1c3284d7c0ea4d5791a3e28de8eba69ecaf98e48fd791d940074",
"md5": "de5a5cf6a1464f8b1eebe4ae7d0d63e2",
"sha256": "e8fa7e006c7d3638d9bd2f5a5aef2740372f97d8639aa30eae283d2247337d72"
},
"downloads": -1,
"filename": "llm2vec-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "de5a5cf6a1464f8b1eebe4ae7d0d63e2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 28200,
"upload_time": "2024-07-30T12:24:40",
"upload_time_iso_8601": "2024-07-30T12:24:40.796155Z",
"url": "https://files.pythonhosted.org/packages/64/8a/a4101e6d1c3284d7c0ea4d5791a3e28de8eba69ecaf98e48fd791d940074/llm2vec-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-30 12:24:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "McGill-NLP",
"github_project": "llm2vec",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "llm2vec"
}
JSON
