<!--- BADGES: START, copied from sentence transformers, will be replaced with the actual once (removed for anonymity)--->
[][#github-license]
[][#pypi-package]
[][#pypi-package]
[#github-license]: https://github.com/
[#pypi-package]: https://pypi.org/project/triple-encoders/
<p align="center">
<img align="center" src="img/triple-encoder-logo_with_border.png" width="430px" />
</p>
<p align="center">
🤗 <a href="anonymous" target="_blank">Models</a> | 📊 <a href="anonymous" target="_blank">Datasets</a> | 📃 <a href="anonymous" target="_blank">Paper</a>
</p>
`triple-encoders` is a library for contextualizing distributed [Sentence Transformers](https://sbert.net/) representations.
At inference, triple encoders can be used for retrieval-based sequence modeling via sequential modular late-interaction:
<p align="center">
<img align="center" src="img/triple-encoder.jpg" width="1000px" />
</p>
Representations are encoded **separately** and the contextualization is **weightless**:
1. *mean-pooling* to pairwise contextualize sentence representations (creates a distributed query)
2. *cosine similarity* to measure the similarity between all query vectors and the retrieval candidates.
3. *summation* to aggregate the similarity (similar to average-based late interaction of [ColBERT](https://github.com/stanford-futuredata/ColBERT)).
## Key Features
- 1️⃣ **One dense vector vs distributed dense vectors**: in our paper we demonstrate that our late interaction-based approach outperforms single-vector representations on long sequences, including zero-shot settings.
- 🏎️💨 **Relative compute**: as every representation is encoded separately, you only need to encode, compute mixtures and similarities for the latest added representation (in dialog: the latest utterance).
- 📚 **No Limit on context-length**: our distributed sentence transformer architecture is not limited to any sequence length. You can use your entire sequence as query!
- 🌎 **Multilingual support**: `triple-encoders` can be used with any [Sentence Transformers](https://sbert.net/) model. This means that you can model multilingual sequences by simply training on a multilingual model checkpoint.
## Installation
You can install `triple-encoders` via pip:
```bash
pip install triple-encoders
```
Note that `triple-encoders` requires Python 3.6 or higher.
# Getting Started
Our experiments for sequence modeling and short-term planning conducted in the paper can be found in the `notebooks` folder. The hyperparameter that we used for training are the default parameters in the `trainer.py` file.
## Retrieval-based Sequence Modeling
We provide an example of how to use triple-encoders for conversational sequence modeling (response selection) with 2 dialog speakers. If you want to use triple-encoders for other sequence modeling tasks, you can use the `TripleEncodersForSequenceModeling` class.
### Loading the model
```python
from triple_encoders.TripleEncodersForConversationalSequenceModeling import TripleEncodersForConversationalSequenceModeling
triple_path = ''
# load model
model = TripleEncodersForConversationalSequenceModeling(triple_path)
```
### Inference
```python
# load candidates for response selection
candidates = ['I am doing great too!','Where did you go?', 'ACL is an interesting conference']
# load candidates and store index
model.load_candidates_from_strings(candidates, output_directory_candidates_dump='output/path/to/save/candidates')
# create a sequence
sequence = model.contextualize_sequence(["Hi!",'Hey, how are you?'], k_last_rows=2)
# model sequence (compute scores for candidates)
sequence = model.sequence_modeling(sequence)
# retrieve utterance from dialog partner
new_utterance = "I'm fine, thanks. How are you?"
# pass it to the model with dialog_partner=True
sequence = model.contextualize_utterance(new_utterance, sequence, dialog_partner=True)
# model sequence (compute scores for candidates)
sequence = model.sequence_modeling(sequence)
# retrieve candidates to provide a response
response = model.retrieve_candidates(sequence, 3)
response
#(['I am doing great too!','Where did you go?', 'ACL is an interesting conference'],
# tensor([0.4944, 0.2392, 0.0483]))
```
**Speed:**
- Time to load candidates: 31.815 ms
- Time to contextualize sequence: 18.078 ms
- Time to model sequence: 0.256 ms
- Time to contextualize new utterance: 15.858 ms
- Time to model new utterance: 0.213 ms
- Time to retrieve candidates: 0.093 ms
### Evaluation
```python
from datasets import load_dataset
dataset = load_dataset("daily_dialog")
test = dataset['test']['dialog']
df = model.evaluate_seq_dataset(test, k_last_rows=2)
df
# pandas dataframe with the average rank for each history length
```
## Short-Term Planning (STP)
Short-term planning enables you to re-rank candidate replies from LLMs to reach a goal utterance over multiple turns.
### Inference
```python
from triple_encoders.TripleEncodersForSTP import TripleEncodersForSTP
model = TripleEncodersForSTP(triple_path)
context = ['Hey, how are you ?',
'I am good, how about you ?',
'I am good too.']
candidates = ['Want to eat something out ?',
'Want to go for a walk ?']
goal = ' I am hungry.'
result = model.short_term_planning(candidates, goal, context)
result
# 'Want to eat something out ?'
```
### Evaluation
```python
from datasets import load_dataset
from triple_encoders.TripleEncodersForSTP import TripleEncodersForSTP
dataset = load_dataset("daily_dialog")
test = dataset['test']['dialog']
model = TripleEncodersForSTP(triple_path, llm_model_name_or_path='your favorite large language model')
df = model.evaluate_stp_dataset(test)
# pandas dataframe with the average rank and Hits@k for each history length, goal_distance
```
# Training Triple Encoders
You can train your own triple encoders with Contextualized Curved Contrastive Learning (C3L) using our trainer.
The hyperparameters that we used for training are the default parameters in the `trainer.py` file.
Note that we pre-trained our best model with Curved Contrastive Learning (CCL) (from [imaginaryNLP](https://github.com/Justus-Jonas/imaginaryNLP)) before training with C3L.
```python
from triple_encoders.trainer import TripleEncoderTrainer
from datasets import load_dataset
dataset = load_dataset("daily_dialog")
trainer = TripleEncoderTrainer(base_model_name_or_path=,
batch_size=48,
observation_window=5,
speaker_token=True, # used for conversational sequence modeling
num_epochs=3,
warmup_steps=10000)
trainer.generate_datasets(
dataset["train"]["dialog"],
dataset["validation"]["dialog"],
dataset["test"]["dialog"],
)
trainer.train("output/path/to/save/model")
```
## Citation
If you use triple-encoders in your research, please cite the following paper:
```
% todo
@article{anonymous,
title={Triple Encoders: Represenations That Fire Together, Wire Together},
author={Justus-Jonas Erker, Florian Mai, Nils Reimers, Gerasimos Spanakis, Iryna Gurevych},
journal={axiv},
year={2024}
}
```
# Contact
Contact person: Justus-Jonas Erker, justus-jonas.erker@tu-darmstadt.de
https://www.ukp.tu-darmstadt.de/
https://www.tu-darmstadt.de/
Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
# License
triple-encoders is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text.
### Acknowledgement
this package is based upon the [imaginaryNLP](https://github.com/Justus-Jonas/imaginaryNLP) and [Sentence Transformers](https://sbert.net/).
Raw data
{
"_id": null,
"home_page": "",
"name": "triple-encoders",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": "",
"keywords": "PyTorch NLP deep learning Sentence Transformer Triple Encoders Dialog Systems",
"author": "Justus-Jonas Erker",
"author_email": "justus-jonas.erker@tu-darmstadt.de",
"download_url": "https://files.pythonhosted.org/packages/39/ab/57a13b2b689689852354c1c7de76238ef0b0616b56e8dc4d8e7d8b686d9e/triple-encoders-0.0.2.tar.gz",
"platform": null,
"description": "<!--- BADGES: START, copied from sentence transformers, will be replaced with the actual once (removed for anonymity)--->\r\n[][#github-license]\r\n[][#pypi-package]\r\n[][#pypi-package]\r\n\r\n\r\n[#github-license]: https://github.com/\r\n[#pypi-package]: https://pypi.org/project/triple-encoders/\r\n\r\n<p align=\"center\">\r\n <img align=\"center\" src=\"img/triple-encoder-logo_with_border.png\" width=\"430px\" />\r\n</p>\r\n\r\n<p align=\"center\">\r\n \ud83e\udd17 <a href=\"anonymous\" target=\"_blank\">Models</a> | \ud83d\udcca <a href=\"anonymous\" target=\"_blank\">Datasets</a> | \ud83d\udcc3 <a href=\"anonymous\" target=\"_blank\">Paper</a>\r\n</p>\r\n\r\n`triple-encoders` is a library for contextualizing distributed [Sentence Transformers](https://sbert.net/) representations. \r\nAt inference, triple encoders can be used for retrieval-based sequence modeling via sequential modular late-interaction: \r\n<p align=\"center\">\r\n <img align=\"center\" src=\"img/triple-encoder.jpg\" width=\"1000px\" />\r\n</p>\r\n\r\nRepresentations are encoded **separately** and the contextualization is **weightless**:\r\n1. *mean-pooling* to pairwise contextualize sentence representations (creates a distributed query)\r\n2. *cosine similarity* to measure the similarity between all query vectors and the retrieval candidates.\r\n3. *summation* to aggregate the similarity (similar to average-based late interaction of [ColBERT](https://github.com/stanford-futuredata/ColBERT)).\r\n\r\n## Key Features\r\n- 1\ufe0f\u20e3 **One dense vector vs distributed dense vectors**: in our paper we demonstrate that our late interaction-based approach outperforms single-vector representations on long sequences, including zero-shot settings.\r\n- \ud83c\udfce\ufe0f\ud83d\udca8 **Relative compute**: as every representation is encoded separately, you only need to encode, compute mixtures and similarities for the latest added representation (in dialog: the latest utterance). \r\n- \ud83d\udcda **No Limit on context-length**: our distributed sentence transformer architecture is not limited to any sequence length. You can use your entire sequence as query!\r\n- \ud83c\udf0e **Multilingual support**: `triple-encoders` can be used with any [Sentence Transformers](https://sbert.net/) model. This means that you can model multilingual sequences by simply training on a multilingual model checkpoint. \r\n\r\n\r\n## Installation\r\nYou can install `triple-encoders` via pip:\r\n```bash\r\npip install triple-encoders\r\n``` \r\nNote that `triple-encoders` requires Python 3.6 or higher.\r\n\r\n# Getting Started\r\n\r\nOur experiments for sequence modeling and short-term planning conducted in the paper can be found in the `notebooks` folder. The hyperparameter that we used for training are the default parameters in the `trainer.py` file.\r\n\r\n## Retrieval-based Sequence Modeling\r\nWe provide an example of how to use triple-encoders for conversational sequence modeling (response selection) with 2 dialog speakers. If you want to use triple-encoders for other sequence modeling tasks, you can use the `TripleEncodersForSequenceModeling` class.\r\n\r\n### Loading the model\r\n```python\r\nfrom triple_encoders.TripleEncodersForConversationalSequenceModeling import TripleEncodersForConversationalSequenceModeling\r\n\r\ntriple_path = ''\r\n\r\n# load model\r\nmodel = TripleEncodersForConversationalSequenceModeling(triple_path)\r\n```\r\n\r\n### Inference\r\n\r\n```python\r\n# load candidates for response selection \r\ncandidates = ['I am doing great too!','Where did you go?', 'ACL is an interesting conference']\r\n\r\n# load candidates and store index\r\nmodel.load_candidates_from_strings(candidates, output_directory_candidates_dump='output/path/to/save/candidates')\r\n\r\n# create a sequence\r\nsequence = model.contextualize_sequence([\"Hi!\",'Hey, how are you?'], k_last_rows=2)\r\n\r\n# model sequence (compute scores for candidates)\r\nsequence = model.sequence_modeling(sequence)\r\n\r\n# retrieve utterance from dialog partner\r\nnew_utterance = \"I'm fine, thanks. How are you?\"\r\n\r\n# pass it to the model with dialog_partner=True \r\nsequence = model.contextualize_utterance(new_utterance, sequence, dialog_partner=True)\r\n\r\n# model sequence (compute scores for candidates)\r\nsequence = model.sequence_modeling(sequence)\r\n\r\n# retrieve candidates to provide a response\r\nresponse = model.retrieve_candidates(sequence, 3)\r\nresponse\r\n#(['I am doing great too!','Where did you go?', 'ACL is an interesting conference'],\r\n# tensor([0.4944, 0.2392, 0.0483]))\r\n```\r\n**Speed:** \r\n- Time to load candidates: 31.815 ms\r\n- Time to contextualize sequence: 18.078 ms\r\n- Time to model sequence: 0.256 ms\r\n- Time to contextualize new utterance: 15.858 ms\r\n- Time to model new utterance: 0.213 ms\r\n- Time to retrieve candidates: 0.093 ms\r\n\r\n### Evaluation\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"daily_dialog\")\r\ntest = dataset['test']['dialog']\r\n\r\ndf = model.evaluate_seq_dataset(test, k_last_rows=2)\r\ndf\r\n# pandas dataframe with the average rank for each history length\r\n```\r\n\r\n## Short-Term Planning (STP)\r\nShort-term planning enables you to re-rank candidate replies from LLMs to reach a goal utterance over multiple turns.\r\n\r\n### Inference\r\n\r\n```python\r\nfrom triple_encoders.TripleEncodersForSTP import TripleEncodersForSTP\r\n\r\nmodel = TripleEncodersForSTP(triple_path)\r\n\r\ncontext = ['Hey, how are you ?',\r\n 'I am good, how about you ?',\r\n 'I am good too.']\r\n\r\ncandidates = ['Want to eat something out ?',\r\n 'Want to go for a walk ?']\r\n\r\ngoal = ' I am hungry.'\r\n\r\nresult = model.short_term_planning(candidates, goal, context)\r\n\r\nresult\r\n# 'Want to eat something out ?'\r\n```\r\n### Evaluation\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom triple_encoders.TripleEncodersForSTP import TripleEncodersForSTP\r\n\r\ndataset = load_dataset(\"daily_dialog\")\r\ntest = dataset['test']['dialog']\r\n\r\nmodel = TripleEncodersForSTP(triple_path, llm_model_name_or_path='your favorite large language model')\r\n\r\ndf = model.evaluate_stp_dataset(test)\r\n# pandas dataframe with the average rank and Hits@k for each history length, goal_distance\r\n```\r\n\r\n\r\n# Training Triple Encoders\r\nYou can train your own triple encoders with Contextualized Curved Contrastive Learning (C3L) using our trainer. \r\nThe hyperparameters that we used for training are the default parameters in the `trainer.py` file. \r\nNote that we pre-trained our best model with Curved Contrastive Learning (CCL) (from [imaginaryNLP](https://github.com/Justus-Jonas/imaginaryNLP)) before training with C3L.\r\n\r\n```python\r\nfrom triple_encoders.trainer import TripleEncoderTrainer\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"daily_dialog\")\r\n\r\ntrainer = TripleEncoderTrainer(base_model_name_or_path=,\r\n batch_size=48,\r\n observation_window=5,\r\n speaker_token=True, # used for conversational sequence modeling\r\n num_epochs=3,\r\n warmup_steps=10000)\r\n\r\ntrainer.generate_datasets(\r\n dataset[\"train\"][\"dialog\"],\r\n dataset[\"validation\"][\"dialog\"],\r\n dataset[\"test\"][\"dialog\"],\r\n)\r\n\r\n\r\ntrainer.train(\"output/path/to/save/model\")\r\n```\r\n## Citation\r\nIf you use triple-encoders in your research, please cite the following paper:\r\n```\r\n% todo\r\n@article{anonymous,\r\n title={Triple Encoders: Represenations That Fire Together, Wire Together},\r\n author={Justus-Jonas Erker, Florian Mai, Nils Reimers, Gerasimos Spanakis, Iryna Gurevych},\r\n journal={axiv},\r\n year={2024}\r\n}\r\n```\r\n# Contact\r\nContact person: Justus-Jonas Erker, justus-jonas.erker@tu-darmstadt.de\r\n\r\nhttps://www.ukp.tu-darmstadt.de/\r\n\r\nhttps://www.tu-darmstadt.de/\r\n\r\nDon't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.\r\nThis repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication. \r\n\r\n# License\r\ntriple-encoders is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text.\r\n\r\n\r\n### Acknowledgement\r\nthis package is based upon the [imaginaryNLP](https://github.com/Justus-Jonas/imaginaryNLP) and [Sentence Transformers](https://sbert.net/).\r\n\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Distributed Sentence Transformer Representations with Triple Encoders",
"version": "0.0.2",
"project_urls": {
"Download": "https://github.com/UKPLab/arxiv2024-triple-encoders"
},
"split_keywords": [
"pytorch",
"nlp",
"deep",
"learning",
"sentence",
"transformer",
"triple",
"encoders",
"dialog",
"systems"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b1f36a44041b3a5e212f66276193b089a6adb6d2ac34bb8101d7d35f3cb947d9",
"md5": "71e33d60c3dbe90e73092fb59d951d01",
"sha256": "5629d98b6be79a6f37301593b72c8179eef2cc1699679aa5d68a3458a7a9ceee"
},
"downloads": -1,
"filename": "triple_encoders-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "71e33d60c3dbe90e73092fb59d951d01",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 29647,
"upload_time": "2024-02-19T16:18:29",
"upload_time_iso_8601": "2024-02-19T16:18:29.600009Z",
"url": "https://files.pythonhosted.org/packages/b1/f3/6a44041b3a5e212f66276193b089a6adb6d2ac34bb8101d7d35f3cb947d9/triple_encoders-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "39ab57a13b2b689689852354c1c7de76238ef0b0616b56e8dc4d8e7d8b686d9e",
"md5": "e27bc98daed0f09033afc5901b7daecd",
"sha256": "b48ea1bf4ef08e56fb8529b7797caab859c1590d59ce0913c5d80d6ec1bc4ae3"
},
"downloads": -1,
"filename": "triple-encoders-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "e27bc98daed0f09033afc5901b7daecd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 28549,
"upload_time": "2024-02-19T16:18:32",
"upload_time_iso_8601": "2024-02-19T16:18:32.699273Z",
"url": "https://files.pythonhosted.org/packages/39/ab/57a13b2b689689852354c1c7de76238ef0b0616b56e8dc4d8e7d8b686d9e/triple-encoders-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-19 16:18:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "UKPLab",
"github_project": "arxiv2024-triple-encoders",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "triple-encoders"
}
JSON
