# llm-toys
[](https://github.com/psf/black)
[](https://pypi.org/project/llm-toys/)
Small(7B and below), production-ready finetuned LLMs for a diverse set of useful tasks.
Supported tasks: Paraphrasing, Changing the tone of a passage, Summary and Topic generation from a dailogue,
~~Retrieval augmented QA(WIP)~~.
We finetune LoRAs on quantized 3B and 7B models. The 3B model is finetuned on specific tasks, while the 7B model is
finetuned on all the tasks.
The goal is to be able to finetune and use all these models on a very modest consumer grade hardware.
## Installation
```bash
pip install llm-toys
```
> Note that we are using the transformers and peft packages from the source directory,
> not the installed package. 4bit bitsandbytes quantization was only working with the
> main brach of transformers and peft. Once transformers version 4.31.0 and peft version 0.4.0 is
> published to pypi we will use the published version.
## Usage
### Task specific 3B models
#### Paraphrasing
```python
from llm_toys.tasks import Paraphraser
paraphraser = Paraphraser()
paraphraser.paraphrase("Hey, can yuo hepl me cancel my last order?")
# "Could you kindly assist me in canceling my previous order?"
```
#### Tone change
```python
paraphraser.paraphrase("Hey, can yuo hepl me cancel my last order?", tone="casual")
# "Hey, could you help me cancel my order?"
paraphraser.paraphrase("Hey, can yuo hepl me cancel my last order?", tone="professional")
# "I would appreciate guidance on canceling my previous order."
paraphraser.paraphrase("Hey, can yuo hepl me cancel my last order?", tone="witty")
# "Hey, I need your help with my last order. Can you wave your magic wand and make it disappear?"
```
#### Dialogue Summary and Topic generation
```python
from llm_toys.tasks import SummaryAndTopicGenerator
summary_theme_generator = SummaryAndTopicGenerator()
summary_theme_generator.generate_summary_and_topic(
"""
#Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!
#Person2#: What's got you so hyped?
#Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.
#Person2#: Which movie is it?
#Person1#: It's called "Whisper of the Wind." It's about a girl on a magical journey to save her village.
#Person2#: Sounds amazing! I'm in for the premiere.
#Person1#: Great! We're in for a visual masterpiece and a heartfelt story.
#Person2#: Can't wait to be transported to their world.
#Person1#: It'll be an unforgettable experience, for sure!
""".strip()
)
# {"summary": "#Person1# is excited for the premiere of the latest Studio Ghibli movie.
# #Person1# thinks the animation, storytelling, and heartfelt story will be unforgettable.
# #Person2# is also excited for the premiere.",
# "topic": "Studio ghibli movie"}
```
### General 7B model
```python
from llm_toys.tasks import GeneralTaskAssitant
from llm_toys.config import TaskType
gta = GeneralTaskAssitant()
gta.complete(TaskType.PARAPHRASE_TONE, "Hey, can yuo hepl me cancel my last order?")
# "Could you assist me in canceling my previous order?"
gta.complete(TaskType.PARAPHRASE_TONE, "Hey, can yuo hepl me cancel my last order?", tone="casual")
# "Hey, can you help me cancel my last order?"
gta.complete(TaskType.PARAPHRASE_TONE, "Hey, can yuo hepl me cancel my last order?", tone="professional")
# "I would appreciate if you could assist me in canceling my previous order."
gta.complete(TaskType.PARAPHRASE_TONE, "Hey, can yuo hepl me cancel my last order?", tone="witty")
# "Oops! Looks like I got a little carried away with my shopping spree. Can you help me cancel my last order?"
chat = """
#Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!
#Person2#: What's got you so hyped?
#Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.
#Person2#: Which movie is it?
#Person1#: It's called "Whisper of the Wind." It's about a girl on a magical journey to save her village.
#Person2#: Sounds amazing! I'm in for the premiere.
#Person1#: Great! We're in for a visual masterpiece and a heartfelt story.
#Person2#: Can't wait to be transported to their world.
#Person1#: It'll be an unforgettable experience, for sure!
""".strip()
gta.complete(TaskType.DIALOGUE_SUMMARY_TOPIC, chat)
# {"summary": "#Person1# tells #Person2# about the upcoming Studio Ghibli movie.
# #Person1# thinks it's magical and #Person2#'s excited to watch it.",
# "topic": "Movie premiere"}
```
## Training
### Data
- [Paraphrasing and Tone change](data/paraphrase_tone.json): Contains passages and their paraphrased versions as well
as the passage in different tones like casual, professional and witty. Used to models to rephrase and change the
tone of a passage. Data was generated using gpt-35-turbo. A small sample of training passages have also been picked
up from quora quesions and squad_2 datasets.
- [Dialogue Summary and Topic generation](data/dialogue_summary_topic.json): Contains Dialogues and their Summary
and Topic. The training data is ~1k records from the training split of the
[Dialogsum dataset](https://github.com/cylnlp/dialogsum). It also contains ~20 samples from the dev split.
Data points with longer Summaries and Topics were given priority in the sampling. Note that some(~30) topics
were edited manually in final training data as the original labeled Topic was just a word and not descriptive enough.
## Evaluation
### Paraphrasing and Tone change
WIP
### Dialogue Summary and Topic generation
Evaluation is done on 500 records from the [Dialogsum test](https://github.com/cylnlp/dialogsum/tree/main/DialogSum_Data)
split.
```python
# llm-toys/RedPajama-INCITE-Base-3B-v1-dialogue-summary-topic
{"rouge1": 0.453, "rouge2": 0.197, "rougeL": 0.365, "topic_similarity": 0.888}
# llm-toys/falcon-7b-paraphrase-tone-dialogue-summary-topic
{'rouge1': 0.448, 'rouge2': 0.195, 'rougeL': 0.359, 'topic_similarity': 0.886}
```
## Roadmap
- [ ] Add tests.
- [ ] Ability to switch the LoRAs(for task wise models) without re-initializing the backbone model and tokenizer.
- [ ] Retrieval augmented QA.
- [ ] Explore the generalizability of 3B model across more tasks.
- [ ] Explore even smaller models.
- [ ] Evaluation strategy for tasks where we don"t have a test/eval dataset handy.
Raw data
{
"_id": null,
"home_page": "https://github.com/kuutsav/llm-toys",
"name": "llm-toys",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10.0",
"maintainer_email": "",
"keywords": "",
"author": "Kumar Utsav",
"author_email": "krum.utsav@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1c/71/bdf21072875cba7b19998fa103eae67c5478f130754b879cefcecb59ae73/llm-toys-0.1.1.tar.gz",
"platform": null,
"description": "# llm-toys\n\n[](https://github.com/psf/black)\n[](https://pypi.org/project/llm-toys/)\n\nSmall(7B and below), production-ready finetuned LLMs for a diverse set of useful tasks.\n\nSupported tasks: Paraphrasing, Changing the tone of a passage, Summary and Topic generation from a dailogue,\n~~Retrieval augmented QA(WIP)~~.\n\nWe finetune LoRAs on quantized 3B and 7B models. The 3B model is finetuned on specific tasks, while the 7B model is\nfinetuned on all the tasks.\n\nThe goal is to be able to finetune and use all these models on a very modest consumer grade hardware.\n\n## Installation\n\n```bash\npip install llm-toys\n```\n\n> Note that we are using the transformers and peft packages from the source directory, \n> not the installed package. 4bit bitsandbytes quantization was only working with the \n> main brach of transformers and peft. Once transformers version 4.31.0 and peft version 0.4.0 is \n> published to pypi we will use the published version.\n\n## Usage\n\n### Task specific 3B models\n\n#### Paraphrasing\n\n```python\nfrom llm_toys.tasks import Paraphraser\n\nparaphraser = Paraphraser()\nparaphraser.paraphrase(\"Hey, can yuo hepl me cancel my last order?\")\n# \"Could you kindly assist me in canceling my previous order?\"\n```\n\n#### Tone change\n\n```python\nparaphraser.paraphrase(\"Hey, can yuo hepl me cancel my last order?\", tone=\"casual\")\n# \"Hey, could you help me cancel my order?\"\n\nparaphraser.paraphrase(\"Hey, can yuo hepl me cancel my last order?\", tone=\"professional\")\n# \"I would appreciate guidance on canceling my previous order.\"\n\nparaphraser.paraphrase(\"Hey, can yuo hepl me cancel my last order?\", tone=\"witty\")\n# \"Hey, I need your help with my last order. Can you wave your magic wand and make it disappear?\"\n```\n\n#### Dialogue Summary and Topic generation\n\n```python\nfrom llm_toys.tasks import SummaryAndTopicGenerator\n\nsummary_theme_generator = SummaryAndTopicGenerator()\nsummary_theme_generator.generate_summary_and_topic(\n \"\"\"\n #Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!\n #Person2#: What's got you so hyped?\n #Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.\n #Person2#: Which movie is it?\n #Person1#: It's called \"Whisper of the Wind.\" It's about a girl on a magical journey to save her village.\n #Person2#: Sounds amazing! I'm in for the premiere.\n #Person1#: Great! We're in for a visual masterpiece and a heartfelt story.\n #Person2#: Can't wait to be transported to their world.\n #Person1#: It'll be an unforgettable experience, for sure!\n \"\"\".strip()\n)\n# {\"summary\": \"#Person1# is excited for the premiere of the latest Studio Ghibli movie.\n# #Person1# thinks the animation, storytelling, and heartfelt story will be unforgettable.\n# #Person2# is also excited for the premiere.\",\n# \"topic\": \"Studio ghibli movie\"}\n```\n\n### General 7B model\n\n```python\nfrom llm_toys.tasks import GeneralTaskAssitant\nfrom llm_toys.config import TaskType\n\ngta = GeneralTaskAssitant()\ngta.complete(TaskType.PARAPHRASE_TONE, \"Hey, can yuo hepl me cancel my last order?\")\n# \"Could you assist me in canceling my previous order?\"\n\ngta.complete(TaskType.PARAPHRASE_TONE, \"Hey, can yuo hepl me cancel my last order?\", tone=\"casual\")\n# \"Hey, can you help me cancel my last order?\"\n\ngta.complete(TaskType.PARAPHRASE_TONE, \"Hey, can yuo hepl me cancel my last order?\", tone=\"professional\")\n# \"I would appreciate if you could assist me in canceling my previous order.\"\n\ngta.complete(TaskType.PARAPHRASE_TONE, \"Hey, can yuo hepl me cancel my last order?\", tone=\"witty\")\n# \"Oops! Looks like I got a little carried away with my shopping spree. Can you help me cancel my last order?\"\n\nchat = \"\"\"\n#Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!\n#Person2#: What's got you so hyped?\n#Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.\n#Person2#: Which movie is it?\n#Person1#: It's called \"Whisper of the Wind.\" It's about a girl on a magical journey to save her village.\n#Person2#: Sounds amazing! I'm in for the premiere.\n#Person1#: Great! We're in for a visual masterpiece and a heartfelt story.\n#Person2#: Can't wait to be transported to their world.\n#Person1#: It'll be an unforgettable experience, for sure!\n\"\"\".strip()\ngta.complete(TaskType.DIALOGUE_SUMMARY_TOPIC, chat)\n# {\"summary\": \"#Person1# tells #Person2# about the upcoming Studio Ghibli movie.\n# #Person1# thinks it's magical and #Person2#'s excited to watch it.\",\n# \"topic\": \"Movie premiere\"}\n```\n\n## Training\n\n### Data\n\n- [Paraphrasing and Tone change](data/paraphrase_tone.json): Contains passages and their paraphrased versions as well\nas the passage in different tones like casual, professional and witty. Used to models to rephrase and change the\ntone of a passage. Data was generated using gpt-35-turbo. A small sample of training passages have also been picked\nup from quora quesions and squad_2 datasets.\n\n- [Dialogue Summary and Topic generation](data/dialogue_summary_topic.json): Contains Dialogues and their Summary\nand Topic. The training data is ~1k records from the training split of the\n[Dialogsum dataset](https://github.com/cylnlp/dialogsum). It also contains ~20 samples from the dev split.\nData points with longer Summaries and Topics were given priority in the sampling. Note that some(~30) topics\nwere edited manually in final training data as the original labeled Topic was just a word and not descriptive enough.\n\n## Evaluation\n\n### Paraphrasing and Tone change\n\nWIP\n\n### Dialogue Summary and Topic generation\n\nEvaluation is done on 500 records from the [Dialogsum test](https://github.com/cylnlp/dialogsum/tree/main/DialogSum_Data)\nsplit.\n\n```python\n# llm-toys/RedPajama-INCITE-Base-3B-v1-dialogue-summary-topic\n{\"rouge1\": 0.453, \"rouge2\": 0.197, \"rougeL\": 0.365, \"topic_similarity\": 0.888}\n\n# llm-toys/falcon-7b-paraphrase-tone-dialogue-summary-topic\n{'rouge1': 0.448, 'rouge2': 0.195, 'rougeL': 0.359, 'topic_similarity': 0.886}\n```\n\n## Roadmap\n\n- [ ] Add tests.\n- [ ] Ability to switch the LoRAs(for task wise models) without re-initializing the backbone model and tokenizer.\n- [ ] Retrieval augmented QA.\n- [ ] Explore the generalizability of 3B model across more tasks.\n- [ ] Explore even smaller models.\n- [ ] Evaluation strategy for tasks where we don\"t have a test/eval dataset handy.\n",
"bugtrack_url": null,
"license": "https://opensource.org/license/mit/",
"summary": "Small(7B and below), production-ready finetuned LLMs for a diverse set of useful tasks.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/kuutsav/llm-toys"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f2d79bc709f7fa73309d83363f535dad73513c8b7346857e801c32b2b20d80bf",
"md5": "4be82e75ceb2e82fbd76108870ff711d",
"sha256": "b726db848d04fe14112ff4d2b5eaea6bf27b9330b2ca7dee454572c039d2772b"
},
"downloads": -1,
"filename": "llm_toys-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4be82e75ceb2e82fbd76108870ff711d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10.0",
"size": 7395478,
"upload_time": "2023-07-17T11:28:00",
"upload_time_iso_8601": "2023-07-17T11:28:00.058773Z",
"url": "https://files.pythonhosted.org/packages/f2/d7/9bc709f7fa73309d83363f535dad73513c8b7346857e801c32b2b20d80bf/llm_toys-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1c71bdf21072875cba7b19998fa103eae67c5478f130754b879cefcecb59ae73",
"md5": "81cc37d9b9d0cb65ae9b927b51a90cf8",
"sha256": "ee2073ed21afd8d5d37d7e33955565db28e5cb152a8f9f4d8f6de0eae70cb52c"
},
"downloads": -1,
"filename": "llm-toys-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "81cc37d9b9d0cb65ae9b927b51a90cf8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10.0",
"size": 6181200,
"upload_time": "2023-07-17T11:28:04",
"upload_time_iso_8601": "2023-07-17T11:28:04.648931Z",
"url": "https://files.pythonhosted.org/packages/1c/71/bdf21072875cba7b19998fa103eae67c5478f130754b879cefcecb59ae73/llm-toys-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-17 11:28:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kuutsav",
"github_project": "llm-toys",
"github_not_found": true,
"lcname": "llm-toys"
}
JSON
