<p align="center">
<img src="assets/logo.webp" alt="logo" width="200"/>
</p>
**LLMTrack** is a Python package designed to streamline the usage of language models, especially during batch generation. It offers features for easy loading of models, generation caching to optimize performance, detailed logging, and continuous token usage recording on a per-model basis. Whether you're working on research or deploying models in production, EfficientLLM helps you manage and monitor your models efficiently.
## Installation
```
pip install llmtrack
```
## LLM Loading
```python
from llmtrack import get_llm
llm = get_llm(model_name="openai/gpt-4o-mini")
print(llm.generate("Generate ONLY a random word"))
```
Public LLM APIs are specified by simply specifying `model_name` consisting of API providers and model names. The supported APIs include :
* OpenAI, e.g., "openai/xxxx" (xxxx should be replaced by specific model names)
* The environment variable has to be setup: `OPENAI_API_KEY`
* Specific model names: See [the document](https://platform.openai.com/docs/models)
* Azure OpenAI, e.g., "azure_openai/chatgpt-4k"
* The three environment variables have to be setup: `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`
* Ask providers for specific model names
* MoonShot, e.g., "moonshot/moonshot-v1-8k"
* The environment variable has to be setup: `MOONSHOT_API_KEY`
## Caching
```python
from llmtrack import get_llm
llm = get_llm(model_name="openai/gpt-4o-mini", cache=True)
print(llm.generate("Generate ONLY a random word "))
```
After running the code above, the generation cache will be stored in `cahe_llmtrack/openai/gpt-4o-mini`, following the naming rule `cahe_llmtrack/{API provider}/{model name}`.
If you invoke the same model with the same prompt, the cache will be used.
> Note: You can verify this by checking whether token usage increases with the next function: Token Usage Tracking.
## Token Usage Tracking
```python
from llmtrack import get_llm
llm = get_llm("openai/gpt-4o-mini", cache=True, token_usage=True)
print(llm.generate("Generate ONLY a random word "))
print(llm.generate("Generate ONLY a random word "))
```
Let's track the token usage at the `./gpt-4o-mini_token_usage.json`. Only one record exists, although we invoke the LLM twice.
```json
{"prompt": 17, "completion": 4, "total": 21, "time": "2024-08-25 14:02:36"}
```
## Logging (TBA)
Raw data
{
"_id": null,
"home_page": "https://github.com/xinzhel/llmtrack",
"name": "llmtrack",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Xinzhe Li",
"author_email": "xinzheli212@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/30/6e/f629d371301a1dcf03cf7bc367c8dfe8b623117ba09ee313cdb590a182e2/llmtrack-0.2.0.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"assets/logo.webp\" alt=\"logo\" width=\"200\"/>\n</p>\n\n\n**LLMTrack** is a Python package designed to streamline the usage of language models, especially during batch generation. It offers features for easy loading of models, generation caching to optimize performance, detailed logging, and continuous token usage recording on a per-model basis. Whether you're working on research or deploying models in production, EfficientLLM helps you manage and monitor your models efficiently.\n\n## Installation\n```\npip install llmtrack\n```\n## LLM Loading\n```python\nfrom llmtrack import get_llm\nllm = get_llm(model_name=\"openai/gpt-4o-mini\")\nprint(llm.generate(\"Generate ONLY a random word\"))\n```\n\nPublic LLM APIs are specified by simply specifying `model_name` consisting of API providers and model names. The supported APIs include : \n* OpenAI, e.g., \"openai/xxxx\" (xxxx should be replaced by specific model names)\n * The environment variable has to be setup: `OPENAI_API_KEY` \n * Specific model names: See [the document](https://platform.openai.com/docs/models) \n* Azure OpenAI, e.g., \"azure_openai/chatgpt-4k\" \n * The three environment variables have to be setup: `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`\n * Ask providers for specific model names \n* MoonShot, e.g., \"moonshot/moonshot-v1-8k\" \n * The environment variable has to be setup: `MOONSHOT_API_KEY`\n\n## Caching\n```python\nfrom llmtrack import get_llm\nllm = get_llm(model_name=\"openai/gpt-4o-mini\", cache=True)\nprint(llm.generate(\"Generate ONLY a random word \"))\n```\nAfter running the code above, the generation cache will be stored in `cahe_llmtrack/openai/gpt-4o-mini`, following the naming rule `cahe_llmtrack/{API provider}/{model name}`.\n\nIf you invoke the same model with the same prompt, the cache will be used. \n> Note: You can verify this by checking whether token usage increases with the next function: Token Usage Tracking.\n\n## Token Usage Tracking\n```python\nfrom llmtrack import get_llm\nllm = get_llm(\"openai/gpt-4o-mini\", cache=True, token_usage=True)\nprint(llm.generate(\"Generate ONLY a random word \"))\nprint(llm.generate(\"Generate ONLY a random word \"))\n```\nLet's track the token usage at the `./gpt-4o-mini_token_usage.json`. Only one record exists, although we invoke the LLM twice.\n```json\n{\"prompt\": 17, \"completion\": 4, \"total\": 21, \"time\": \"2024-08-25 14:02:36\"}\n```\n\n## Logging (TBA)\n \n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A concise description of your package.",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/xinzhel/llmtrack"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a1ddad8a449b23cfd764d3041a636184008161864385f26dc5884a3850d677fd",
"md5": "93655c4f50aff8235661ee762cc131eb",
"sha256": "540f9bbc684727ca25ffb646ea5e45942b84ac243de38982b36e263f84780e7a"
},
"downloads": -1,
"filename": "llmtrack-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "93655c4f50aff8235661ee762cc131eb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 9662,
"upload_time": "2024-08-25T04:15:23",
"upload_time_iso_8601": "2024-08-25T04:15:23.196017Z",
"url": "https://files.pythonhosted.org/packages/a1/dd/ad8a449b23cfd764d3041a636184008161864385f26dc5884a3850d677fd/llmtrack-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "306ef629d371301a1dcf03cf7bc367c8dfe8b623117ba09ee313cdb590a182e2",
"md5": "97aafc52b754a4119d62935f1f14cb3b",
"sha256": "3ce97ecff0f17bc6aac16fe352646f143768484fd59e00aaab1cc417f7600365"
},
"downloads": -1,
"filename": "llmtrack-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "97aafc52b754a4119d62935f1f14cb3b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 9449,
"upload_time": "2024-08-25T04:15:24",
"upload_time_iso_8601": "2024-08-25T04:15:24.792698Z",
"url": "https://files.pythonhosted.org/packages/30/6e/f629d371301a1dcf03cf7bc367c8dfe8b623117ba09ee313cdb590a182e2/llmtrack-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-25 04:15:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xinzhel",
"github_project": "llmtrack",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "llmtrack"
}
JSON
