<div align="center">
<img alt="LLMeter (Logo)" src="https://github.com/awslabs/llmeter/blob/main/docs/llmeter-logotype-192px.png?raw=true" height="96px" width="396px"/>
**Measuring large language models latency and throughput**
[](https://pypi.python.org/pypi/llmeter)
[](https://pypi.python.org/pypi/llmeter)
[](https://github.com/astral-sh/ruff)
</div>
LLMeter is a pure-python library for simple latency and throughput testing of large language models (LLMs). It's designed to be lightweight to install; straightforward to run standard tests; and versatile to integrate - whether in notebooks, CI/CD, or other workflows.
## 🛠️ Installation
LLMeter requires `python>=3.10`, please make sure your current version of python is compatible.
To install the basic metering functionalities, you can install the minimum package using pip install:
```terminal
pip install llmeter
```
LLMeter also offers extra features that require additional dependencies. Currently these extras include:
- **plotting**: Add methods to generate charts and heatmaps to summarize the results
- **openai**: Enable testing endpoints offered by OpenAI
- **litellm**: Enable testing a range of different models through [LiteLLM](https://github.com/BerriAI/litellm)
- **mlflow**: Enable logging LLMeter experiments to [MLFlow](https://mlflow.org/)
You can install one or more of these extra options using pip:
```terminal
pip install 'llmeter[plotting,openai,litellm,mlflow]'
```
## 🚀 Quick-start
At a high level, you'll start by configuring an LLMeter "Endpoint" for whatever type of LLM you're connecting to:
```python
# For example with Amazon Bedrock...
from llmeter.endpoints import BedrockConverse
endpoint = BedrockConverse(model_id="...")
# ...or OpenAI...
from llmeter.endpoints import OpenAIEndpoint
endpoint = OpenAIEndpoint(model_id="...", api_key="...")
# ...or via LiteLLM...
from llmeter.endpoints import LiteLLM
endpoint = LiteLLM("{provider}/{model_id}")
# ...and so on
```
You can then run the high-level "experiments" offered by LLMeter:
```python
# For example a heatmap of latency by input & output token count:
from llmeter.experiments import LatencyHeatmap
latency_heatmap = LatencyHeatmap(
endpoint=endpoint,
clients=10,
source_file="examples/MaryShelleyFrankenstein.txt",
...
)
heatmap_results = await latency_heatmap.run()
latency_heatmap.plot_heatmap()
# ...Or testing how throughput varies with concurrent request count:
from llmeter.experiments import LoadTest
sweep_test = LoadTest(
endpoint=endpoint,
payload={...},
sequence_of_clients=[1, 5, 20, 50, 100, 500],
)
sweep_results = await sweep_test.run()
sweep_test.plot_sweep_results()
```
Alternatively, you can use the low-level `llmeter.runner.Runner` class to run and analyze request
batches - and build your own custom experiments.
Additional functionality like cost modelling and MLFlow experiment tracking is enabled through `llmeter.callbacks`, and you can write your own callbacks to hook other custom logic into LLMeter test runs.
For more details, check out our selection of end-to-end code examples in the [examples](https://github.com/awslabs/llmeter/tree/main/examples) folder!
## Security
See [CONTRIBUTING](https://github.com/awslabs/llmeter/tree/main/CONTRIBUTING.md#security-issue-notifications) for more information.
## License
This project is licensed under the Apache-2.0 License.
Raw data
{
"_id": null,
"home_page": "https://github.com/awslabs/llmeter",
"name": "llmeter",
"maintainer": "llmeter-maintainers",
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": "llmeter-maintainers@amazon.com",
"keywords": "llm, genai, testing, performance",
"author": "Amazon Web Services",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/6e/3d/2b36c50601be77f5cc4d1c532f7183b6b9c419f4b077f5ef36d023df6838/llmeter-0.1.4.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n<img alt=\"LLMeter (Logo)\" src=\"https://github.com/awslabs/llmeter/blob/main/docs/llmeter-logotype-192px.png?raw=true\" height=\"96px\" width=\"396px\"/>\n\n**Measuring large language models latency and throughput**\n\n[](https://pypi.python.org/pypi/llmeter)\n[](https://pypi.python.org/pypi/llmeter)\n[](https://github.com/astral-sh/ruff)\n\n</div>\n\nLLMeter is a pure-python library for simple latency and throughput testing of large language models (LLMs). It's designed to be lightweight to install; straightforward to run standard tests; and versatile to integrate - whether in notebooks, CI/CD, or other workflows.\n\n## \ud83d\udee0\ufe0f Installation\n\nLLMeter requires `python>=3.10`, please make sure your current version of python is compatible.\n\nTo install the basic metering functionalities, you can install the minimum package using pip install:\n\n```terminal\npip install llmeter\n```\n\nLLMeter also offers extra features that require additional dependencies. Currently these extras include:\n\n- **plotting**: Add methods to generate charts and heatmaps to summarize the results\n- **openai**: Enable testing endpoints offered by OpenAI\n- **litellm**: Enable testing a range of different models through [LiteLLM](https://github.com/BerriAI/litellm)\n- **mlflow**: Enable logging LLMeter experiments to [MLFlow](https://mlflow.org/)\n\nYou can install one or more of these extra options using pip:\n\n```terminal\npip install 'llmeter[plotting,openai,litellm,mlflow]'\n```\n\n## \ud83d\ude80 Quick-start\n\nAt a high level, you'll start by configuring an LLMeter \"Endpoint\" for whatever type of LLM you're connecting to:\n\n```python\n# For example with Amazon Bedrock...\nfrom llmeter.endpoints import BedrockConverse\nendpoint = BedrockConverse(model_id=\"...\")\n\n# ...or OpenAI...\nfrom llmeter.endpoints import OpenAIEndpoint\nendpoint = OpenAIEndpoint(model_id=\"...\", api_key=\"...\")\n\n# ...or via LiteLLM...\nfrom llmeter.endpoints import LiteLLM\nendpoint = LiteLLM(\"{provider}/{model_id}\")\n\n# ...and so on\n```\n\nYou can then run the high-level \"experiments\" offered by LLMeter:\n\n```python\n# For example a heatmap of latency by input & output token count:\nfrom llmeter.experiments import LatencyHeatmap\nlatency_heatmap = LatencyHeatmap(\n endpoint=endpoint,\n clients=10,\n source_file=\"examples/MaryShelleyFrankenstein.txt\",\n ...\n)\nheatmap_results = await latency_heatmap.run()\nlatency_heatmap.plot_heatmap()\n\n# ...Or testing how throughput varies with concurrent request count:\nfrom llmeter.experiments import LoadTest\nsweep_test = LoadTest(\n endpoint=endpoint,\n payload={...},\n sequence_of_clients=[1, 5, 20, 50, 100, 500],\n)\nsweep_results = await sweep_test.run()\nsweep_test.plot_sweep_results()\n```\n\nAlternatively, you can use the low-level `llmeter.runner.Runner` class to run and analyze request\nbatches - and build your own custom experiments.\n\nAdditional functionality like cost modelling and MLFlow experiment tracking is enabled through `llmeter.callbacks`, and you can write your own callbacks to hook other custom logic into LLMeter test runs.\n\nFor more details, check out our selection of end-to-end code examples in the [examples](https://github.com/awslabs/llmeter/tree/main/examples) folder!\n\n## Security\n\nSee [CONTRIBUTING](https://github.com/awslabs/llmeter/tree/main/CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A lightweight, cross-platform latency and throughput profiler for LLMs",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/awslabs/llmeter",
"Repository": "https://github.com/awslabs/llmeter"
},
"split_keywords": [
"llm",
" genai",
" testing",
" performance"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "84a9fb05813f2c1c94b3dc0c2dc03aa5b704d96888e64e20952323924330fa7d",
"md5": "fd1f10f495f8245aaf1c1d8b73f1ec5e",
"sha256": "4968e630dae775fcb4140587ef4980029136a86c2f0bc25655f8f4eb9bbfd471"
},
"downloads": -1,
"filename": "llmeter-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fd1f10f495f8245aaf1c1d8b73f1ec5e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 58084,
"upload_time": "2024-12-19T04:42:23",
"upload_time_iso_8601": "2024-12-19T04:42:23.081179Z",
"url": "https://files.pythonhosted.org/packages/84/a9/fb05813f2c1c94b3dc0c2dc03aa5b704d96888e64e20952323924330fa7d/llmeter-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6e3d2b36c50601be77f5cc4d1c532f7183b6b9c419f4b077f5ef36d023df6838",
"md5": "3bcd9ad29b499cc07a0c2d9626d6653b",
"sha256": "f2e726f78042550d1b98386d0398bedcc071d4a6566e96c03e003a6904564571"
},
"downloads": -1,
"filename": "llmeter-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "3bcd9ad29b499cc07a0c2d9626d6653b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 45532,
"upload_time": "2024-12-19T04:42:25",
"upload_time_iso_8601": "2024-12-19T04:42:25.916802Z",
"url": "https://files.pythonhosted.org/packages/6e/3d/2b36c50601be77f5cc4d1c532f7183b6b9c419f4b077f5ef36d023df6838/llmeter-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-19 04:42:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "awslabs",
"github_project": "llmeter",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llmeter"
}
JSON
