<div align="center">
<img alt="LLMeter (Logo)" src="https://github.com/awslabs/llmeter/blob/main/docs/llmeter-logotype-192px.png?raw=true" height="96px" width="396px"/>
**Measuring large language models latency and throughput**
[](https://pypi.python.org/pypi/llmeter)
[](https://pypi.python.org/pypi/llmeter)
[](https://github.com/astral-sh/ruff)
</div>
LLMeter is a pure-python library for simple latency and throughput testing of large language models (LLMs). It's designed to be lightweight to install; straightforward to run standard tests; and versatile to integrate - whether in notebooks, CI/CD, or other workflows.
## 🛠️ Installation
LLMeter requires `python>=3.10`, please make sure your current version of python is compatible.
To install the basic metering functionalities, you can install the minimum package using pip install:
```terminal
pip install llmeter
```
LLMeter also offers extra features that require additional dependencies. Currently these extras include:
- **plotting**: Add methods to generate charts and heatmaps to summarize the results
- **openai**: Enable testing endpoints offered by OpenAI
- **litellm**: Enable testing a range of different models through [LiteLLM](https://github.com/BerriAI/litellm)
You can install one or more of these extra options using pip:
```terminal
pip install llmeter[plotting, openai, litellm]
```
## 🚀 Quick-start
At a high level, you'll start by configuring an LLMeter "Endpoint" for whatever type of LLM you're connecting to:
```python
# For example with Amazon Bedrock...
from llmeter.endpoints import BedrockConverse
endpoint = BedrockConverse(model_id="...")
# ...or OpenAI...
from llmeter.endpoints import OpenAIEndpoint
endpoint = OpenAIEndpoint(model_id="...", api_key="...")
# ...or via LiteLLM...
from llmeter.endpoints import LiteLLM
endpoint = LiteLLM("{provider}/{model_id}")
# ...and so on
```
You can then run the high-level "experiments" offered by LLMeter:
```python
# For example a heatmap of latency by input & output token count:
from llmeter.experiments import LatencyHeatmap
latency_heatmap = LatencyHeatmap(
endpoint=endpoint,
clients=10,
source_file="examples/MaryShelleyFrankenstein.txt",
...
)
heatmap_results = await latency_heatmap.run()
latency_heatmap.plot_heatmap()
# ...Or testing how throughput varies with concurrent request count:
from llmeter.experiments import LoadTest
sweep_test = LoadTest(
endpoint=endpoint,
payload={...},
sequence_of_clients=[1, 5, 20, 50, 100, 500],
)
sweep_results = await sweep_test.run()
sweep_test.plot_sweep_results()
```
Alternatively, you can use the low-level `llmeter.runner.Runner` class to run and analyze request
batches - and build your own custom experiments.
For more details, check out our selection of end-to-end code examples in the [examples](https://github.com/awslabs/llmeter/tree/main/examples) folder!
## Security
See [CONTRIBUTING](https://github.com/awslabs/llmeter/tree/main/CONTRIBUTING.md#security-issue-notifications) for more information.
## License
This project is licensed under the Apache-2.0 License.
Raw data
{
"_id": null,
"home_page": "https://github.com/awslabs/llmeter",
"name": "llmeter",
"maintainer": "llmeter-maintainers",
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": "llmeter-maintainers@amazon.com",
"keywords": "llm, genai, testing, performance",
"author": "Amazon Web Services",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/12/98/7c854d54b1ea2d88b4dcd9ffde8dbca1da6eca5abbb03adbb182036aeb69/llmeter-0.1.2.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n<img alt=\"LLMeter (Logo)\" src=\"https://github.com/awslabs/llmeter/blob/main/docs/llmeter-logotype-192px.png?raw=true\" height=\"96px\" width=\"396px\"/>\n\n**Measuring large language models latency and throughput**\n\n[](https://pypi.python.org/pypi/llmeter)\n[](https://pypi.python.org/pypi/llmeter)\n[](https://github.com/astral-sh/ruff)\n\n</div>\n\nLLMeter is a pure-python library for simple latency and throughput testing of large language models (LLMs). It's designed to be lightweight to install; straightforward to run standard tests; and versatile to integrate - whether in notebooks, CI/CD, or other workflows.\n\n## \ud83d\udee0\ufe0f Installation\n\nLLMeter requires `python>=3.10`, please make sure your current version of python is compatible.\n\nTo install the basic metering functionalities, you can install the minimum package using pip install:\n\n```terminal\npip install llmeter\n```\n\nLLMeter also offers extra features that require additional dependencies. Currently these extras include:\n\n- **plotting**: Add methods to generate charts and heatmaps to summarize the results\n- **openai**: Enable testing endpoints offered by OpenAI\n- **litellm**: Enable testing a range of different models through [LiteLLM](https://github.com/BerriAI/litellm)\n\nYou can install one or more of these extra options using pip:\n\n```terminal\npip install llmeter[plotting, openai, litellm]\n```\n\n## \ud83d\ude80 Quick-start\n\nAt a high level, you'll start by configuring an LLMeter \"Endpoint\" for whatever type of LLM you're connecting to:\n\n```python\n# For example with Amazon Bedrock...\nfrom llmeter.endpoints import BedrockConverse\nendpoint = BedrockConverse(model_id=\"...\")\n\n# ...or OpenAI...\nfrom llmeter.endpoints import OpenAIEndpoint\nendpoint = OpenAIEndpoint(model_id=\"...\", api_key=\"...\")\n\n# ...or via LiteLLM...\nfrom llmeter.endpoints import LiteLLM\nendpoint = LiteLLM(\"{provider}/{model_id}\")\n\n# ...and so on\n```\n\nYou can then run the high-level \"experiments\" offered by LLMeter:\n\n```python\n# For example a heatmap of latency by input & output token count:\nfrom llmeter.experiments import LatencyHeatmap\nlatency_heatmap = LatencyHeatmap(\n endpoint=endpoint,\n clients=10,\n source_file=\"examples/MaryShelleyFrankenstein.txt\",\n ...\n)\nheatmap_results = await latency_heatmap.run()\nlatency_heatmap.plot_heatmap()\n\n# ...Or testing how throughput varies with concurrent request count:\nfrom llmeter.experiments import LoadTest\nsweep_test = LoadTest(\n endpoint=endpoint,\n payload={...},\n sequence_of_clients=[1, 5, 20, 50, 100, 500],\n)\nsweep_results = await sweep_test.run()\nsweep_test.plot_sweep_results()\n```\n\nAlternatively, you can use the low-level `llmeter.runner.Runner` class to run and analyze request\nbatches - and build your own custom experiments.\n\nFor more details, check out our selection of end-to-end code examples in the [examples](https://github.com/awslabs/llmeter/tree/main/examples) folder!\n\n## Security\n\nSee [CONTRIBUTING](https://github.com/awslabs/llmeter/tree/main/CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A lightweight, cross-platform latency and throughput profiler for LLMs",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/awslabs/llmeter",
"Repository": "https://github.com/awslabs/llmeter"
},
"split_keywords": [
"llm",
" genai",
" testing",
" performance"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b6efee80a7e9a1e94461a3dfaa886af0f184b708f6b743bc220a162b64d0d747",
"md5": "58214f6012624bf38a7210bb2a955fe0",
"sha256": "a75f199eff1a6e9a8999c26931a46e05b33e54a2dc2fdf3447038e15de560b81"
},
"downloads": -1,
"filename": "llmeter-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "58214f6012624bf38a7210bb2a955fe0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 33918,
"upload_time": "2024-10-07T08:53:56",
"upload_time_iso_8601": "2024-10-07T08:53:56.652408Z",
"url": "https://files.pythonhosted.org/packages/b6/ef/ee80a7e9a1e94461a3dfaa886af0f184b708f6b743bc220a162b64d0d747/llmeter-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "12987c854d54b1ea2d88b4dcd9ffde8dbca1da6eca5abbb03adbb182036aeb69",
"md5": "812b38cf3eec107c9e764086e3100240",
"sha256": "271cfb8440c4424dbb0feaa52557aa2706647166a979653d2136ed67683401d1"
},
"downloads": -1,
"filename": "llmeter-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "812b38cf3eec107c9e764086e3100240",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 27257,
"upload_time": "2024-10-07T08:53:58",
"upload_time_iso_8601": "2024-10-07T08:53:58.407096Z",
"url": "https://files.pythonhosted.org/packages/12/98/7c854d54b1ea2d88b4dcd9ffde8dbca1da6eca5abbb03adbb182036aeb69/llmeter-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-07 08:53:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "awslabs",
"github_project": "llmeter",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llmeter"
}
JSON
