# Construe: An LLM Benchmark Utility
**An LLM inferencing benchmark tool focusing on device-specific latency and memory usage.**
## Quick Start
This package is intended to be installed with `pip` and it will create a command line program `construe` on your `$PATH` to execute benchmarking comamnds:
```
$ pip install construe
$ which construe
$ construe --help
```
There are several top-level configurations that you can specify either as an environment variable or a command line option before the command. The environment variables are as follows:
- `$CONSTRUE_ENV` or `$ENV`: specify the name of the experimental environment for comparison purposes.
- `$CONSTRUE_DEVICE` or `$TORCH_DEVICE`: specify the name of the default device to use with PyTorch e.g. cpu, mps, or cuda.
The command line utility help is as follows:
```
Usage: construe [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-d, --device TEXT specify the pytorch device to run on e.g. cpu, mps or
cuda
-e, --env TEXT name of the experimental environment for comparison
(default is hostname)
-h, --help Show this message and exit.
Commands:
basic
moondream
```
## Basic Benchmarks
The basic benchmarks implement dot product benchmarks from the [PyTorch documentation](https://pytorch.org/tutorials/recipes/recipes/benchmark.html). These benchmarks can be run using `construe basic`; for example by running:
```
$ construe -e "MacBook Pro 2022 M1" basic -o results-macbook.pickle
```
The `-e` flag specifies the environment for comparison purposes and the `-o` flag saves the measurements out to disk as a Pickle file that can be loaded for comparison to other environments later.
Command usage is as follows:
```
Usage: construe basic [OPTIONS]
Options:
-e, --env TEXT name of the experimental environment for
comparison (default is hostname)
-o, --saveto TEXT path to write the measurements pickle data to
-t, --num-threads INTEGER specify number of threads for benchmark (default
to maximum)
-F, --fuzz / --no-fuzz fuzz the tensor sizes of the inputs to the
benchmark
-S, --seed INTEGER set the random seed for random generation
-h, --help Show this message and exit.
```
## Moondream Benchmarks
The [moondream](https://huggingface.co/vikhyatk/moondream2) package contains small image-to-text computer vision models that can be used in the first step of a [content moderation](https://www.cloudraft.io/blog/content-moderation-using-llamaindex-and-llm) workflow (e.g. image to text, moderate text). This benchmark executes the model for _encoding_ and _inferencing_ on a small number of images and reports the average time for both operations and the line-by-line memory usage of the model.
It can be run as follows:
```
$ construe moondream
```
Command usage is as follows:
```
Usage: construe moondream [OPTIONS]
Options:
-h, --help Show this message and exit.
```
## Releases
To release the construe library and deploy to PyPI run the following commands:
```
$ python -m build
$ twine upload dist/*
```
Raw data
{
"_id": null,
"home_page": null,
"name": "construe",
"maintainer": "Rotational Labs",
"docs_url": null,
"requires_python": "<4,>=3.10",
"maintainer_email": "support@rotational.io",
"keywords": "llm, benchmark, construe, inference latency, inference memory",
"author": "Rotational Labs",
"author_email": "support@rotational.io",
"download_url": "https://files.pythonhosted.org/packages/ef/b6/9841472fd0979c07a2514d246da1e5cc50957f26d1c2ddab84f969aa8ca9/construe-0.2.0.tar.gz",
"platform": null,
"description": "# Construe: An LLM Benchmark Utility\n\n**An LLM inferencing benchmark tool focusing on device-specific latency and memory usage.**\n\n## Quick Start\n\nThis package is intended to be installed with `pip` and it will create a command line program `construe` on your `$PATH` to execute benchmarking comamnds:\n\n```\n$ pip install construe\n$ which construe\n$ construe --help\n```\n\nThere are several top-level configurations that you can specify either as an environment variable or a command line option before the command. The environment variables are as follows:\n\n- `$CONSTRUE_ENV` or `$ENV`: specify the name of the experimental environment for comparison purposes.\n- `$CONSTRUE_DEVICE` or `$TORCH_DEVICE`: specify the name of the default device to use with PyTorch e.g. cpu, mps, or cuda.\n\nThe command line utility help is as follows:\n\n```\nUsage: construe [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --version Show the version and exit.\n -d, --device TEXT specify the pytorch device to run on e.g. cpu, mps or\n cuda\n -e, --env TEXT name of the experimental environment for comparison\n (default is hostname)\n -h, --help Show this message and exit.\n\nCommands:\n basic\n moondream\n```\n\n## Basic Benchmarks\n\nThe basic benchmarks implement dot product benchmarks from the [PyTorch documentation](https://pytorch.org/tutorials/recipes/recipes/benchmark.html). These benchmarks can be run using `construe basic`; for example by running:\n\n```\n$ construe -e \"MacBook Pro 2022 M1\" basic -o results-macbook.pickle\n```\n\nThe `-e` flag specifies the environment for comparison purposes and the `-o` flag saves the measurements out to disk as a Pickle file that can be loaded for comparison to other environments later.\n\nCommand usage is as follows:\n\n```\nUsage: construe basic [OPTIONS]\n\nOptions:\n -e, --env TEXT name of the experimental environment for\n comparison (default is hostname)\n -o, --saveto TEXT path to write the measurements pickle data to\n -t, --num-threads INTEGER specify number of threads for benchmark (default\n to maximum)\n -F, --fuzz / --no-fuzz fuzz the tensor sizes of the inputs to the\n benchmark\n -S, --seed INTEGER set the random seed for random generation\n -h, --help Show this message and exit.\n```\n\n## Moondream Benchmarks\n\nThe [moondream](https://huggingface.co/vikhyatk/moondream2) package contains small image-to-text computer vision models that can be used in the first step of a [content moderation](https://www.cloudraft.io/blog/content-moderation-using-llamaindex-and-llm) workflow (e.g. image to text, moderate text). This benchmark executes the model for _encoding_ and _inferencing_ on a small number of images and reports the average time for both operations and the line-by-line memory usage of the model.\n\nIt can be run as follows:\n\n```\n$ construe moondream\n```\n\nCommand usage is as follows:\n\n```\nUsage: construe moondream [OPTIONS]\n\nOptions:\n -h, --help Show this message and exit.\n```\n\n## Releases\n\nTo release the construe library and deploy to PyPI run the following commands:\n\n```\n$ python -m build\n$ twine upload dist/*\n```\n",
"bugtrack_url": null,
"license": "BSD 3",
"summary": "An LLM inferencing benchmark tool focusing on device-specific latency and memory usage",
"version": "0.2.0",
"project_urls": {
"Download": "https://github.com/rotationalio/llm-benchmark/tarball/v0.2.0",
"Source": "https://github.com/rotationalio/llm-benchmark",
"Tracker": "https://github.com/rotationalio/llm-benchmark/issues"
},
"split_keywords": [
"llm",
" benchmark",
" construe",
" inference latency",
" inference memory"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "37e1e5d0439475db4b5a6c1b6699de1917d751afe8f17fb0f8d24108112de6a4",
"md5": "8aaf74b89e54d8a0fc9dfd38e57b5a79",
"sha256": "6f8b2442a91c0d65c984180a56572b51baf6082b64cf5f8109b8bf4149d2d68d"
},
"downloads": -1,
"filename": "construe-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8aaf74b89e54d8a0fc9dfd38e57b5a79",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.10",
"size": 14140,
"upload_time": "2024-11-13T03:52:24",
"upload_time_iso_8601": "2024-11-13T03:52:24.081592Z",
"url": "https://files.pythonhosted.org/packages/37/e1/e5d0439475db4b5a6c1b6699de1917d751afe8f17fb0f8d24108112de6a4/construe-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "efb69841472fd0979c07a2514d246da1e5cc50957f26d1c2ddab84f969aa8ca9",
"md5": "3f0505b8352a68e61de848ad437a2bfd",
"sha256": "ec8dd39dd47bfbdec39335636e8e13d932dd85086e5608c4afbe8df845a450ad"
},
"downloads": -1,
"filename": "construe-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "3f0505b8352a68e61de848ad437a2bfd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.10",
"size": 15623,
"upload_time": "2024-11-13T03:52:24",
"upload_time_iso_8601": "2024-11-13T03:52:24.985431Z",
"url": "https://files.pythonhosted.org/packages/ef/b6/9841472fd0979c07a2514d246da1e5cc50957f26d1c2ddab84f969aa8ca9/construe-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-13 03:52:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rotationalio",
"github_project": "llm-benchmark",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "einops",
"specs": [
[
"==",
"0.8.0"
]
]
},
{
"name": "memory-profiler",
"specs": [
[
"==",
"0.61.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"2.1.3"
]
]
},
{
"name": "pillow",
"specs": [
[
"==",
"11.0.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"1.0.1"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.3"
]
]
},
{
"name": "torch",
"specs": [
[
"==",
"2.5.1"
]
]
},
{
"name": "torchvision",
"specs": [
[
"==",
"0.20.1"
]
]
},
{
"name": "tqdm",
"specs": [
[
"==",
"4.67.0"
]
]
},
{
"name": "transformers",
"specs": [
[
"==",
"4.46.2"
]
]
}
],
"lcname": "construe"
}
JSON
