# TorchEval
<p align="center">
<a href="https://github.com/pytorch/torcheval/actions?query=branch%3Amain"><img src="https://img.shields.io/github/actions/workflow/status/pytorch/torcheval/.github/workflows/unit_test.yaml?branch=main" alt="build status"></a>
<a href="https://pypi.org/project/torcheval"><img src="https://img.shields.io/pypi/v/torcheval" alt="pypi version"></a>
<a href="https://pypi.org/project/torcheval-nightly"><img src="https://img.shields.io/pypi/v/torcheval-nightly?label=nightly" alt="pypi nightly version"></a>
<a href="https://github.com/pytorch/torcheval/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/torcheval" alt="bsd license"></a>
</div>
<a href="https://pytorch.github.io/torcheval"><img src="https://img.shields.io/badge/docs-main-brightgreen" alt="docs"></a>
<p>
**This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.**
A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.
## Installing TorchEval
Requires Python >= 3.8 and PyTorch >= 1.11
From pip:
```bash
pip install torcheval
```
For nighly build version
```bash
pip install --pre torcheval-nightly
```
From source:
```bash
git clone https://github.com/pytorch/torcheval
cd torcheval
pip install -r requirements.txt
python setup.py install
```
## Quick Start
Take a look at the [quickstart notebook](https://github.com/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb), or fork it on [Colab](https://colab.research.google.com/github/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb).
There are more examples in the [examples](https://github.com/pytorch/torcheval/blob/main/examples) directory:
```bash
cd torcheval
python examples/simple_example.py
```
## Documentation
Documentation can be found at at [pytorch.org/torcheval](https://pytorch.org/torcheval)
## Using TorchEval
TorchEval can be run on CPU, GPU, and in a multi-process or multi-GPU setting. Metrics are provided in two interfaces, functional and class based. The functional interfaces can be found in `torcheval.metrics.functional` and are useful when your program runs in a single process setting. To use multi-process or multi-gpu configurations, the class-based interfaces, found in `torcheval.metrics` provide a much simpler experience. The class based interfaces also allow you to defer some of the computation of the metric by calling `update()` multiple times before `compute()`. This can be advantageous even in a single process setting due to saved computation overhead.
### Single Process
For use in a single process program, the simplest use case utilizes a functional metric. We simply import the metric function and feed in our outputs and targets. The example below shows a minimal PyTorch training loop that evaluates the multiclass accuracy of every fourth batch of data.
#### Functional Version (immediate computation of metric)
```python
import torch
from torcheval.metrics.functional import multiclass_accuracy
NUM_BATCHES = 16
BATCH_SIZE = 8
INPUT_SIZE = 10
NUM_CLASSES = 6
eval_frequency = 4
model = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())
optim = torch.optim.Adagrad(model.parameters(), lr=0.001)
loss_fn = torch.nn.CrossEntropyLoss()
metric_history = []
for batch in range(NUM_BATCHES):
input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))
target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)
outputs = model(input)
loss = loss_fn(outputs, target)
optim.zero_grad()
loss.backward()
optim.step()
# metric only computed every 4 batches,
# data from previous three batches is lost
if (batch + 1) % eval_frequency == 0:
metric_history.append(multiclass_accuracy(outputs, target))
```
### Single Process with Deferred Computation
#### Class Version (enables deferred computation of metric)
```python
import torch
from torcheval.metrics import MulticlassAccuracy
NUM_BATCHES = 16
BATCH_SIZE = 8
INPUT_SIZE = 10
NUM_CLASSES = 6
eval_frequency = 4
model = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())
optim = torch.optim.Adagrad(model.parameters(), lr=0.001)
loss_fn = torch.nn.CrossEntropyLoss()
metric = MulticlassAccuracy()
metric_history = []
for batch in range(NUM_BATCHES):
input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))
target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)
outputs = model(input)
loss = loss_fn(outputs, target)
optim.zero_grad()
loss.backward()
optim.step()
# metric only computed every 4 batches,
# data from previous three batches is included
metric.update(input, target)
if (batch + 1) % eval_frequency == 0:
metric_history.append(metric.compute())
# remove old data so that the next call
# to compute is only based off next 4 batches
metric.reset()
```
### Multi-Process or Multi-GPU
For usage on multiple devices a minimal example is given below. In the normal `torch.distributed` paradigm, each device is allocated its own process gets a unique numerical ID called a "global rank", counting up from 0.
#### Class Version (enables deferred computation and multi-processing)
```python
import torch
from torcheval.metrics.toolkit import sync_and_compute
from torcheval.metrics import MulticlassAccuracy
# Using torch.distributed
local_rank = int(os.environ["LOCAL_RANK"]) #rank on local machine, i.e. unique ID within a machine
global_rank = int(os.environ["RANK"]) #rank in global pool, i.e. unique ID within the entire process group
world_size = int(os.environ["WORLD_SIZE"]) #total number of processes or "ranks" in the entire process group
device = torch.device(
f"cuda:{local_rank}"
if torch.cuda.is_available() and torch.cuda.device_count() >= world_size
else "cpu"
)
metric = MulticlassAccuracy(device=device)
num_epochs, num_batches = 4, 8
for epoch in range(num_epochs):
for i in range(num_batches):
input = torch.randint(high=5, size=(10,), device=device)
target = torch.randint(high=5, size=(10,), device=device)
# Add data to metric locally
metric.update(input, target)
# metric.compute() will returns metric value from
# all seen data on the local process since last reset()
local_compute_result = metric.compute()
# sync_and_compute(metric) syncs metric data across all ranks and computes the metric value
global_compute_result = sync_and_compute(metric)
if global_rank == 0:
print(global_compute_result)
# metric.reset() clears the data on each process so that subsequent
# calls to compute() only act on new data
metric.reset()
```
See the [example directory](https://github.com/pytorch/torcheval/tree/main/examples) for more examples.
## Contributing
We welcome PRs! See the [CONTRIBUTING](CONTRIBUTING.md) file.
## License
TorchEval is BSD licensed, as found in the [LICENSE](LICENSE) file.
Raw data
{
"_id": null,
"home_page": "https://github.com/pytorch/torcheval",
"name": "torcheval-nightly",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "pytorch, evaluation, metrics",
"author": "torcheval team",
"author_email": "yicongd@fb.com",
"download_url": "https://files.pythonhosted.org/packages/03/68/4229b5d401705ca2cbb771498b810b4003520a43754617f57438b08eb704/torcheval-nightly-2024.4.22.tar.gz",
"platform": null,
"description": "# TorchEval\n\n<p align=\"center\">\n<a href=\"https://github.com/pytorch/torcheval/actions?query=branch%3Amain\"><img src=\"https://img.shields.io/github/actions/workflow/status/pytorch/torcheval/.github/workflows/unit_test.yaml?branch=main\" alt=\"build status\"></a>\n<a href=\"https://pypi.org/project/torcheval\"><img src=\"https://img.shields.io/pypi/v/torcheval\" alt=\"pypi version\"></a>\n<a href=\"https://pypi.org/project/torcheval-nightly\"><img src=\"https://img.shields.io/pypi/v/torcheval-nightly?label=nightly\" alt=\"pypi nightly version\"></a>\n<a href=\"https://github.com/pytorch/torcheval/blob/main/LICENSE\"><img src=\"https://img.shields.io/pypi/l/torcheval\" alt=\"bsd license\"></a>\n</div>\n<a href=\"https://pytorch.github.io/torcheval\"><img src=\"https://img.shields.io/badge/docs-main-brightgreen\" alt=\"docs\"></a>\n<p>\n\n**This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.**\n\nA library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.\n\n## Installing TorchEval\nRequires Python >= 3.8 and PyTorch >= 1.11\n\nFrom pip:\n\n```bash\npip install torcheval\n```\n\nFor nighly build version\n```bash\npip install --pre torcheval-nightly\n```\n\nFrom source:\n\n```bash\ngit clone https://github.com/pytorch/torcheval\ncd torcheval\npip install -r requirements.txt\npython setup.py install\n```\n\n## Quick Start\n\nTake a look at the [quickstart notebook](https://github.com/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb), or fork it on [Colab](https://colab.research.google.com/github/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb).\n\nThere are more examples in the [examples](https://github.com/pytorch/torcheval/blob/main/examples) directory:\n\n```bash\ncd torcheval\npython examples/simple_example.py\n```\n\n## Documentation\n\nDocumentation can be found at at [pytorch.org/torcheval](https://pytorch.org/torcheval)\n\n## Using TorchEval\n\nTorchEval can be run on CPU, GPU, and in a multi-process or multi-GPU setting. Metrics are provided in two interfaces, functional and class based. The functional interfaces can be found in `torcheval.metrics.functional` and are useful when your program runs in a single process setting. To use multi-process or multi-gpu configurations, the class-based interfaces, found in `torcheval.metrics` provide a much simpler experience. The class based interfaces also allow you to defer some of the computation of the metric by calling `update()` multiple times before `compute()`. This can be advantageous even in a single process setting due to saved computation overhead.\n\n### Single Process\nFor use in a single process program, the simplest use case utilizes a functional metric. We simply import the metric function and feed in our outputs and targets. The example below shows a minimal PyTorch training loop that evaluates the multiclass accuracy of every fourth batch of data.\n\n#### Functional Version (immediate computation of metric)\n```python\nimport torch\nfrom torcheval.metrics.functional import multiclass_accuracy\n\nNUM_BATCHES = 16\nBATCH_SIZE = 8\nINPUT_SIZE = 10\nNUM_CLASSES = 6\neval_frequency = 4\n\nmodel = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())\noptim = torch.optim.Adagrad(model.parameters(), lr=0.001)\nloss_fn = torch.nn.CrossEntropyLoss()\n\nmetric_history = []\nfor batch in range(NUM_BATCHES):\n input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))\n target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)\n outputs = model(input)\n\n loss = loss_fn(outputs, target)\n optim.zero_grad()\n loss.backward()\n optim.step()\n\n # metric only computed every 4 batches,\n # data from previous three batches is lost\n if (batch + 1) % eval_frequency == 0:\n metric_history.append(multiclass_accuracy(outputs, target))\n```\n### Single Process with Deferred Computation\n\n#### Class Version (enables deferred computation of metric)\n```python\nimport torch\nfrom torcheval.metrics import MulticlassAccuracy\n\nNUM_BATCHES = 16\nBATCH_SIZE = 8\nINPUT_SIZE = 10\nNUM_CLASSES = 6\neval_frequency = 4\n\nmodel = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())\noptim = torch.optim.Adagrad(model.parameters(), lr=0.001)\nloss_fn = torch.nn.CrossEntropyLoss()\nmetric = MulticlassAccuracy()\n\nmetric_history = []\nfor batch in range(NUM_BATCHES):\n input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))\n target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)\n outputs = model(input)\n\n loss = loss_fn(outputs, target)\n optim.zero_grad()\n loss.backward()\n optim.step()\n\n # metric only computed every 4 batches,\n # data from previous three batches is included\n metric.update(input, target)\n if (batch + 1) % eval_frequency == 0:\n metric_history.append(metric.compute())\n # remove old data so that the next call\n # to compute is only based off next 4 batches\n metric.reset()\n```\n\n### Multi-Process or Multi-GPU\nFor usage on multiple devices a minimal example is given below. In the normal `torch.distributed` paradigm, each device is allocated its own process gets a unique numerical ID called a \"global rank\", counting up from 0.\n\n#### Class Version (enables deferred computation and multi-processing)\n```python\nimport torch\nfrom torcheval.metrics.toolkit import sync_and_compute\nfrom torcheval.metrics import MulticlassAccuracy\n\n# Using torch.distributed\nlocal_rank = int(os.environ[\"LOCAL_RANK\"]) #rank on local machine, i.e. unique ID within a machine\nglobal_rank = int(os.environ[\"RANK\"]) #rank in global pool, i.e. unique ID within the entire process group\nworld_size = int(os.environ[\"WORLD_SIZE\"]) #total number of processes or \"ranks\" in the entire process group\n\ndevice = torch.device(\n f\"cuda:{local_rank}\"\n if torch.cuda.is_available() and torch.cuda.device_count() >= world_size\n else \"cpu\"\n)\n\nmetric = MulticlassAccuracy(device=device)\nnum_epochs, num_batches = 4, 8\n\nfor epoch in range(num_epochs):\n for i in range(num_batches):\n input = torch.randint(high=5, size=(10,), device=device)\n target = torch.randint(high=5, size=(10,), device=device)\n\n # Add data to metric locally\n metric.update(input, target)\n\n # metric.compute() will returns metric value from\n # all seen data on the local process since last reset()\n local_compute_result = metric.compute()\n\n # sync_and_compute(metric) syncs metric data across all ranks and computes the metric value\n global_compute_result = sync_and_compute(metric)\n if global_rank == 0:\n print(global_compute_result)\n\n # metric.reset() clears the data on each process so that subsequent\n # calls to compute() only act on new data\n metric.reset()\n```\nSee the [example directory](https://github.com/pytorch/torcheval/tree/main/examples) for more examples.\n\n## Contributing\nWe welcome PRs! See the [CONTRIBUTING](CONTRIBUTING.md) file.\n\n## License\nTorchEval is BSD licensed, as found in the [LICENSE](LICENSE) file.\n",
"bugtrack_url": null,
"license": "BSD-3",
"summary": "A library for providing a simple interface to create new metrics and an easy-to-use toolkit for metric computations and checkpointing.",
"version": "2024.4.22",
"project_urls": {
"Homepage": "https://github.com/pytorch/torcheval"
},
"split_keywords": [
"pytorch",
" evaluation",
" metrics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6209365e7552e024f8711de8b30818a4c7d1b5681aea85c6116cc74722aa5799",
"md5": "48037e4fa045757d51d8c54fe2cfe43e",
"sha256": "94a8743832f2c642565f022034cc5aecbf580b74d59aed641d01a35f36a9673b"
},
"downloads": -1,
"filename": "torcheval_nightly-2024.4.22-py3-none-any.whl",
"has_sig": false,
"md5_digest": "48037e4fa045757d51d8c54fe2cfe43e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 191601,
"upload_time": "2024-04-22T12:08:41",
"upload_time_iso_8601": "2024-04-22T12:08:41.878771Z",
"url": "https://files.pythonhosted.org/packages/62/09/365e7552e024f8711de8b30818a4c7d1b5681aea85c6116cc74722aa5799/torcheval_nightly-2024.4.22-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "03684229b5d401705ca2cbb771498b810b4003520a43754617f57438b08eb704",
"md5": "26c5930bd6881e663acc80e9c3c6066d",
"sha256": "41ae1590da647502be5e2c765d144666a7c727ec0f707a37ded8a44f435f9fa3"
},
"downloads": -1,
"filename": "torcheval-nightly-2024.4.22.tar.gz",
"has_sig": false,
"md5_digest": "26c5930bd6881e663acc80e9c3c6066d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 106545,
"upload_time": "2024-04-22T12:08:47",
"upload_time_iso_8601": "2024-04-22T12:08:47.364140Z",
"url": "https://files.pythonhosted.org/packages/03/68/4229b5d401705ca2cbb771498b810b4003520a43754617f57438b08eb704/torcheval-nightly-2024.4.22.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-22 12:08:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pytorch",
"github_project": "torcheval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "torcheval-nightly"
}
JSON
