<h1 align="center">torchutil</h1>
<div align="center">
[](https://pypi.python.org/pypi/torchutil)
[](https://opensource.org/licenses/MIT)
[](https://pepy.tech/project/torchutil)
General utilities for developing deep learning projects using PyTorch
`pip install torchutil`
</div>
## Table of contents
- [Checkpoint](#checkpoint)
* [`torchutil.checkpoint.best_path`](#torchutilcheckpointbest_path)
* [`torchutil.checkpoint.latest_path`](#torchutilcheckpointlatest_path)
* [`torchutil.checkpoint.load`](#torchutilcheckpointload)
* [`torchutil.checkpoint.save`](#torchutilcheckpointsave)
- [Cuda](#cuda)
* [`torchutil.cuda.utilization`](#torchutilcudautilization)
- [Download](#download)
* [`torchutil.download.file`](#torchutildownloadfile)
* [`torchutil.download.tarbz2`](#torchutildownloadtarbz2)
* [`torchutil.download.targz`](#torchutildownloadtargz)
* [`torchutil.download.zip`](#torchutildownloadzip)
- [Gradients](#gradients)
* [`torchutil.gradients.stats`](#torchutilgradientsstats)
- [Inference](#inference)
* [`torchutil.inference.context`](#torchutilinferencecontext)
- [Iterator](#iterator)
* [`torchutil.iterator`](#torchutiliterator)
* [`torchutil.multiprocess_iterator`](#torchutilmultiprocess_iterator)
- [Mask](#mask)
* [`torchutil.mask.from_lengths`](#torchutilmaskfromlengths)
- [Metrics](#metrics)
* [`torchutil.metrics.Accuracy`](#torchutilmetricsaccuracy)
* [`torchutil.metrics.Average`](#torchutilmetricsaverage)
* [`torchutil.metrics.F1`](#torchutilmetricsf1)
* [`torchutil.metrics.L1`](#torchutilmetricsl1)
* [`torchutil.metrics.MeanStd`](#torchutilmetricsmeanstd)
* [`torchutil.metrics.Precision`](#torchutilmetricsprecision)
* [`torchutil.metrics.PearsonCorrelation`](#torchutilmetricspearsoncorrelation)
* [`torchutil.metrics.Recall`](#torchutilmetricsrecall)
* [`torchutil.metrics.RMSE`](#torchutilmetricsrmse)
- [Notify](#notify)
* [`torchutil.notify`](#torchutilnotify)
- [Paths](#paths)
* [`torchutil.paths.chdir`](#torchutilpathschdir)
* [`torchutil.paths.measure`](#torchutilpathsmeasure)
* [`torchutil.paths.purge`](#torchutilpathspurge)
- [Tensorboard](#tensorboard)
* [`torchutil.tensorboard.update`](#torchutiltensorboardupdate)
- [Time](#time)
* [`torchutil.time.context`](#torchutiltimecontext)
* [`torchutil.time.results`](#torchutiltimeresults)
* [`torchutil.time.reset`](#torchutiltimereset)
## Checkpoint
```python
import torch
import torchutil
# Checkpoint location
file = 'model.pt'
# Initialize PyTorch model
model = torch.nn.Sequential(torch.nn.Conv1d())
# Initialize optimizer
optimizer = torch.nn.Adam(model.parameters())
# Save
torchutil.checkpoint.save(file, model, optimizer, step=0, epoch=0)
# Load for training
model, optimizer, state = torchutil.checkpoint.load(file, model, optimizer)
step, epoch = state['step'], state['epoch']
# Load for inference
model, *_ = torchutil.checkpoint.load(file, model, optimizer)
```
### `torchutil.checkpoint.best_path`
```python
def best_path(
directory: Union[str, bytes, os.PathLike],
glob: str = '*.pt',
best_fn: Callable = highest_score
) -> Tuple[Union[str, bytes, os.PathLike], float]:
"""Retrieve the path to the best checkpoint
Arguments
directory - The directory to search for checkpoint files
glob - The regular expression matching checkpoints
best_fn - Takes a list of checkpoint paths and returns the latest
Default assumes checkpoint names are training step count.
Returns
best_file - The filename of the checkpoint with the best score
best_score - The corresponding score
"""
```
### `torchutil.checkpoint.latest_path`
```python
def latest_path(
directory: Union[str, bytes, os.PathLike],
glob: str = '*.pt',
latest_fn: Callable = largest_number_filename,
) -> Union[str, bytes, os.PathLike]:
"""Retrieve the path to the most recent checkpoint in a directory
Arguments
directory - The directory to search for checkpoint files
glob - The regular expression matching checkpoints
latest_fn - Takes a list of checkpoint paths and returns the latest.
Default assumes checkpoint names are training step count.
Returns
The latest checkpoint in directory according to latest_fn
"""
```
### `torchutil.checkpoint.load`
```python
def load(
file: Union[str, bytes, os.PathLike],
model: torch.nn.Module,
optimizer: Optional[torch.optim.Optimizer] = None,
map_location: str = 'cpu') -> Tuple[
torch.nn.Module,
Union[None, torch.optim.Optimizer],
Dict
]:
"""Load model checkpoint
Arguments
file - The checkpoint file
model - The PyTorch model
optimizer - Optional PyTorch optimizer for training
map_location - The device to load the checkpoint on
Returns
model - The model with restored weights
optimizer - Optional optimizer with restored parameters
state - Additional values that the user defined during save
"""
```
### `torchutil.checkpoint.save`
```python
def save(
file: Union[str, bytes, os.PathLike],
model: torch.nn.Module,
optimizer: torch.optim.Optimizer,
**kwargs):
"""Save training checkpoint to disk
Arguments
file - The checkpoint file
model - The PyTorch model
optimizer - The PyTorch optimizer
kwargs - Additional values to save
"""
```
## Cuda
```python
import torch
import torchutil
# Directory to write Tensorboard files
directory = 'tensorboard'
# Training step
step = 0
# Log VRAM utilization in MB to Tensorboard
torchutil.tensorboard.update(
directory,
step,
scalars=torchutil.cuda.utilization(torch.device('cuda:0'), 'MB'))
```
### `torchutil.cuda.utilization`
```python
def utilization(device: torch.Device, unit: str ='B') -> Dict[str, float]:
"""Get the current VRAM utilization of a specified device
Arguments
device
The device to query for VRAM utilization
unit
Unit of memory utilization (bytes to terabytes); default bytes
Returns
Allocated and reserved VRAM utilization in the specified unit
"""
```
## Download
### `torchutil.download.file`
```python
def file(url: 'str', path: Union[str, bytes, os.PathLike]):
"""Download file from url
Arguments
url - The URL to download
path - The location to save results
"""
```
### `torchutil.download.tarbz2`
```python
def tarbz2(url: 'str', path: Union[str, bytes, os.PathLike]):
"""Download and extract tar bz2 file to location
Arguments
url - The URL to download
path - The location to save results
"""
```
### `torchutil.download.targz`
```python
def targz(url: 'str', path: Union[str, bytes, os.PathLike]):
"""Download and extract tar gz file to location
Arguments
url - The URL to download
path - The location to save results
"""
```
### `torchutil.download.zip`
```python
def zip(url: 'str', path: Union[str, bytes, os.PathLike]):
"""Download and extract zip file to location
Arguments
url - The URL to download
path - The location to save results
"""
```
## Gradients
```python
import torch
import torchutil
# Directory to write Tensorboard files
directory = 'tensorboard'
# Training step
step = 0
# Setup model and optimizer
# ...
# Compute forward pass and loss
# ...
# Zero gradients
optimizer.zero_grad()
# Compute gradients
loss.backward()
# Monitor gradients on tensorboard
torchutil.tensorboard.update(
directory,
step,
scalars=torchutil.gradients.stats(model))
# Apply gradient update
optimizer.step()
```
### `torchutil.gradients.stats`
```python
def stats(model: torch.nn.Module) -> Dict[str, float]:
"""Get gradient statistics
Arguments
model
The torch model
Returns
The L2 norm, maximum, and minimum gradients
"""
```
## Inference
### `torchutil.inference.context`
```python
@contextlib.contextmanager
def context(model: torch.nn.Module, autocast: bool = True) -> None:
"""Inference-time handling of model training flag and optimizations
Arguments
model
The torch model performing inference
autocast
Whether to use mixed precision
"""
```
## Iterator
```python
import time
import torchutil
def wait(seconds):
time.sleep(seconds)
n = 8
iterable = range(n)
# Monitor single-process job
for i in torchutil.iterator(iterable, message='single-process'):
wait(i)
# Monitor multi-process job
torchutil.multiprocess_iterator(wait, iterable, message='multi-process')
```
### `torchutil.iterator`
```python
def iterator(
iterable: Iterable,
message: Optional[str] = None,
initial: int = 0,
total: Optional[int] = None
) -> Iterable:
"""Create a tqdm iterator
Arguments
iterable
Items to iterate over
message
Static message to display
initial
Position to display corresponding to index zero of iterable
total
Length of the iterable; defaults to len(iterable)
Returns
Monitored iterable
"""
```
### `torchutil.multiprocess_iterator`
```python
def multiprocess_iterator(
process: Callable,
iterable: Iterable,
message: Optional[str] = None,
initial: int = 0,
total: Optional[int] = None,
num_workers: int = os.cpu_count(),
worker_chunk_size: Optional[int] = None
) -> List:
"""Create a multiprocess tqdm iterator
Arguments
process
The single-argument function called by each multiprocess worker
iterable
Items to iterate over
message
Static message to display
initial
Position to display corresponding to index zero of iterable
total
Length of the iterable; defaults to len(iterable)
num_workers
Multiprocessing pool size; defaults to number of logical CPU cores
worker_chunk_size
Number of items sent to each multiprocessing worker
Returns
Return values of calling process on each item, in original order
"""
```
## Mask
### `torchutil.mask.from_lengths`
```python
def from_lengths(lengths: torch.Tensor) -> torch.Tensor:
"""Create boolean mask from sequence lengths
Arguments
lengths
The integer-type sequence lengths
Returns
mask
The boolean-type sequence mask
"""
```
## Metrics
```python
import torch
import torchutil
# Define a custom, batch-updating loss metric
class Loss(torchutil.metrics.Average):
def update(self, predicted, target):
# Compute your loss and the number of elements to average over
loss = ...
count = ...
super().update(loss, count)
# Instantiate metrics
loss = Loss()
rmse = torchutil.metrics.RMSE()
# Generator that produces batches of predicted and target tensors
iterable = ...
# Update metrics
for predicted_tensor, target_tensor in iterable:
loss.update(predicted_tensor, target_tensor)
rmse.update(predicted_tensor, target_tensor)
# Get results
print('loss': loss())
print('rmse': rmse())
```
### `torchutil.metrics.Accuracy`
```python
class Accuracy(Metric):
"""Batch-updating accuracy metric"""
def __call__(self)-> float:
"""Retrieve the current accuracy value
Returns
The current accuracy value
"""
def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:
"""Update accuracy
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset accuracy"""
```
### `torchutil.metrics.Average`
```python
class Average(Metric):
"""Batch-updating average metric"""
def __call__(self)-> float:
"""Retrieve the current average value
Returns
The current average value
"""
def update(self, values: torch.Tensor, count: int) -> None:
"""Update running average
Arguments
values
The values to average
target
The number of values
"""
def reset(self) -> None:
"""Reset running average"""
```
### `torchutil.metrics.F1`
```python
class F1(Metric):
"""Batch-updating F1 score"""
def __call__(self) -> float:
"""Retrieve the current F1 value
Returns
The current F1 value
"""
def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:
"""Update F1
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset F1"""
```
### `torchutil.metrics.L1`
```python
class L1(Metric):
"""Batch updating L1 score"""
def __call__(self) -> float:
"""Retrieve the current L1 value
Returns
The current L1 value
"""
def update(self, predicted, target) -> None:
"""Update L1
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset L1"""
```
### `torchutil.metrics.MeanStd`
```python
class MeanStd(Metric):
"""Batch updating mean and standard deviation"""
def __call__(self) -> Tuple[float, float]:
"""Retrieve the current mean and standard deviation
Returns
The current mean and standard deviation
"""
def update(self, values: torch.Tensor) -> None:
"""Update mean and standard deviation
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset mean and standard deviation"""
```
### `torchutil.metrics.PearsonCorrelation`
```python
class PearsonCorrelation(Metric):
"""Batch-updating Pearson correlation"""
def __init__(
self,
predicted_mean: float,
predicted_std: float,
target_mean: float,
target_std: float
) -> None:
"""
Arguments
predicted_mean - Mean of predicted values
predicted_std - Standard deviation of predicted values
target_mean - Mean of target values
target_std - Standard deviation of target values
"""
def __call__(self) -> float:
"""Retrieve the current correlation value
Returns
The current correlation value
"""
def update(self, predicted, target) -> None:
"""Update Pearson correlation
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset Pearson correlation"""
```
### `torchutil.metrics.Precision`
```python
class Precision(Metric):
"""Batch-updating precision metric"""
def __call__(self) -> float:
"""Retrieve the current precision value
Returns
The current precision value
"""
def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:
"""Update precision
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset precision"""
```
### `torchutil.metrics.Recall`
```python
class Recall(Metric):
"""Batch-updating recall metric"""
def __call__(self) -> float:
"""Retrieve the current recall value
The current recall value
"""
def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:
"""Update recall
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset recall"""
```
### `torchutil.metrics.RMSE`
```python
class RMSE(Metric):
"""Batch-updating RMSE metric"""
def __call__(self) -> float:
"""Retrieve the current rmse value
Returns
The current rmse value
"""
def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:
"""Update RMSE
Arguments
predicted
The model prediction
target
The corresponding ground truth
"""
def reset(self) -> None:
"""Reset RMSE"""
```
## Notify
To use the `torchutil` notification system, set the `PYTORCH_NOTIFICATION_URL`
environment variable to a supported webhook as explained in
[the Apprise documentation](https://pypi.org/project/apprise/).
```python
import torchutil
# Send notification when function returns
@torchutil.notify('train')
def train():
...
# Equivalent using context manager
def train():
with torchutil.notify('train'):
...
```
### `torchutil.notify`
```python
@contextlib.contextmanager
def notify(
description: str,
track_time: bool = True,
notify_on_fail: bool = True):
"""Context manager for sending job notifications
Arguments
description - The name of the job being run
track_time - Whether to report time elapsed
notify_on_fail - Whether to send a notification on failure
"""
```
## Paths
### `torchutil.paths.chdir`
```python
@contextlib.contextmanager
def chdir(directory: Union[str, bytes, os.PathLike]) -> None:
"""Context manager for changing the current working directory
Arguments
directory
The desired working directory
"""
```
This function is both a context manager and decorator.
```python
import tempfile
from pathlib import Path
import torchutil
# Create a directory
directory = tempfile.TemporaryDirectory()
# Create a file
file = 'tmp.txt'
(Path(directory.name) / file).touch()
# File is not in current working directory
assert not Path(file).exists()
# Change working directory using context manager
with torchutil.paths.chdir(directory.name):
assert Path(file).exists()
# File is not in current working directory
assert not Path(file).exists()
# Change working directory using decorator
@torchutil.paths.chdir(directory.name)
def exists(file):
assert Path(file).exists()
exists(file)
# File is not in current working directory
assert not Path(file).exists()
# Remove temporary paths
directory.cleanup()
```
### `torchutil.paths.measure`
```python
def measure(
globs: Optional[List[Union[str, List[str]]]] = None,
roots: Optional[
List[
Union[
Union[str, bytes, os.PathLike],
List[Union[str, bytes, os.PathLike]]
]
]
] = None,
recursive: bool = False,
unit='B'
) -> Union[int, float]:
"""Measure data usage of files and directories
Arguments
globs
Globs matching paths to measure
roots
Directories to apply glob searches; current directory by default
recursive
Apply globs to all subdirectories of root directories
unit
Unit of memory utilization (bytes to terabytes); default bytes
Returns
Data usage in the specified unit
"""
```
This function also has a command-line interface.
```
python -m torchutil.paths.measure \
[-h] \
--globs GLOBS \
[--roots ROOTS] \
[--recursive] \
[--unit]
Measure data usage of files and directories
arguments:
--globs GLOBS
Globs matching paths to measure
optional arguments:
-h, --help
show this help message and exit
--roots ROOTS
Directories to apply glob searches; current directory by default
--recursive
Apply globs to all subdirectories of root directories
--unit
Unit of memory utilization (bytes to terabytes); default bytes
```
### `torchutil.paths.purge`
```python
def purge(
globs: Optional[List[Union[str, List[str]]]] = None,
roots: Optional[
List[
Union[
Union[str, bytes, os.PathLike],
List[Union[str, bytes, os.PathLike]]
]
]
] = None,
recursive: bool = False,
force: bool = False
) -> None:
"""Remove all files and directories within directory matching glob
Arguments
globs
Globs matching paths to delete
roots
Directories to apply glob searches; current directory by default
recursive
Apply globs to all subdirectories of root directories
force
Skip user confirmation of deletion
"""
```
This function also has a command-line interface.
```
python -m torchutil.paths.purge \
[-h] \
--globs GLOBS \
[--roots ROOTS] \
[--recursive] \
[--force]
Remove files and directories
arguments:
--globs GLOBS
Globs matching paths to delete
optional arguments:
-h, --help
show this help message and exit
--roots ROOTS
Directories to apply glob searches; current directory by default
--recursive
Apply globs to all subdirectories of root directories
--force
Skip user confirmation of deletion
```
## Tensorboard
```python
import matplotlib.pyplot as plt
import torch
import torchutil
# Directory to write Tensorboard files
directory = 'tensorboard'
# Training step
step = 0
# Example audio
audio = torch.zeros(1, 16000)
sample_rate = 16000
# Example figure
figure = plt.figure()
plt.plot([0, 1, 2, 3])
# Example image
image = torch.zeros(256, 256, 3)
# Example scalar
loss = 0
# Update Tensorboard
torchutil.tensorboard.update(
directory,
step,
audio={'audio': audio},
sample_rate=sample_rate,
figures={'figure': figure},
images={'image': image},
scalars={'loss': loss})
```
### `torchutil.tensorboard.update`
```python
def update(
directory: Union[str, bytes, os.PathLike],
step: int,
audio: Optional[Dict[str, torch.Tensor]] = None,
sample_rate: Optional[int] = None,
figures: Optional[Dict[str, matplotlib.figure.Figure]] = None,
images: Optional[Dict[str, torch.Tensor]] = None,
scalars: Optional[Dict[str, Union[float, int, torch.Tensor]]] = None):
"""Update Tensorboard
Arguments
directory - Directory to write Tensorboard files
step - Training step
audio - Optional dictionary of 2D audio tensors to monitor
sample_rate - Audio sample rate; required if audio is not None
figures - Optional dictionary of Matplotlib figures to monitor
images - Optional dictionary of 3D image tensors to monitor
scalars - Optional dictionary of scalars to monitor
"""
```
## Time
```python
import time
import torchutil
# Perform timing
with torchutil.time.context('outer'):
time.sleep(1)
for i in range(2):
time.sleep(1)
with torchutil.time.context('inner'):
time.sleep(1)
# Prints {'inner': 2.0020763874053955, 'outer': 5.005248308181763, 'total': 5.005248308181763}
print(torchutil.time.results())
```
### `torchutil.time.context`
```python
@contextlib.contextmanager
def context(name: str):
"""Wrapper to handle context changes of global timer
Arguments
name - Name of the timer to add time to
"""
```
### `torchutil.time.reset`
```python
def reset():
"""Clear timer state"""
```
### `torchutil.time.results`
```python
def results() -> dict:
"""Get timing results
Returns
Timing results: {name: elapsed_time} for all names
"""
```
Raw data
{
"_id": null,
"home_page": "https://github.com/maxrmorrison/torchutil",
"name": "torchutil",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "pytorch, utility, training",
"author": "Max Morrison",
"author_email": "maxrmorrison@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/60/a5/f73abbc76d85d2295ab4057dd12eb4cea434d77a0c9e1e5f499be45e6b80/torchutil-0.0.14.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">torchutil</h1>\n<div align=\"center\">\n\n[](https://pypi.python.org/pypi/torchutil)\n[](https://opensource.org/licenses/MIT)\n[](https://pepy.tech/project/torchutil)\n\nGeneral utilities for developing deep learning projects using PyTorch\n\n`pip install torchutil`\n</div>\n\n\n## Table of contents\n\n- [Checkpoint](#checkpoint)\n * [`torchutil.checkpoint.best_path`](#torchutilcheckpointbest_path)\n * [`torchutil.checkpoint.latest_path`](#torchutilcheckpointlatest_path)\n * [`torchutil.checkpoint.load`](#torchutilcheckpointload)\n * [`torchutil.checkpoint.save`](#torchutilcheckpointsave)\n- [Cuda](#cuda)\n * [`torchutil.cuda.utilization`](#torchutilcudautilization)\n- [Download](#download)\n * [`torchutil.download.file`](#torchutildownloadfile)\n * [`torchutil.download.tarbz2`](#torchutildownloadtarbz2)\n * [`torchutil.download.targz`](#torchutildownloadtargz)\n * [`torchutil.download.zip`](#torchutildownloadzip)\n- [Gradients](#gradients)\n * [`torchutil.gradients.stats`](#torchutilgradientsstats)\n- [Inference](#inference)\n * [`torchutil.inference.context`](#torchutilinferencecontext)\n- [Iterator](#iterator)\n * [`torchutil.iterator`](#torchutiliterator)\n * [`torchutil.multiprocess_iterator`](#torchutilmultiprocess_iterator)\n- [Mask](#mask)\n * [`torchutil.mask.from_lengths`](#torchutilmaskfromlengths)\n- [Metrics](#metrics)\n * [`torchutil.metrics.Accuracy`](#torchutilmetricsaccuracy)\n * [`torchutil.metrics.Average`](#torchutilmetricsaverage)\n * [`torchutil.metrics.F1`](#torchutilmetricsf1)\n * [`torchutil.metrics.L1`](#torchutilmetricsl1)\n * [`torchutil.metrics.MeanStd`](#torchutilmetricsmeanstd)\n * [`torchutil.metrics.Precision`](#torchutilmetricsprecision)\n * [`torchutil.metrics.PearsonCorrelation`](#torchutilmetricspearsoncorrelation)\n * [`torchutil.metrics.Recall`](#torchutilmetricsrecall)\n * [`torchutil.metrics.RMSE`](#torchutilmetricsrmse)\n- [Notify](#notify)\n * [`torchutil.notify`](#torchutilnotify)\n- [Paths](#paths)\n * [`torchutil.paths.chdir`](#torchutilpathschdir)\n * [`torchutil.paths.measure`](#torchutilpathsmeasure)\n * [`torchutil.paths.purge`](#torchutilpathspurge)\n- [Tensorboard](#tensorboard)\n * [`torchutil.tensorboard.update`](#torchutiltensorboardupdate)\n- [Time](#time)\n * [`torchutil.time.context`](#torchutiltimecontext)\n * [`torchutil.time.results`](#torchutiltimeresults)\n * [`torchutil.time.reset`](#torchutiltimereset)\n\n\n## Checkpoint\n\n```python\nimport torch\nimport torchutil\n\n# Checkpoint location\nfile = 'model.pt'\n\n# Initialize PyTorch model\nmodel = torch.nn.Sequential(torch.nn.Conv1d())\n\n# Initialize optimizer\noptimizer = torch.nn.Adam(model.parameters())\n\n# Save\ntorchutil.checkpoint.save(file, model, optimizer, step=0, epoch=0)\n\n# Load for training\nmodel, optimizer, state = torchutil.checkpoint.load(file, model, optimizer)\nstep, epoch = state['step'], state['epoch']\n\n# Load for inference\nmodel, *_ = torchutil.checkpoint.load(file, model, optimizer)\n```\n\n\n### `torchutil.checkpoint.best_path`\n\n```python\ndef best_path(\n directory: Union[str, bytes, os.PathLike],\n glob: str = '*.pt',\n best_fn: Callable = highest_score\n) -> Tuple[Union[str, bytes, os.PathLike], float]:\n \"\"\"Retrieve the path to the best checkpoint\n\n Arguments\n directory - The directory to search for checkpoint files\n glob - The regular expression matching checkpoints\n best_fn - Takes a list of checkpoint paths and returns the latest\n Default assumes checkpoint names are training step count.\n\n Returns\n best_file - The filename of the checkpoint with the best score\n best_score - The corresponding score\n \"\"\"\n```\n\n\n### `torchutil.checkpoint.latest_path`\n\n```python\ndef latest_path(\n directory: Union[str, bytes, os.PathLike],\n glob: str = '*.pt',\n latest_fn: Callable = largest_number_filename,\n ) -> Union[str, bytes, os.PathLike]:\n \"\"\"Retrieve the path to the most recent checkpoint in a directory\n\n Arguments\n directory - The directory to search for checkpoint files\n glob - The regular expression matching checkpoints\n latest_fn - Takes a list of checkpoint paths and returns the latest.\n Default assumes checkpoint names are training step count.\n\n Returns\n The latest checkpoint in directory according to latest_fn\n \"\"\"\n```\n\n\n### `torchutil.checkpoint.load`\n\n```python\ndef load(\n file: Union[str, bytes, os.PathLike],\n model: torch.nn.Module,\n optimizer: Optional[torch.optim.Optimizer] = None,\n map_location: str = 'cpu') -> Tuple[\n torch.nn.Module,\n Union[None, torch.optim.Optimizer],\n Dict\n ]:\n \"\"\"Load model checkpoint\n\n Arguments\n file - The checkpoint file\n model - The PyTorch model\n optimizer - Optional PyTorch optimizer for training\n map_location - The device to load the checkpoint on\n\n Returns\n model - The model with restored weights\n optimizer - Optional optimizer with restored parameters\n state - Additional values that the user defined during save\n \"\"\"\n```\n\n### `torchutil.checkpoint.save`\n\n```python\ndef save(\n file: Union[str, bytes, os.PathLike],\n model: torch.nn.Module,\n optimizer: torch.optim.Optimizer,\n **kwargs):\n \"\"\"Save training checkpoint to disk\n\n Arguments\n file - The checkpoint file\n model - The PyTorch model\n optimizer - The PyTorch optimizer\n kwargs - Additional values to save\n \"\"\"\n```\n\n## Cuda\n\n```python\nimport torch\nimport torchutil\n\n# Directory to write Tensorboard files\ndirectory = 'tensorboard'\n\n# Training step\nstep = 0\n\n# Log VRAM utilization in MB to Tensorboard\ntorchutil.tensorboard.update(\n directory,\n step,\n scalars=torchutil.cuda.utilization(torch.device('cuda:0'), 'MB'))\n```\n\n\n### `torchutil.cuda.utilization`\n\n```python\ndef utilization(device: torch.Device, unit: str ='B') -> Dict[str, float]:\n \"\"\"Get the current VRAM utilization of a specified device\n\n Arguments\n device\n The device to query for VRAM utilization\n unit\n Unit of memory utilization (bytes to terabytes); default bytes\n\n Returns\n Allocated and reserved VRAM utilization in the specified unit\n \"\"\"\n```\n\n\n## Download\n\n### `torchutil.download.file`\n\n```python\ndef file(url: 'str', path: Union[str, bytes, os.PathLike]):\n \"\"\"Download file from url\n\n Arguments\n url - The URL to download\n path - The location to save results\n \"\"\"\n```\n\n\n### `torchutil.download.tarbz2`\n\n```python\ndef tarbz2(url: 'str', path: Union[str, bytes, os.PathLike]):\n \"\"\"Download and extract tar bz2 file to location\n\n Arguments\n url - The URL to download\n path - The location to save results\n \"\"\"\n```\n\n\n### `torchutil.download.targz`\n\n```python\ndef targz(url: 'str', path: Union[str, bytes, os.PathLike]):\n \"\"\"Download and extract tar gz file to location\n\n Arguments\n url - The URL to download\n path - The location to save results\n \"\"\"\n```\n\n\n### `torchutil.download.zip`\n\n```python\ndef zip(url: 'str', path: Union[str, bytes, os.PathLike]):\n \"\"\"Download and extract zip file to location\n\n Arguments\n url - The URL to download\n path - The location to save results\n \"\"\"\n```\n\n\n## Gradients\n\n```python\nimport torch\nimport torchutil\n\n# Directory to write Tensorboard files\ndirectory = 'tensorboard'\n\n# Training step\nstep = 0\n\n# Setup model and optimizer\n# ...\n\n# Compute forward pass and loss\n# ...\n\n# Zero gradients\noptimizer.zero_grad()\n\n# Compute gradients\nloss.backward()\n\n# Monitor gradients on tensorboard\ntorchutil.tensorboard.update(\n directory,\n step,\n scalars=torchutil.gradients.stats(model))\n\n# Apply gradient update\noptimizer.step()\n```\n\n\n### `torchutil.gradients.stats`\n\n```python\ndef stats(model: torch.nn.Module) -> Dict[str, float]:\n \"\"\"Get gradient statistics\n\n Arguments\n model\n The torch model\n\n Returns\n The L2 norm, maximum, and minimum gradients\n \"\"\"\n```\n\n\n## Inference\n\n### `torchutil.inference.context`\n\n```python\n@contextlib.contextmanager\ndef context(model: torch.nn.Module, autocast: bool = True) -> None:\n \"\"\"Inference-time handling of model training flag and optimizations\n\n Arguments\n model\n The torch model performing inference\n autocast\n Whether to use mixed precision\n \"\"\"\n```\n\n\n## Iterator\n\n```python\nimport time\nimport torchutil\n\ndef wait(seconds):\n time.sleep(seconds)\n\nn = 8\niterable = range(n)\n\n# Monitor single-process job\nfor i in torchutil.iterator(iterable, message='single-process'):\n wait(i)\n\n# Monitor multi-process job\ntorchutil.multiprocess_iterator(wait, iterable, message='multi-process')\n```\n\n\n### `torchutil.iterator`\n\n```python\ndef iterator(\n iterable: Iterable,\n message: Optional[str] = None,\n initial: int = 0,\n total: Optional[int] = None\n) -> Iterable:\n \"\"\"Create a tqdm iterator\n\n Arguments\n iterable\n Items to iterate over\n message\n Static message to display\n initial\n Position to display corresponding to index zero of iterable\n total\n Length of the iterable; defaults to len(iterable)\n\n Returns\n Monitored iterable\n \"\"\"\n```\n\n\n### `torchutil.multiprocess_iterator`\n\n```python\ndef multiprocess_iterator(\n process: Callable,\n iterable: Iterable,\n message: Optional[str] = None,\n initial: int = 0,\n total: Optional[int] = None,\n num_workers: int = os.cpu_count(),\n worker_chunk_size: Optional[int] = None\n) -> List:\n \"\"\"Create a multiprocess tqdm iterator\n\n Arguments\n process\n The single-argument function called by each multiprocess worker\n iterable\n Items to iterate over\n message\n Static message to display\n initial\n Position to display corresponding to index zero of iterable\n total\n Length of the iterable; defaults to len(iterable)\n num_workers\n Multiprocessing pool size; defaults to number of logical CPU cores\n worker_chunk_size\n Number of items sent to each multiprocessing worker\n\n Returns\n Return values of calling process on each item, in original order\n \"\"\"\n```\n\n\n## Mask\n\n### `torchutil.mask.from_lengths`\n\n```python\ndef from_lengths(lengths: torch.Tensor) -> torch.Tensor:\n \"\"\"Create boolean mask from sequence lengths\n\n Arguments\n lengths\n The integer-type sequence lengths\n\n Returns\n mask\n The boolean-type sequence mask\n \"\"\"\n```\n\n\n## Metrics\n\n```python\nimport torch\nimport torchutil\n\n# Define a custom, batch-updating loss metric\nclass Loss(torchutil.metrics.Average):\n def update(self, predicted, target):\n\n # Compute your loss and the number of elements to average over\n loss = ...\n count = ...\n\n super().update(loss, count)\n\n# Instantiate metrics\nloss = Loss()\nrmse = torchutil.metrics.RMSE()\n\n# Generator that produces batches of predicted and target tensors\niterable = ...\n\n# Update metrics\nfor predicted_tensor, target_tensor in iterable:\n loss.update(predicted_tensor, target_tensor)\n rmse.update(predicted_tensor, target_tensor)\n\n# Get results\nprint('loss': loss())\nprint('rmse': rmse())\n```\n\n\n### `torchutil.metrics.Accuracy`\n\n```python\nclass Accuracy(Metric):\n \"\"\"Batch-updating accuracy metric\"\"\"\n\n def __call__(self)-> float:\n \"\"\"Retrieve the current accuracy value\n\n Returns\n The current accuracy value\n \"\"\"\n\n def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:\n \"\"\"Update accuracy\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset accuracy\"\"\"\n```\n\n\n### `torchutil.metrics.Average`\n\n```python\nclass Average(Metric):\n \"\"\"Batch-updating average metric\"\"\"\n\n def __call__(self)-> float:\n \"\"\"Retrieve the current average value\n\n Returns\n The current average value\n \"\"\"\n\n def update(self, values: torch.Tensor, count: int) -> None:\n \"\"\"Update running average\n\n Arguments\n values\n The values to average\n target\n The number of values\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset running average\"\"\"\n```\n\n\n### `torchutil.metrics.F1`\n\n```python\nclass F1(Metric):\n \"\"\"Batch-updating F1 score\"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current F1 value\n\n Returns\n The current F1 value\n \"\"\"\n\n def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:\n \"\"\"Update F1\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset F1\"\"\"\n```\n\n\n### `torchutil.metrics.L1`\n\n```python\nclass L1(Metric):\n \"\"\"Batch updating L1 score\"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current L1 value\n\n Returns\n The current L1 value\n \"\"\"\n\n def update(self, predicted, target) -> None:\n \"\"\"Update L1\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset L1\"\"\"\n```\n\n\n### `torchutil.metrics.MeanStd`\n\n```python\nclass MeanStd(Metric):\n \"\"\"Batch updating mean and standard deviation\"\"\"\n\n def __call__(self) -> Tuple[float, float]:\n \"\"\"Retrieve the current mean and standard deviation\n\n Returns\n The current mean and standard deviation\n \"\"\"\n\n def update(self, values: torch.Tensor) -> None:\n \"\"\"Update mean and standard deviation\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset mean and standard deviation\"\"\"\n```\n\n\n### `torchutil.metrics.PearsonCorrelation`\n\n```python\nclass PearsonCorrelation(Metric):\n \"\"\"Batch-updating Pearson correlation\"\"\"\n\n def __init__(\n self,\n predicted_mean: float,\n predicted_std: float,\n target_mean: float,\n target_std: float\n ) -> None:\n \"\"\"\n Arguments\n predicted_mean - Mean of predicted values\n predicted_std - Standard deviation of predicted values\n target_mean - Mean of target values\n target_std - Standard deviation of target values\n \"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current correlation value\n\n Returns\n The current correlation value\n \"\"\"\n\n def update(self, predicted, target) -> None:\n \"\"\"Update Pearson correlation\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset Pearson correlation\"\"\"\n```\n\n\n### `torchutil.metrics.Precision`\n\n```python\nclass Precision(Metric):\n \"\"\"Batch-updating precision metric\"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current precision value\n\n Returns\n The current precision value\n \"\"\"\n\n def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:\n \"\"\"Update precision\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset precision\"\"\"\n```\n\n\n### `torchutil.metrics.Recall`\n\n```python\nclass Recall(Metric):\n \"\"\"Batch-updating recall metric\"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current recall value\n\n\n The current recall value\n \"\"\"\n\n def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:\n \"\"\"Update recall\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset recall\"\"\"\n```\n\n\n### `torchutil.metrics.RMSE`\n\n```python\nclass RMSE(Metric):\n \"\"\"Batch-updating RMSE metric\"\"\"\n\n def __call__(self) -> float:\n \"\"\"Retrieve the current rmse value\n\n Returns\n The current rmse value\n \"\"\"\n\n def update(self, predicted: torch.Tensor, target: torch.Tensor) -> None:\n \"\"\"Update RMSE\n\n Arguments\n predicted\n The model prediction\n target\n The corresponding ground truth\n \"\"\"\n\n def reset(self) -> None:\n \"\"\"Reset RMSE\"\"\"\n```\n\n\n## Notify\n\nTo use the `torchutil` notification system, set the `PYTORCH_NOTIFICATION_URL`\nenvironment variable to a supported webhook as explained in\n[the Apprise documentation](https://pypi.org/project/apprise/).\n\n```python\nimport torchutil\n\n# Send notification when function returns\n@torchutil.notify('train')\ndef train():\n ...\n\n# Equivalent using context manager\ndef train():\n with torchutil.notify('train'):\n ...\n```\n\n\n### `torchutil.notify`\n\n```python\n@contextlib.contextmanager\ndef notify(\n description: str,\n track_time: bool = True,\n notify_on_fail: bool = True):\n \"\"\"Context manager for sending job notifications\n\n Arguments\n description - The name of the job being run\n track_time - Whether to report time elapsed\n notify_on_fail - Whether to send a notification on failure\n \"\"\"\n```\n\n\n## Paths\n\n### `torchutil.paths.chdir`\n\n```python\n@contextlib.contextmanager\ndef chdir(directory: Union[str, bytes, os.PathLike]) -> None:\n \"\"\"Context manager for changing the current working directory\n\n Arguments\n directory\n The desired working directory\n \"\"\"\n```\n\nThis function is both a context manager and decorator.\n\n```python\nimport tempfile\nfrom pathlib import Path\n\nimport torchutil\n\n# Create a directory\ndirectory = tempfile.TemporaryDirectory()\n\n# Create a file\nfile = 'tmp.txt'\n(Path(directory.name) / file).touch()\n\n# File is not in current working directory\nassert not Path(file).exists()\n\n# Change working directory using context manager\nwith torchutil.paths.chdir(directory.name):\n assert Path(file).exists()\n\n# File is not in current working directory\nassert not Path(file).exists()\n\n# Change working directory using decorator\n@torchutil.paths.chdir(directory.name)\ndef exists(file):\n assert Path(file).exists()\nexists(file)\n\n# File is not in current working directory\nassert not Path(file).exists()\n\n# Remove temporary paths\ndirectory.cleanup()\n```\n\n\n### `torchutil.paths.measure`\n\n```python\ndef measure(\n globs: Optional[List[Union[str, List[str]]]] = None,\n roots: Optional[\n List[\n Union[\n Union[str, bytes, os.PathLike],\n List[Union[str, bytes, os.PathLike]]\n ]\n ]\n ] = None,\n recursive: bool = False,\n unit='B'\n) -> Union[int, float]:\n \"\"\"Measure data usage of files and directories\n\n Arguments\n globs\n Globs matching paths to measure\n roots\n Directories to apply glob searches; current directory by default\n recursive\n Apply globs to all subdirectories of root directories\n unit\n Unit of memory utilization (bytes to terabytes); default bytes\n\n Returns\n Data usage in the specified unit\n \"\"\"\n```\n\nThis function also has a command-line interface.\n\n```\npython -m torchutil.paths.measure \\\n [-h] \\\n --globs GLOBS \\\n [--roots ROOTS] \\\n [--recursive] \\\n [--unit]\n\nMeasure data usage of files and directories\n\narguments:\n --globs GLOBS\n Globs matching paths to measure\n\noptional arguments:\n -h, --help\n show this help message and exit\n --roots ROOTS\n Directories to apply glob searches; current directory by default\n --recursive\n Apply globs to all subdirectories of root directories\n --unit\n Unit of memory utilization (bytes to terabytes); default bytes\n```\n\n\n### `torchutil.paths.purge`\n\n```python\ndef purge(\n globs: Optional[List[Union[str, List[str]]]] = None,\n roots: Optional[\n List[\n Union[\n Union[str, bytes, os.PathLike],\n List[Union[str, bytes, os.PathLike]]\n ]\n ]\n ] = None,\n recursive: bool = False,\n force: bool = False\n) -> None:\n \"\"\"Remove all files and directories within directory matching glob\n\n Arguments\n globs\n Globs matching paths to delete\n roots\n Directories to apply glob searches; current directory by default\n recursive\n Apply globs to all subdirectories of root directories\n force\n Skip user confirmation of deletion\n \"\"\"\n```\n\nThis function also has a command-line interface.\n\n```\npython -m torchutil.paths.purge \\\n [-h] \\\n --globs GLOBS \\\n [--roots ROOTS] \\\n [--recursive] \\\n [--force]\n\nRemove files and directories\n\narguments:\n --globs GLOBS\n Globs matching paths to delete\n\noptional arguments:\n -h, --help\n show this help message and exit\n --roots ROOTS\n Directories to apply glob searches; current directory by default\n --recursive\n Apply globs to all subdirectories of root directories\n --force\n Skip user confirmation of deletion\n```\n\n\n## Tensorboard\n\n```python\nimport matplotlib.pyplot as plt\nimport torch\nimport torchutil\n\n# Directory to write Tensorboard files\ndirectory = 'tensorboard'\n\n# Training step\nstep = 0\n\n# Example audio\naudio = torch.zeros(1, 16000)\nsample_rate = 16000\n\n# Example figure\nfigure = plt.figure()\nplt.plot([0, 1, 2, 3])\n\n# Example image\nimage = torch.zeros(256, 256, 3)\n\n# Example scalar\nloss = 0\n\n# Update Tensorboard\ntorchutil.tensorboard.update(\n directory,\n step,\n audio={'audio': audio},\n sample_rate=sample_rate,\n figures={'figure': figure},\n images={'image': image},\n scalars={'loss': loss})\n```\n\n\n### `torchutil.tensorboard.update`\n\n```python\ndef update(\n directory: Union[str, bytes, os.PathLike],\n step: int,\n audio: Optional[Dict[str, torch.Tensor]] = None,\n sample_rate: Optional[int] = None,\n figures: Optional[Dict[str, matplotlib.figure.Figure]] = None,\n images: Optional[Dict[str, torch.Tensor]] = None,\n scalars: Optional[Dict[str, Union[float, int, torch.Tensor]]] = None):\n \"\"\"Update Tensorboard\n\n Arguments\n directory - Directory to write Tensorboard files\n step - Training step\n audio - Optional dictionary of 2D audio tensors to monitor\n sample_rate - Audio sample rate; required if audio is not None\n figures - Optional dictionary of Matplotlib figures to monitor\n images - Optional dictionary of 3D image tensors to monitor\n scalars - Optional dictionary of scalars to monitor\n \"\"\"\n```\n\n\n## Time\n\n```python\nimport time\nimport torchutil\n\n# Perform timing\nwith torchutil.time.context('outer'):\n time.sleep(1)\n for i in range(2):\n time.sleep(1)\n with torchutil.time.context('inner'):\n time.sleep(1)\n\n# Prints {'inner': 2.0020763874053955, 'outer': 5.005248308181763, 'total': 5.005248308181763}\nprint(torchutil.time.results())\n```\n\n\n### `torchutil.time.context`\n\n```python\n@contextlib.contextmanager\ndef context(name: str):\n \"\"\"Wrapper to handle context changes of global timer\n\n Arguments\n name - Name of the timer to add time to\n \"\"\"\n```\n\n\n### `torchutil.time.reset`\n\n```python\ndef reset():\n \"\"\"Clear timer state\"\"\"\n```\n\n\n### `torchutil.time.results`\n\n```python\ndef results() -> dict:\n \"\"\"Get timing results\n\n Returns\n Timing results: {name: elapsed_time} for all names\n \"\"\"\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "PyTorch utilities for developing deep learning frameworks",
"version": "0.0.14",
"project_urls": {
"Homepage": "https://github.com/maxrmorrison/torchutil"
},
"split_keywords": [
"pytorch",
" utility",
" training"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d931f9e4fffcbb710270483123c8561cb7a8863c9ae51579c78a37b9414a2816",
"md5": "fca9703841c8ea2f06cd83cff58e01f2",
"sha256": "06959a4098b49dbdea2155d6475cad23ac7a1765ca7a4a487e9dad13002f0ace"
},
"downloads": -1,
"filename": "torchutil-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fca9703841c8ea2f06cd83cff58e01f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 22213,
"upload_time": "2024-11-07T20:17:04",
"upload_time_iso_8601": "2024-11-07T20:17:04.169396Z",
"url": "https://files.pythonhosted.org/packages/d9/31/f9e4fffcbb710270483123c8561cb7a8863c9ae51579c78a37b9414a2816/torchutil-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60a5f73abbc76d85d2295ab4057dd12eb4cea434d77a0c9e1e5f499be45e6b80",
"md5": "af2d14e484a5f047e33da6b2123e36d3",
"sha256": "d74f72d42456bffc8dda7078e71b2f3282eb5ca00d4d4417a235bb2077b3c9b5"
},
"downloads": -1,
"filename": "torchutil-0.0.14.tar.gz",
"has_sig": false,
"md5_digest": "af2d14e484a5f047e33da6b2123e36d3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 21675,
"upload_time": "2024-11-07T20:17:05",
"upload_time_iso_8601": "2024-11-07T20:17:05.779985Z",
"url": "https://files.pythonhosted.org/packages/60/a5/f73abbc76d85d2295ab4057dd12eb4cea434d77a0c9e1e5f499be45e6b80/torchutil-0.0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-07 20:17:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "maxrmorrison",
"github_project": "torchutil",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "torchutil"
}
JSON
