# Pytorch Implementation of Common GAN metrics
## Install
```
pip install pytorch-gan-metrics
```
- `torch>=1.8.2`
- `torchvision>=0.9.2`
## Quick Start
```python
from pytorch_gan_metrics import get_inception_score, get_fid
images = ... # [N, 3, H, W] normalized to [0, 1]
IS, IS_std = get_inception_score(images) # Inception Score
FID = get_fid(images, 'path/to/statistics.npz') # Frechet Inception Distance
```
`path/to/statistics.npz` is compatiable with [official](https://github.com/bioinf-jku/TTUR) FID implementation.
## Notes
The FID implementation is inspired from [pytorch-fid](https://github.com/mseitzer/pytorch-fid).
This repository is developed for personal research. If you think this package can also benefit your life, please feel free to open issues.
## Features
- Currently, this package supports following metrics:
- [Inception Score](https://github.com/openai/improved-gan) (IS)
- [Fréchet Inception Distance](https://github.com/bioinf-jku/TTUR) (FID)
- The computation procedure of IS and FID are integrated to avoid multiple forward propagations.
- Support reading images on the fly to avoid out of memory especially for large scale images.
- Support computation on GPU to speed up some cpu operations such as `np.cov` and `scipy.linalg.sqrtm`.
## Reproducing Results of Official Implementations on CIFAR-10
| |Train IS |Test IS |Train(50k) vs Test(10k)<br>FID|
|-------------------|:--------:|:--------:|:----------------------------:|
|Official |11.24±0.20|10.98±0.22|3.1508 |
|pytorch-gan-metrics|11.26±0.14|10.96±0.35|3.1518 |
|pytorch-gan-metrics<br>`use_torch=True`|11.26±0.15|10.96±0.19|3.1509 |
The results are slightly different from official implementations due to the framework difference between PyTorch and TensorFlow.
## Documentation
### Prepare Statistics (for FID)
- [Download](https://drive.google.com/drive/folders/1UBdzl6GtNMwNQ5U-4ESlIer43tNjiGJC?usp=sharing) precalculated statistics or
- Calculate statistics for your custom dataset using command line tool
```bash
python -m pytorch_gan_metrics.calc_fid_stats \
--path path/to/images \
--stats path/to/statistics.npz
```
See [calc_fid_stats.py](./pytorch_gan_metrics/calc_fid_stats.py) for details.
### Inception Features
- When getting IS or FID, the `InceptionV3` will be loaded into `torch.device('cuda:0')` if GPU is availabel; Otherwise, `torch.device('cpu')` will be used.
- Change `device` argument in `get_*` functions to set torch device.
### Using `torch.Tensor` as images
- Prepare images in type `torch.float32` with shape `[N, 3, H, W]` and normalized to `[0,1]`.
```python
from pytorch_gan_metrics import (get_inception_score,
get_fid,
get_inception_score_and_fid)
images = ... # [N, 3, H, W]
assert 0 <= images.min() and images.max() <= 1
# Inception Score
IS, IS_std = get_inception_score(
images)
# Frechet Inception Distance
FID = get_fid(
images, 'path/to/statistics.npz')
# Inception Score & Frechet Inception Distance
(IS, IS_std), FID = get_inception_score_and_fid(
images, 'path/to/statistics.npz')
```
### Using PyTorch DataLoader to Provide Images
- Use `pytorch_gan_metrics.ImageDataset` to collect images on your storage or use your custom `torch.utils.data.Dataset`.
```python
from pytorch_gan_metrics import ImageDataset
dataset = ImageDataset(path_to_dir, exts=['png', 'jpg'])
loader = DataLoader(dataset, batch_size=50, num_workers=4)
```
- It is possible to wrap a generative model in a dataset to support generating images on the fly. Remember to set `num_workers=0` to avoid copying models across multiprocess.
```python
class GeneratorDataset(Dataset):
def __init__(self, G, z_dim):
self.G = G
self.z_dim = z_dim
def __len__(self):
return 50000
def __getitem__(self, index):
return self.G(torch.randn(1, self.z_dim).cuda())[0]
dataset = GeneratorDataset(G, z=128)
loader = DataLoader(dataset, batch_size=50, num_workers=0)
```
- Calculate metrics
```python
from pytorch_gan_metrics import (get_inception_score,
get_fid,
get_inception_score_and_fid)
# Inception Score
IS, IS_std = get_inception_score(
loader)
# Frechet Inception Distance
FID = get_fid(
loader, 'path/to/statistics.npz')
# Inception Score & Frechet Inception Distance
(IS, IS_std), FID = get_inception_score_and_fid(
loader, 'path/to/statistics.npz')
```
### Load Images from a Directory
- Calculate metrics for images in a directory and its subfolders.
```python
from pytorch_gan_metrics import (
get_inception_score_from_directory,
get_fid_from_directory,
get_inception_score_and_fid_from_directory)
IS, IS_std = get_inception_score_from_directory(
'path/to/images')
FID = get_fid_from_directory(
'path/to/images', 'path/to/statistics.npz')
(IS, IS_std), FID = get_inception_score_and_fid_from_directory(
'path/to/images', 'path/to/statistics.npz')
```
### Accelerating Matrix Computation by PyTorch
- Set `use_torch=True` when calling functions `get_*` such as `get_inception_score`, `get_fid`, etc.
- **WARNING** when `use_torch=True` is used, the FID might be `nan` due to the unstable implementation of matrix sqrt.
- This option is recommended to be used when evaluating generative models on a server which is equipped with high efficiency GPUs while the cpu frequency is low.
## Tested Versions
- `python 3.9 + torch 1.8.2 + CUDA 10.2`
- `python 3.9 + torch 1.11.0 + CUDA 10.2`
- `python 3.9 + torch 1.12.1 + CUDA 10.2`
## License
This implementation is licensed under the Apache License 2.0.
This implementation is derived from [pytorch-fid](https://github.com/mseitzer/pytorch-fid), licensed under the Apache License 2.0.
FID was introduced by Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter in "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", see [https://arxiv.org/abs/1706.08500](https://arxiv.org/abs/1706.08500)
The original implementation of FID is by the Institute of Bioinformatics, JKU Linz, licensed under the Apache License 2.0.
See [https://github.com/bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR).
Raw data
{
"_id": null,
"home_page": "https://github.com/w86763777/pytorch-gan-metrics",
"name": "pytorch-gan-metrics",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "PyTorch,GAN,Inception Score,IS,Frechet Inception Distance,FID",
"author": "Yi-Lun Wu",
"author_email": "w86763777@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/56/1f/92ab220f701a35d7c22c595ad2f280913c10d6316249eed4d8cd93fa6c46/pytorch_gan_metrics-0.5.3.tar.gz",
"platform": null,
"description": "# Pytorch Implementation of Common GAN metrics\n\n\n\n## Install\n```\npip install pytorch-gan-metrics\n```\n- `torch>=1.8.2`\n- `torchvision>=0.9.2`\n\n## Quick Start\n```python\nfrom pytorch_gan_metrics import get_inception_score, get_fid\n\nimages = ... # [N, 3, H, W] normalized to [0, 1]\nIS, IS_std = get_inception_score(images) # Inception Score\nFID = get_fid(images, 'path/to/statistics.npz') # Frechet Inception Distance\n```\n`path/to/statistics.npz` is compatiable with [official](https://github.com/bioinf-jku/TTUR) FID implementation.\n\n## Notes\nThe FID implementation is inspired from [pytorch-fid](https://github.com/mseitzer/pytorch-fid).\n\nThis repository is developed for personal research. If you think this package can also benefit your life, please feel free to open issues.\n\n## Features\n- Currently, this package supports following metrics:\n - [Inception Score](https://github.com/openai/improved-gan) (IS)\n - [Fr\u00e9chet Inception Distance](https://github.com/bioinf-jku/TTUR) (FID)\n- The computation procedure of IS and FID are integrated to avoid multiple forward propagations.\n- Support reading images on the fly to avoid out of memory especially for large scale images.\n- Support computation on GPU to speed up some cpu operations such as `np.cov` and `scipy.linalg.sqrtm`.\n\n## Reproducing Results of Official Implementations on CIFAR-10\n\n| |Train IS |Test IS |Train(50k) vs Test(10k)<br>FID|\n|-------------------|:--------:|:--------:|:----------------------------:|\n|Official |11.24\u00b10.20|10.98\u00b10.22|3.1508 |\n|pytorch-gan-metrics|11.26\u00b10.14|10.96\u00b10.35|3.1518 |\n|pytorch-gan-metrics<br>`use_torch=True`|11.26\u00b10.15|10.96\u00b10.19|3.1509 |\n\nThe results are slightly different from official implementations due to the framework difference between PyTorch and TensorFlow.\n\n## Documentation\n\n### Prepare Statistics (for FID)\n- [Download](https://drive.google.com/drive/folders/1UBdzl6GtNMwNQ5U-4ESlIer43tNjiGJC?usp=sharing) precalculated statistics or\n- Calculate statistics for your custom dataset using command line tool\n ```bash\n python -m pytorch_gan_metrics.calc_fid_stats \\\n --path path/to/images \\\n --stats path/to/statistics.npz\n ```\n See [calc_fid_stats.py](./pytorch_gan_metrics/calc_fid_stats.py) for details.\n\n### Inception Features\n- When getting IS or FID, the `InceptionV3` will be loaded into `torch.device('cuda:0')` if GPU is availabel; Otherwise, `torch.device('cpu')` will be used.\n- Change `device` argument in `get_*` functions to set torch device.\n\n### Using `torch.Tensor` as images\n\n- Prepare images in type `torch.float32` with shape `[N, 3, H, W]` and normalized to `[0,1]`.\n ```python\n from pytorch_gan_metrics import (get_inception_score,\n get_fid,\n get_inception_score_and_fid)\n images = ... # [N, 3, H, W]\n assert 0 <= images.min() and images.max() <= 1\n # Inception Score\n IS, IS_std = get_inception_score(\n images)\n # Frechet Inception Distance\n FID = get_fid(\n images, 'path/to/statistics.npz')\n # Inception Score & Frechet Inception Distance\n (IS, IS_std), FID = get_inception_score_and_fid(\n images, 'path/to/statistics.npz')\n\n ```\n\n### Using PyTorch DataLoader to Provide Images\n\n- Use `pytorch_gan_metrics.ImageDataset` to collect images on your storage or use your custom `torch.utils.data.Dataset`.\n ```python\n from pytorch_gan_metrics import ImageDataset\n\n dataset = ImageDataset(path_to_dir, exts=['png', 'jpg'])\n loader = DataLoader(dataset, batch_size=50, num_workers=4)\n ```\n\n- It is possible to wrap a generative model in a dataset to support generating images on the fly. Remember to set `num_workers=0` to avoid copying models across multiprocess.\n ```python\n class GeneratorDataset(Dataset):\n def __init__(self, G, z_dim):\n self.G = G\n self.z_dim = z_dim\n\n def __len__(self):\n return 50000\n\n def __getitem__(self, index):\n return self.G(torch.randn(1, self.z_dim).cuda())[0]\n\n dataset = GeneratorDataset(G, z=128)\n loader = DataLoader(dataset, batch_size=50, num_workers=0)\n ```\n\n- Calculate metrics\n ```python\n from pytorch_gan_metrics import (get_inception_score,\n get_fid,\n get_inception_score_and_fid)\n # Inception Score\n IS, IS_std = get_inception_score(\n loader)\n # Frechet Inception Distance\n FID = get_fid(\n loader, 'path/to/statistics.npz')\n # Inception Score & Frechet Inception Distance\n (IS, IS_std), FID = get_inception_score_and_fid(\n loader, 'path/to/statistics.npz')\n ```\n\n### Load Images from a Directory\n\n- Calculate metrics for images in a directory and its subfolders.\n ```python\n from pytorch_gan_metrics import (\n get_inception_score_from_directory,\n get_fid_from_directory,\n get_inception_score_and_fid_from_directory)\n\n IS, IS_std = get_inception_score_from_directory(\n 'path/to/images')\n FID = get_fid_from_directory(\n 'path/to/images', 'path/to/statistics.npz')\n (IS, IS_std), FID = get_inception_score_and_fid_from_directory(\n 'path/to/images', 'path/to/statistics.npz')\n ```\n\n### Accelerating Matrix Computation by PyTorch\n\n- Set `use_torch=True` when calling functions `get_*` such as `get_inception_score`, `get_fid`, etc.\n\n- **WARNING** when `use_torch=True` is used, the FID might be `nan` due to the unstable implementation of matrix sqrt.\n\n- This option is recommended to be used when evaluating generative models on a server which is equipped with high efficiency GPUs while the cpu frequency is low.\n\n## Tested Versions\n- `python 3.9 + torch 1.8.2 + CUDA 10.2`\n- `python 3.9 + torch 1.11.0 + CUDA 10.2`\n- `python 3.9 + torch 1.12.1 + CUDA 10.2`\n\n## License\n\nThis implementation is licensed under the Apache License 2.0.\n\nThis implementation is derived from [pytorch-fid](https://github.com/mseitzer/pytorch-fid), licensed under the Apache License 2.0.\n\nFID was introduced by Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter in \"GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium\", see [https://arxiv.org/abs/1706.08500](https://arxiv.org/abs/1706.08500)\n\nThe original implementation of FID is by the Institute of Bioinformatics, JKU Linz, licensed under the Apache License 2.0.\nSee [https://github.com/bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR).\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Package for calculating GAN metrics using Pytorch",
"version": "0.5.3",
"project_urls": {
"Homepage": "https://github.com/w86763777/pytorch-gan-metrics"
},
"split_keywords": [
"pytorch",
"gan",
"inception score",
"is",
"frechet inception distance",
"fid"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "02832c65b41b01ec26fcd2aea0943f4e2373759314776b9f0a2db9ee59e459cc",
"md5": "d7d51087239b0fd1c087718ca9e782d0",
"sha256": "e70443cf6ec8b6f74acb1a9643c406d2134de771c728ab2b08dc7a137e617b92"
},
"downloads": -1,
"filename": "pytorch_gan_metrics-0.5.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d7d51087239b0fd1c087718ca9e782d0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 18703,
"upload_time": "2023-09-08T04:45:55",
"upload_time_iso_8601": "2023-09-08T04:45:55.492440Z",
"url": "https://files.pythonhosted.org/packages/02/83/2c65b41b01ec26fcd2aea0943f4e2373759314776b9f0a2db9ee59e459cc/pytorch_gan_metrics-0.5.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "561f92ab220f701a35d7c22c595ad2f280913c10d6316249eed4d8cd93fa6c46",
"md5": "5183263aec8acd80c00dc7224c4ad21e",
"sha256": "df5a32f57f5b669dfcc8cc008221ebc9022234f3e3a2dd199f66d71756408102"
},
"downloads": -1,
"filename": "pytorch_gan_metrics-0.5.3.tar.gz",
"has_sig": false,
"md5_digest": "5183263aec8acd80c00dc7224c4ad21e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 20435,
"upload_time": "2023-09-08T04:45:57",
"upload_time_iso_8601": "2023-09-08T04:45:57.476160Z",
"url": "https://files.pythonhosted.org/packages/56/1f/92ab220f701a35d7c22c595ad2f280913c10d6316249eed4d8cd93fa6c46/pytorch_gan_metrics-0.5.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-08 04:45:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "w86763777",
"github_project": "pytorch-gan-metrics",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "packaging",
"specs": []
},
{
"name": "tqdm",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "torch",
"specs": [
[
">=",
"1.8.2"
]
]
},
{
"name": "torchvision",
"specs": [
[
">=",
"0.9.2"
]
]
}
],
"tox": true,
"lcname": "pytorch-gan-metrics"
}
JSON
