# GAN Evaluator for Inception Score (IS) and Frechet Inception Distance (FID) in PyTorch
[](https://pypi.org/project/gan-evaluator/)
[](https://pypi.python.org/pypi/gan-evaluator/)
[](https://pypi.python.org/pypi/gan-evaluator/)
[](https://pypi.python.org/pypi/gan-evaluator/)
[](https://www.python.org/)
Contributors: Chen Liu (chen.liu.cl2482@yale.edu), Alex Wong (alex.wong@yale.edu)
*Please kindly **Star** [](https://github.com/ChenLiu-1996/GAN-evaluator/) this repo for better reach if you find it useful. Let's help out the community!*
## Main Contributions
1. We created a GAN evaluator for IS and FID that
- [is easy to use](#usage),
- accepts data as either dataloaders or individual batches, and
- supports on-the-fly evaluation during training.
2. We provided a simple demo script to demonstrate one common use case.
## NEWS
### [Feb 18, 2023]
**Now available on PyPI!** Now you can **pip install it** to your desired environment via:
```
pip install gan-evaluator
```
And in your Python project, wherever you need the `GAN_Evaluator`, you can import via:
```
from gan_evaluator import GAN_Evaluator
```
NOTE 1: **You no longer need to copy any code from this repo in order to use `GAN_Evalutor`**! At this point, the primary purpose of this repo is description and demonstration. With that said, you surely can clone this repo and try out the demo script. Also, you may find it easier to copy and modify the code if you want slightly different behaviors.
NOTE 2: During `pip install gan-evaluator`, the dependencies of `GAN_Evaluator` (but not of the demo script) are also installed.
## Demo Script: Use [DCGAN](https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial) to generate [SVHN](http://ufldl.stanford.edu/housenumbers/) digits
The script can be found in [`src/train_dcgan_svhn.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/train_dcgan_svhn.py)
- Usage from the demo script, to give you a taste.
**Declaration**
```
evaluator = GAN_Evaluator(device=device,
num_images_real=len(train_loader.dataset),
num_images_fake=len(train_loader.dataset))
```
**Before traing loop**
```
evaluator.load_all_real_imgs(real_loader=train_loader, idx_in_loader=0)
```
**Inside traing loop**
```
if shall_plot:
IS_mean, IS_std, FID = evaluator.fill_fake_img_batch(fake_batch=x_fake)
else:
evaluator.fill_fake_img_batch(fake_batch=x_fake, return_results=False)
```
**After each epoch of training**
```
evaluator.clear_fake_imgs()
```
- Some visualizations of the demo script:
- Real (top) and Generated (bottom) images.
<img src = "https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/debug_plot/dcgan_svhn/epoch_0046_batch_0199_generated.png" width=800>
- IS and FID curves.
<img src = "https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/debug_plot/dcgan_svhn/IS_FID_curve.png" width=800>
## Details: The Evaluator for IS and FID
<details>
<summary><b>Introduction to the Evaluator</b></summary>
<br>
More details can be found in [`src/utils/gan_evaluator.py/GAN_Evaluator`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/utils/gan_evaluator.py#L13).
```
This evaluator computes the following metrics:
- Inception Score (IS)
- Frechet Inception Distance (FID)
This evaluator will take in the real images and the fake/generated images.
Then it will compute the activations from the real and fake images as well as the
predictions from the fake images.
The (fake) predictions will be used to compute IS, while
the (real, fake) activations will be used to compute FID.
If input image resolution < 75 x 75, we will upsample the image to accommodate Inception v3.
The real and fake images can be provided to this evaluator in either of the following formats:
1. dataloader
`load_all_real_imgs`
`load_all_fake_imgs`
2. per-batch
`fill_real_img_batch`
`fill_fake_img_batch`
!!! Please note: the latest IS and FID will be returned upon completion of either of the following:
`load_all_fake_imgs`
`fill_fake_img_batch`
Return format:
(IS mean, IS std, FID)
*So please make sure you load real images before the fake images.*
Common Use Cases:
1. For the purpose of on-the-fly evaluation during GAN training:
We recommend pre-loading the real images using the dataloader format, and
populate the fake images using the per-batch format as training goes on.
- At the end of each epoch, you can clean the fake images using:
`clear_fake_imgs`
- In *unusual* cases where your real images change (such as in progressive growing GANs),
you may want to clear the real images. You can do so via:
`clear_real_imgs`
2. For the purpose of offline evaluation of a saved dataset:
We recommend pre-loading the real images and fake images.
```
</details>
## Repository Hierarchy
```
GAN-evaluator
├── config
| └── `dcgan_svhn.yaml`
├── data (*)
├── debug_plot (*)
├── logs (*)
└── src
├── utils
| ├── `gan_evaluator.py`: THIS CONTAINS OUR `GAN_Evaluator`.
| └── other utility files.
└── `train_dcgan_svhn.py`: our demo script.
```
Folders marked with (*), if not exist, will be created automatically when you run [`train_dcgan_svhn.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/train_dcgan_svhn.py).
## Usage
- To run our demo script, do the following after activating the proper environment.
```
git clone git@github.com:ChenLiu-1996/GAN-evaluator.git
cd src
python train_dcgan_svhn.py --config ../config/dcgan_svhn.yaml
```
- To integrate our evaluator into your existing project, you can simply copy [`src/utils/gan_evaluator.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/utils/gan_evaluator.py) to an appropriate folder in your project, and import `GAN_Evaluator` to wherever you find necessary. [**Update: Now you can directly install via pip!**](#feb-18-2023)
- We will add our citation bibtex, and we would appreciate if you reference our work in case this repository helps you in your research.
## Citation
To be added.
## Environement Setup
<details>
<summary><b>Packages Needed</b></summary>
<br>
The `GAN_Evaluator` module itself only uses `numpy`, `scipy`, `torch`, `torchvision`, and (for aesthetics) `tqdm`.
To run the example script, it additionally requires `matplotlib`, `argparse`, and `yaml`.
</details>
<details>
<summary><b>On our Yale Vision Lab server</b></summary>
- There is a virtualenv ready to use, located at
`/media/home/chliu/.virtualenv/mondi-image-gen/`.
- Alternatively, you can start from an existing environment "torch191-py38env",
and install the following packages:
```
python3 -m pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
python3 -m pip install wget gdown numpy matplotlib pyyaml click scipy yacs scikit-learn
```
If you see error messages such as `Failed to build CUDA kernels for bias_act.`, you can fix it with:
```
python3 -m pip install ninja
```
</details>
## Acknowledgements
1. The code for the `GAN_Evaluator` (specifically, the computation of IS and FID) is inspired by:
- https://github.com/sbarratt/inception-score-pytorch/blob/master/inception_score.py
- https://www.kaggle.com/code/ibtesama/gan-in-pytorch-with-fid
- https://github.com/mseitzer/pytorch-fid/blob/master/src/pytorch_fid/fid_score.py
- **Note: We did not validate the "mathmetical correctness" in their computations. Please use mindfully.**
2. The code for the demo script (specifically, architecture and training of DCGAN) is inspired by:
- https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial
Raw data
{
"_id": null,
"home_page": "https://github.com/ChenLiu-1996/GAN-evaluator",
"name": "gan-evaluator",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "GAN,evaluator,IS,FID,inception",
"author": "Chen Liu",
"author_email": "chen.liu.cl2482@yale.edu",
"download_url": "https://files.pythonhosted.org/packages/86/df/a7e83c42a31c243fab4ef99a66b20b0292239a2ae1799692743ab09e58cb/gan-evaluator-1.15.tar.gz",
"platform": null,
"description": "# GAN Evaluator for Inception Score (IS) and Frechet Inception Distance (FID) in PyTorch\n\n[](https://pypi.org/project/gan-evaluator/)\n[](https://pypi.python.org/pypi/gan-evaluator/)\n[](https://pypi.python.org/pypi/gan-evaluator/)\n[](https://pypi.python.org/pypi/gan-evaluator/)\n[](https://www.python.org/)\n\nContributors: Chen Liu (chen.liu.cl2482@yale.edu), Alex Wong (alex.wong@yale.edu)\n\n*Please kindly **Star** [](https://github.com/ChenLiu-1996/GAN-evaluator/) this repo for better reach if you find it useful. Let's help out the community!*\n\n## Main Contributions\n1. We created a GAN evaluator for IS and FID that\n - [is easy to use](#usage),\n - accepts data as either dataloaders or individual batches, and\n - supports on-the-fly evaluation during training.\n2. We provided a simple demo script to demonstrate one common use case.\n\n## NEWS\n### [Feb 18, 2023]\n\n**Now available on PyPI!** Now you can **pip install it** to your desired environment via:\n```\npip install gan-evaluator\n```\nAnd in your Python project, wherever you need the `GAN_Evaluator`, you can import via:\n```\nfrom gan_evaluator import GAN_Evaluator\n```\nNOTE 1: **You no longer need to copy any code from this repo in order to use `GAN_Evalutor`**! At this point, the primary purpose of this repo is description and demonstration. With that said, you surely can clone this repo and try out the demo script. Also, you may find it easier to copy and modify the code if you want slightly different behaviors.\n\nNOTE 2: During `pip install gan-evaluator`, the dependencies of `GAN_Evaluator` (but not of the demo script) are also installed.\n\n## Demo Script: Use [DCGAN](https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial) to generate [SVHN](http://ufldl.stanford.edu/housenumbers/) digits\n\nThe script can be found in [`src/train_dcgan_svhn.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/train_dcgan_svhn.py)\n\n- Usage from the demo script, to give you a taste.\n\n **Declaration**\n ```\n evaluator = GAN_Evaluator(device=device,\n num_images_real=len(train_loader.dataset),\n num_images_fake=len(train_loader.dataset))\n ```\n\n **Before traing loop**\n ```\n evaluator.load_all_real_imgs(real_loader=train_loader, idx_in_loader=0)\n ```\n\n **Inside traing loop**\n ```\n if shall_plot:\n IS_mean, IS_std, FID = evaluator.fill_fake_img_batch(fake_batch=x_fake)\n else:\n evaluator.fill_fake_img_batch(fake_batch=x_fake, return_results=False)\n ```\n\n **After each epoch of training**\n ```\n evaluator.clear_fake_imgs()\n ```\n\n- Some visualizations of the demo script:\n - Real (top) and Generated (bottom) images.\n <img src = \"https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/debug_plot/dcgan_svhn/epoch_0046_batch_0199_generated.png\" width=800>\n\n - IS and FID curves.\n <img src = \"https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/debug_plot/dcgan_svhn/IS_FID_curve.png\" width=800>\n\n## Details: The Evaluator for IS and FID\n<details>\n <summary><b>Introduction to the Evaluator</b></summary>\n<br>\n\nMore details can be found in [`src/utils/gan_evaluator.py/GAN_Evaluator`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/utils/gan_evaluator.py#L13).\n\n```\nThis evaluator computes the following metrics:\n - Inception Score (IS)\n - Frechet Inception Distance (FID)\n\nThis evaluator will take in the real images and the fake/generated images.\nThen it will compute the activations from the real and fake images as well as the\npredictions from the fake images.\nThe (fake) predictions will be used to compute IS, while\nthe (real, fake) activations will be used to compute FID.\nIf input image resolution < 75 x 75, we will upsample the image to accommodate Inception v3.\n\nThe real and fake images can be provided to this evaluator in either of the following formats:\n1. dataloader\n `load_all_real_imgs`\n `load_all_fake_imgs`\n2. per-batch\n `fill_real_img_batch`\n `fill_fake_img_batch`\n\n!!! Please note: the latest IS and FID will be returned upon completion of either of the following:\n `load_all_fake_imgs`\n `fill_fake_img_batch`\nReturn format:\n (IS mean, IS std, FID)\n*So please make sure you load real images before the fake images.*\n\nCommon Use Cases:\n1. For the purpose of on-the-fly evaluation during GAN training:\n We recommend pre-loading the real images using the dataloader format, and\n populate the fake images using the per-batch format as training goes on.\n - At the end of each epoch, you can clean the fake images using:\n `clear_fake_imgs`\n - In *unusual* cases where your real images change (such as in progressive growing GANs),\n you may want to clear the real images. You can do so via:\n `clear_real_imgs`\n2. For the purpose of offline evaluation of a saved dataset:\n We recommend pre-loading the real images and fake images.\n```\n\n</details>\n\n## Repository Hierarchy\n```\nGAN-evaluator\n \u251c\u2500\u2500 config\n | \u2514\u2500\u2500 `dcgan_svhn.yaml`\n \u251c\u2500\u2500 data (*)\n \u251c\u2500\u2500 debug_plot (*)\n \u251c\u2500\u2500 logs (*)\n \u2514\u2500\u2500 src\n \u251c\u2500\u2500 utils\n | \u251c\u2500\u2500 `gan_evaluator.py`: THIS CONTAINS OUR `GAN_Evaluator`.\n | \u2514\u2500\u2500 other utility files.\n \u2514\u2500\u2500 `train_dcgan_svhn.py`: our demo script.\n```\nFolders marked with (*), if not exist, will be created automatically when you run [`train_dcgan_svhn.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/train_dcgan_svhn.py).\n\n## Usage\n- To run our demo script, do the following after activating the proper environment.\n```\ngit clone git@github.com:ChenLiu-1996/GAN-evaluator.git\ncd src\npython train_dcgan_svhn.py --config ../config/dcgan_svhn.yaml\n```\n- To integrate our evaluator into your existing project, you can simply copy [`src/utils/gan_evaluator.py`](https://github.com/ChenLiu-1996/GAN-evaluator/blob/main/src/utils/gan_evaluator.py) to an appropriate folder in your project, and import `GAN_Evaluator` to wherever you find necessary. [**Update: Now you can directly install via pip!**](#feb-18-2023)\n\n- We will add our citation bibtex, and we would appreciate if you reference our work in case this repository helps you in your research.\n\n## Citation\nTo be added.\n\n## Environement Setup\n<details>\n <summary><b>Packages Needed</b></summary>\n<br>\n\nThe `GAN_Evaluator` module itself only uses `numpy`, `scipy`, `torch`, `torchvision`, and (for aesthetics) `tqdm`.\n\nTo run the example script, it additionally requires `matplotlib`, `argparse`, and `yaml`.\n\n</details>\n\n<details>\n <summary><b>On our Yale Vision Lab server</b></summary>\n\n- There is a virtualenv ready to use, located at\n`/media/home/chliu/.virtualenv/mondi-image-gen/`.\n\n- Alternatively, you can start from an existing environment \"torch191-py38env\",\nand install the following packages:\n```\npython3 -m pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113\npython3 -m pip install wget gdown numpy matplotlib pyyaml click scipy yacs scikit-learn\n```\n\nIf you see error messages such as `Failed to build CUDA kernels for bias_act.`, you can fix it with:\n```\npython3 -m pip install ninja\n```\n\n</details>\n\n## Acknowledgements\n1. The code for the `GAN_Evaluator` (specifically, the computation of IS and FID) is inspired by:\n - https://github.com/sbarratt/inception-score-pytorch/blob/master/inception_score.py\n - https://www.kaggle.com/code/ibtesama/gan-in-pytorch-with-fid\n - https://github.com/mseitzer/pytorch-fid/blob/master/src/pytorch_fid/fid_score.py\n - **Note: We did not validate the \"mathmetical correctness\" in their computations. Please use mindfully.**\n2. The code for the demo script (specifically, architecture and training of DCGAN) is inspired by:\n - https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "GAN Evaluator for IS and FID",
"version": "1.15",
"split_keywords": [
"gan",
"evaluator",
"is",
"fid",
"inception"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ee15b3ee8296d1d1bb9d2689f43d272b76b2e3b2c23d466810aedebda1ec74dd",
"md5": "ef8041618e5c5700a45b5255824d8f7c",
"sha256": "aca501d8378756d8ab9b2c29f9b774a727072d034757adb9708f8b13aeae49d1"
},
"downloads": -1,
"filename": "gan_evaluator-1.15-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ef8041618e5c5700a45b5255824d8f7c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 10362,
"upload_time": "2023-03-20T17:09:48",
"upload_time_iso_8601": "2023-03-20T17:09:48.707524Z",
"url": "https://files.pythonhosted.org/packages/ee/15/b3ee8296d1d1bb9d2689f43d272b76b2e3b2c23d466810aedebda1ec74dd/gan_evaluator-1.15-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "86dfa7e83c42a31c243fab4ef99a66b20b0292239a2ae1799692743ab09e58cb",
"md5": "f1dafaff2b2888e4aa4e03cab004e264",
"sha256": "07609c2965a98baa275c26056532bab6efbd1a4c69d0f31efd4ac8059403463f"
},
"downloads": -1,
"filename": "gan-evaluator-1.15.tar.gz",
"has_sig": false,
"md5_digest": "f1dafaff2b2888e4aa4e03cab004e264",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8613,
"upload_time": "2023-03-20T17:09:50",
"upload_time_iso_8601": "2023-03-20T17:09:50.447962Z",
"url": "https://files.pythonhosted.org/packages/86/df/a7e83c42a31c243fab4ef99a66b20b0292239a2ae1799692743ab09e58cb/gan-evaluator-1.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-20 17:09:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "ChenLiu-1996",
"github_project": "GAN-evaluator",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "gan-evaluator"
}
JSON
