<p align="center" width="100%">
<img src="images/IMDL_BenCo.png" alt="OSQ" style="width: 28%; min-width: 150px; display: block; margin: auto;">
</p>
# IMDL-BenCo: Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization
<div align="center">
Xiaochen Ma†, Xuekang Zhu†, Lei Su†, Bo Du†, Zhuohang Jiang†, Bingkui Tong†,
Zeyu Lei†, Xinyu Yang†, Chi-Man Pun, Jiancheng Lv, Jizhe Zhou*
</div>
<div align="center"><span style="font-size: smaller;">
<br>†: joint first author & equal contribution
*: corresponding author</br>
🏎️Special thanks to Dr. <a href="https://cs.scu.edu.cn/info/1359/17839.htm">Wentao Feng</a> for the workplace, computation power, and physical infrastructure support.</span>
</div>
******
[](https://pytorch.org/)
[](https://arxiv.org/abs/2406.10580)
[](https://scu-zjz.github.io/IMDLBenCo-doc/)
<!----
[](https://GitHub.com/scu-zjz/)
---->
## Overview
☑️**Welcome to IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase.**
- This codebase is under long-term maintenance and updating. New features, extra baseline/sota models, and bug fixes will be continuously involved. You can find the corresponding plan here shortly.
- This repo decomposes the IMDL framework into **standardized, reusable components and revises the model construction pipeline**, improving coding efficiency and customization flexibility.
- This repo **fully implements or incorporates training code for state-of-the-art models** to establish a comprehensive IMDL benchmark.
- Cite and star if you feel helpful. This will encourage us a lot 🥰.
☑️**About the Developers:**
- IMDL-BenCo's project leader/supervisor is Associate Professor 🏀[_Jizhe Zhou_ (周吉喆)](https://knightzjz.github.io/), Sichuan University🇨🇳.
- IMDL-BenCo's codebase designer and coding leader is Research Assitant [_Xiaochen Ma_ (马晓晨)](https://me.xiaochen.world/), Sichuan University🇨🇳.
- IMDL-BenCo is jointly sponsored and advised by Prof. _Jiancheng LV_ (吕建成), Sichuan University 🐼, and Prof. _Chi-Man PUN_ (潘治文), University of Macau 🇲🇴, through the [Research Center of Machine Learning and Industry Intelligence, China MOE](https://center.dicalab.cn/) platform.
**Important! The current documentation and tutorials are not complete. This is a project that requires a lot of manpower, and we will do our best to complete it as quickly as possible.
Currently, you can use the demo following the brief tutorial below.**
## Features under developing
This repository has completed training, testing, robustness testing, Grad-CAM, and other functionalities for mainstream models.
However, more features are currently in testing for improved user experience. Updates will be rolled out frequently. Stay tuned!
- [ ] Install and download via PyPI
- [ ] Based on command line invocation, similar to `conda` in Anaconda.
- [ ] Dynamically create all training scripts to support personalized modifications.
- [ ] Information library, downloading, and re-management of IMDL datasets.
- [ ] Support for Weight & Bias visualization.
## Quick start
### Prepare environments
Currently, you can create a PyTorch environment and run the following command to try our repo.
```shell
git clone https://github.com/scu-zjz/IMDLBenCo.git
cd IMDLBenCo
pip install -r requirements.txt
```
### Prepare IML Datasets
- We defined three types of Dataset class
- `JsonDataset`, which gets input image and corresponding ground truth from a JSON file with a protocol like this:
```
[
[
"/Dataset/CASIAv2/Tp/Tp_D_NRN_S_N_arc00013_sec00045_11700.jpg",
"/Dataset/CASIAv2/Gt/Tp_D_NRN_S_N_arc00013_sec00045_11700_gt.png"
],
......
[
"/Dataset/CASIAv2/Au/Au_nat_30198.jpg",
"Negative"
],
......
]
```
where "Negative" represents a totally black ground truth that doesn't need a path (all authentic)
- `ManiDataset` which loads images and ground truth pairs automatically from a directory having sub-directories named `Tp` (for input images) and `Gt` (for ground truths). This class will generate the pairs using the sorted `os.listdir()` function. You can take [this folder](https://github.com/SunnyHaze/IML-ViT/tree/main/images/sample_iml_dataset) as an example.
- `BalancedDataset` is a class used to manage large datasets according to the training method of [CAT-Net](https://github.com/mjkwon2021/CAT-Net). It reads an input file as [`./runs/balanced_dataset.json`](./runs/balanced_dataset.json), which contains types of datasets and corresponding paths. Then, for each epoch, it randomly samples over 1800 images from each dataset, achieving uniform sampling among datasets with various sizes.
### Training
#### Prepare pre-trained weights (if needed)
Some models like TruFor may need pre-trained weights. Thus you need to download them in advance. You can check the guidance to download the weights in each folder under the `./IMDLBenCo/model_zoo` for the model. For example, the guidance for TruFor is under [`IMDLBenCo\model_zoo\trufor\README.md`](IMDLBenCo\model_zoo\trufor\README.md)
#### Run shell script
You can achieve customized training by modifying the dataset path and various parameters. For specific meanings of these parameters, please use python ./IMDLBenco/training_scripts/train.py -h to check.
By default, all provided scrips are called as follows:
```
sh ./runs/demo_train_iml_vit.sh
```
#### Visualize the loss & metrics & figures
Now, you can call a Tensorboard to visualize the training results by a browser.
```
tensorboard --logdir ./
```
### Customize your own model
Our design paradigm aims for the majority of customization for new models (including specific models and their respective losses) to occur within the model_zoo. Therefore, we have adopted a special design paradigm to interface with other modules. It includes the following features:
- Loss functions are defined in `__init__` and computed within `forward()`.
- The parameter list of `forward()` must consist of fixed keys to correspond to the input of required information such as `image`, `mask`, and so forth. Additional types of information can be generated via post_func and their respective fields, accepted through corresponding parameters with the same names in `forward().`
- The return value of the `forward()` function is a well-organized dictionary containing the following information as an example:
```python
# -----------------------------------------
output_dict = {
# loss for backward
"backward_loss": combined_loss,
# predicted mask, will calculate for metrics automatically
"pred_mask": mask_pred,
# predicted binaray label, will calculate for metrics automatically
"pred_label": None,
# ----values below is for visualization----
# automatically visualize with the key-value pairs
"visual_loss": {
# customized float for visualize, the key will shown as the figure name. Any number of keys and any str can be added as key.
"predict_loss": predict_loss,
"edge_loss": edge_loss,
"combined_loss": combined_loss
},
"visual_image": {
# customized tensor for visualize, the key will shown as the figure name. Any number of keys and any str can be added as key.
"pred_mask": mask_pred,
"edge_mask": edge_mask
}
# -----------------------------------------
```
Following this format, it is convenient for the framework to backpropagate the corresponding loss, compute final metrics using masks, and visualize any other scalars and tensors to observe the training process.
## Citation
If you find our work valuable and it has contributed to your research or projects, we kindly request that you cite our paper. Your recognition is a driving force for our continuous improvement and innovation🤗.
```
@misc{ma2024imdlbenco,
title={IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization},
author={Xiaochen Ma and Xuekang Zhu and Lei Su and Bo Du and Zhuohang Jiang and Bingkui Tong and Zeyu Lei and Xinyu Yang and Chi-Man Pun and Jiancheng Lv and Jizhe Zhou},
year={2024},
eprint={2406.10580},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
**************
<div align="center">
<a href="https://info.flagcounter.com/H5vw"><img src="https://s11.flagcounter.com/count2/H5vw/bg_FFFFFF/txt_000000/border_CCCCCC/columns_3/maxflags_12/viewers_0/labels_0/pageviews_1/flags_0/percent_0/" alt="Flag Counter" border="0"></a></div>
Raw data
{
"_id": null,
"home_page": "https://github.com/scu-zjz/IMDLBenCo",
"name": "IMDLBenCo",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.7",
"maintainer_email": null,
"keywords": "AI, image manipulation localization",
"author": "Xiaochen Ma",
"author_email": "xiaochen.ma.cs@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a0/b9/8472b7fe6af60a58e9ce17c0daf979f303e3d40f85c7100419b9e4d9798b/imdlbenco-0.1.10.tar.gz",
"platform": null,
"description": "<p align=\"center\" width=\"100%\">\n<img src=\"images/IMDL_BenCo.png\" alt=\"OSQ\" style=\"width: 28%; min-width: 150px; display: block; margin: auto;\">\n</p>\n\n# IMDL-BenCo: Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization\n<div align=\"center\">\nXiaochen Ma\u2020, Xuekang Zhu\u2020, Lei Su\u2020, Bo Du\u2020, Zhuohang Jiang\u2020, Bingkui Tong\u2020,\nZeyu Lei\u2020, Xinyu Yang\u2020, Chi-Man Pun, Jiancheng Lv, Jizhe Zhou*\n</div> \n<div align=\"center\"><span style=\"font-size: smaller;\">\n<br>\u2020: joint first author & equal contribution\n*: corresponding author</br>\n\ud83c\udfce\ufe0fSpecial thanks to Dr. <a href=\"https://cs.scu.edu.cn/info/1359/17839.htm\">Wentao Feng</a> for the workplace, computation power, and physical infrastructure support.</span> \n</div> \n\n****** \n\n[](https://pytorch.org/) \n[](https://arxiv.org/abs/2406.10580)\n[](https://scu-zjz.github.io/IMDLBenCo-doc/)\n\n\n\n\n<!----\n[](https://GitHub.com/scu-zjz/) \n---->\n\n## Overview\n\u2611\ufe0f**Welcome to IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase.** \n- This codebase is under long-term maintenance and updating. New features, extra baseline/sota models, and bug fixes will be continuously involved. You can find the corresponding plan here shortly.\n- This repo decomposes the IMDL framework into **standardized, reusable components and revises the model construction pipeline**, improving coding efficiency and customization flexibility.\n- This repo **fully implements or incorporates training code for state-of-the-art models** to establish a comprehensive IMDL benchmark.\n- Cite and star if you feel helpful. This will encourage us a lot \ud83e\udd70. \n\n\u2611\ufe0f**About the Developers:** \n- IMDL-BenCo's project leader/supervisor is Associate Professor \ud83c\udfc0[_Jizhe Zhou_ (\u5468\u5409\u5586)](https://knightzjz.github.io/), Sichuan University\ud83c\udde8\ud83c\uddf3. \n- IMDL-BenCo's codebase designer and coding leader is Research Assitant [_Xiaochen Ma_ (\u9a6c\u6653\u6668)](https://me.xiaochen.world/), Sichuan University\ud83c\udde8\ud83c\uddf3. \n- IMDL-BenCo is jointly sponsored and advised by Prof. _Jiancheng LV_ (\u5415\u5efa\u6210), Sichuan University \ud83d\udc3c, and Prof. _Chi-Man PUN_ (\u6f58\u6cbb\u6587), University of Macau \ud83c\uddf2\ud83c\uddf4, through the [Research Center of Machine Learning and Industry Intelligence, China MOE](https://center.dicalab.cn/) platform. \n\n**Important! The current documentation and tutorials are not complete. This is a project that requires a lot of manpower, and we will do our best to complete it as quickly as possible. \nCurrently, you can use the demo following the brief tutorial below.**\n\n\n## Features under developing\nThis repository has completed training, testing, robustness testing, Grad-CAM, and other functionalities for mainstream models.\n\nHowever, more features are currently in testing for improved user experience. Updates will be rolled out frequently. Stay tuned!\n\n- [ ] Install and download via PyPI\n- [ ] Based on command line invocation, similar to `conda` in Anaconda.\n - [ ] Dynamically create all training scripts to support personalized modifications.\n\n- [ ] Information library, downloading, and re-management of IMDL datasets.\n- [ ] Support for Weight & Bias visualization.\n\n\n## Quick start\n### Prepare environments\nCurrently, you can create a PyTorch environment and run the following command to try our repo.\n```shell\ngit clone https://github.com/scu-zjz/IMDLBenCo.git\ncd IMDLBenCo\npip install -r requirements.txt\n```\n\n### Prepare IML Datasets\n- We defined three types of Dataset class\n - `JsonDataset`, which gets input image and corresponding ground truth from a JSON file with a protocol like this:\n ```\n [\n [\n \"/Dataset/CASIAv2/Tp/Tp_D_NRN_S_N_arc00013_sec00045_11700.jpg\",\n \"/Dataset/CASIAv2/Gt/Tp_D_NRN_S_N_arc00013_sec00045_11700_gt.png\"\n ],\n ......\n [\n \"/Dataset/CASIAv2/Au/Au_nat_30198.jpg\",\n \"Negative\"\n ],\n ......\n ]\n ```\n where \"Negative\" represents a totally black ground truth that doesn't need a path (all authentic)\n - `ManiDataset` which loads images and ground truth pairs automatically from a directory having sub-directories named `Tp` (for input images) and `Gt` (for ground truths). This class will generate the pairs using the sorted `os.listdir()` function. You can take [this folder](https://github.com/SunnyHaze/IML-ViT/tree/main/images/sample_iml_dataset) as an example.\n - `BalancedDataset` is a class used to manage large datasets according to the training method of [CAT-Net](https://github.com/mjkwon2021/CAT-Net). It reads an input file as [`./runs/balanced_dataset.json`](./runs/balanced_dataset.json), which contains types of datasets and corresponding paths. Then, for each epoch, it randomly samples over 1800 images from each dataset, achieving uniform sampling among datasets with various sizes.\n\n### Training\n#### Prepare pre-trained weights (if needed)\nSome models like TruFor may need pre-trained weights. Thus you need to download them in advance. You can check the guidance to download the weights in each folder under the `./IMDLBenCo/model_zoo` for the model. For example, the guidance for TruFor is under [`IMDLBenCo\\model_zoo\\trufor\\README.md`](IMDLBenCo\\model_zoo\\trufor\\README.md)\n\n#### Run shell script\nYou can achieve customized training by modifying the dataset path and various parameters. For specific meanings of these parameters, please use python ./IMDLBenco/training_scripts/train.py -h to check.\n\nBy default, all provided scrips are called as follows:\n```\nsh ./runs/demo_train_iml_vit.sh\n```\n\n#### Visualize the loss & metrics & figures\nNow, you can call a Tensorboard to visualize the training results by a browser.\n```\ntensorboard --logdir ./\n```\n\n### Customize your own model\nOur design paradigm aims for the majority of customization for new models (including specific models and their respective losses) to occur within the model_zoo. Therefore, we have adopted a special design paradigm to interface with other modules. It includes the following features:\n\n- Loss functions are defined in `__init__` and computed within `forward()`.\n- The parameter list of `forward()` must consist of fixed keys to correspond to the input of required information such as `image`, `mask`, and so forth. Additional types of information can be generated via post_func and their respective fields, accepted through corresponding parameters with the same names in `forward().`\n- The return value of the `forward()` function is a well-organized dictionary containing the following information as an example:\n```python\n # -----------------------------------------\n output_dict = {\n # loss for backward\n \"backward_loss\": combined_loss,\n # predicted mask, will calculate for metrics automatically\n \"pred_mask\": mask_pred,\n # predicted binaray label, will calculate for metrics automatically\n \"pred_label\": None,\n\n # ----values below is for visualization----\n # automatically visualize with the key-value pairs\n \"visual_loss\": {\n # customized float for visualize, the key will shown as the figure name. Any number of keys and any str can be added as key.\n \"predict_loss\": predict_loss,\n \"edge_loss\": edge_loss,\n \"combined_loss\": combined_loss\n },\n\n \"visual_image\": {\n # customized tensor for visualize, the key will shown as the figure name. Any number of keys and any str can be added as key.\n \"pred_mask\": mask_pred,\n \"edge_mask\": edge_mask\n }\n # -----------------------------------------\n```\n\nFollowing this format, it is convenient for the framework to backpropagate the corresponding loss, compute final metrics using masks, and visualize any other scalars and tensors to observe the training process.\n\n\n\n## Citation\nIf you find our work valuable and it has contributed to your research or projects, we kindly request that you cite our paper. Your recognition is a driving force for our continuous improvement and innovation\ud83e\udd17.\n```\n@misc{ma2024imdlbenco,\n title={IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization},\n author={Xiaochen Ma and Xuekang Zhu and Lei Su and Bo Du and Zhuohang Jiang and Bingkui Tong and Zeyu Lei and Xinyu Yang and Chi-Man Pun and Jiancheng Lv and Jizhe Zhou},\n year={2024},\n eprint={2406.10580},\n archivePrefix={arXiv},\n primaryClass={cs.CV}\n}\n```\n\n**************\n<div align=\"center\">\n<a href=\"https://info.flagcounter.com/H5vw\"><img src=\"https://s11.flagcounter.com/count2/H5vw/bg_FFFFFF/txt_000000/border_CCCCCC/columns_3/maxflags_12/viewers_0/labels_0/pageviews_1/flags_0/percent_0/\" alt=\"Flag Counter\" border=\"0\"></a></div>\n",
"bugtrack_url": null,
"license": "CC-BY-4.0",
"summary": "A comprehensive benchmark and code base for Image manipulation and localization.",
"version": "0.1.10",
"project_urls": {
"Bug Reports": "https://github.com/scu-zjz/IMDLBenCo/issues",
"Documentation": "https://github.com/scu-zjz/IMDLBenCo-doc",
"Github": "https://github.com/scu-zjz/IMDLBenCo/",
"Homepage": "https://github.com/scu-zjz/IMDLBenCo"
},
"split_keywords": [
"ai",
" image manipulation localization"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7739d46178bb0bbceb094502345f4d31326aef5f76dbb446939a7a89f44e167c",
"md5": "d52c33e612f8da7590eea50c5764e4f1",
"sha256": "fb6ed24615ed7ea8e5d7fa6bb358b8ffe0ed8ab4b09626d509df5d835fa71eae"
},
"downloads": -1,
"filename": "IMDLBenCo-0.1.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d52c33e612f8da7590eea50c5764e4f1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.7",
"size": 460076,
"upload_time": "2024-06-27T15:17:34",
"upload_time_iso_8601": "2024-06-27T15:17:34.940785Z",
"url": "https://files.pythonhosted.org/packages/77/39/d46178bb0bbceb094502345f4d31326aef5f76dbb446939a7a89f44e167c/IMDLBenCo-0.1.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a0b98472b7fe6af60a58e9ce17c0daf979f303e3d40f85c7100419b9e4d9798b",
"md5": "03ee9f1adf757f0bb0dda332b1d7d461",
"sha256": "4164edfa53a1268ab3493d2bd420e2a420e54ef4d1f6f14fc9bf6cea701fe7a8"
},
"downloads": -1,
"filename": "imdlbenco-0.1.10.tar.gz",
"has_sig": false,
"md5_digest": "03ee9f1adf757f0bb0dda332b1d7d461",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.7",
"size": 363028,
"upload_time": "2024-06-27T15:17:37",
"upload_time_iso_8601": "2024-06-27T15:17:37.812188Z",
"url": "https://files.pythonhosted.org/packages/a0/b9/8472b7fe6af60a58e9ce17c0daf979f303e3d40f85c7100419b9e4d9798b/imdlbenco-0.1.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-27 15:17:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "scu-zjz",
"github_project": "IMDLBenCo",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "rich",
"specs": []
},
{
"name": "fvcore",
"specs": []
},
{
"name": "jpegio",
"specs": []
},
{
"name": "albumentations",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "scikit_learn",
"specs": []
},
{
"name": "timm",
"specs": []
},
{
"name": "fvcore",
"specs": []
},
{
"name": "tensorboard",
"specs": []
},
{
"name": "grad_cam",
"specs": []
},
{
"name": "ttach",
"specs": []
}
],
"lcname": "imdlbenco"
}
JSON
