| Name | MRFI JSON |
| Version |
2.0.0
JSON |
| download |
| home_page | |
| Summary | A multi-resolution DNN fault injection tool |
| upload_time | 2023-07-15 15:13:19 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | MIT |
| keywords |
pytorch
reliability
faultinjection
faulttolerance
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Multi-Resolution Fault Injector
<p dir="auto" align="center">
<img src="docs/assets/logo_name.png" width=350)>
</p>
## MRFI Overview
[](https://github.com/fffasttime/MRFI/blob/master/LICENSE)
[](https://github.com/fffasttime/MRFI/actions/workflows/codecov.yml)
[](https://codecov.io/gh/fffasttime/MRFI)
Multi-Resolution Fault Injector is a **powerful** neural network fault injector tool based on PyTorch.
Compared with previous network-level fault injection frameworks, the biggest feature is that MRFI can flexibly adjust different injection configurations for different experimental needs.
Injection config and observations on each layer can be set **independently** by one clear config file.
MRFI also provides a large number of commonly used fault injection methods and error models, and allows customization.
In preliminary experiments, we may not want to face complex experimental configurations. MRFI also provide simple API for course-grained fault injection experiment.
Read detail usage on [MRFI Documents >>](https://fffasttime.github.io/MRFI/)
Fault injection is an important classic method for studying reliability.
This is crucial for the design and deployment of neural networks for security critical applications.
The low-level hardware based fault injection methods are time-consuming,
while the high-level network model based methods usually neglect details.
Therefore, MRFI is designed to be an efficient and highly configurable multi-resolution network-level fault injection tool.
Learn more from our [paper of MRFI on Arxiv](https://arxiv.org/pdf/2306.11758.pdf). For other level fault injection tool, see also [Neural-Network-Reliability-Analysis-Toolbox](https://github.com/fffasttime/Neural-Network-Reliability-Analysis-Toolbox).
## Basic Example
### Install MRFI
```bash
# Install by pip
pip install MRFI
# Or download from github
git clone https://github.com/fffasttime/MRFI.git
pip install -r requirements.txt
```
### A coarse-grained configuation fault inject experiment
The following code perform a quantized random integer bit flip injection on LeNet,
and find the relation between bit error rate (BER) and classification accuracy.
```python title="LeNet default fault injection"
from dataset.lenet_cifar import make_testloader, LeNet
from mrfi import MRFI, EasyConfig
from mrfi.experiment import Acc_experiment, Acc_golden, BER_Acc_experiment
testloader = make_testloader(1000, batch_size = 128) # test on 1000 cifar-10 images
# Create fault inject model
fi_model = MRFI(LeNet(trained=True).eval(),
EasyConfig.load_preset('default_fi'))
################### A Simple acccuracy experiment #####################
# Test accuracy under fault injection with select rate = 1e-3 which specified in "default_fi"
print('FI Acc: ', Acc_experiment(fi_model, testloader))
# Test accuracy w/o fault inject
with fi_model.golden_run():
print('golden run Acc: ', Acc_experiment(fi_model, testloader))
######################## BER_Acc_experiment ###########################
# Get selector handler because BER_Acc_experiment needs to modify selection rate in experiment
selector_cfg = fi_model.get_activation_configs('selector')
BER, Acc = BER_Acc_experiment(fi_model, selector_cfg,
make_testloader(1000, batch_size = 128),
[1e-6, 1e-5, 1e-4, 1e-3])
print('Bit error rate and accuracy: ', BER, Acc)
```
A possible output should like is:
> FI Acc: 0.597
>
> golden run Acc: 0.638
>
> Bit error rate and accuracy: [1.e-06 1.e-05 1.e-04 1.e-03] [0.637 0.624 0.539 0.247]
The content of corresponding EasyConfig file `default_fi.yaml`:
```yaml
faultinject:
- type: activation
quantization:
method: SymmericQuantization
dynamic_range: auto
bit_width: 16
enabled: True
selector:
method: RandomPositionByRate
poisson: True
rate: 1e-3
error_mode:
method: IntSignBitFlip
bit_width: 16
module_name: [conv, fc] # Match layers that contain these name
module_type: [Conv2d, Linear] # Or match layers contain these typename
```
For detail usage, see [MRFI Doc - Usage](https://fffasttime.github.io/MRFI/usage/).
Explore MRFI internal functions in [MRFI Doc - function table](https://fffasttime.github.io/MRFI/function_table/).
## Supported Features
**Activation injection**
- [x] Fixed position (Permanent fault)
- [x] Runtime random position (Transient fault)
**Weight injection**
- [x] Fixed position (Permanent fault)
- [x] Runtime random position (Transient fault)
**Injection on quantization model**
- [x] Posting training quantization
- [x] Dynamic quantization
- [x] Fine-grained quantization parameters config
- [x] Add custom quantization
**Error mode**
- [x] Integer bit flip
- [x] Float bit flip
- [x] Stuck-at fault (SetValue)
- [x] Random value
- [x] Add random noise
**Internal observation & visualize**
- [x] Activation & Weight observer
- [x] Error propagation observer
- [x] Easy to save and visualize result, work well with `numpy` and `matplotlib`
**Flexibility**
- [x] Add custom `error_mode`, `selector`, `quantization` and `observer`
- [x] Distinguish network-level, layer-level, channel-level, neuron-level and bit-level fault tolerance difference
**Performance**
- [x] Automatically use GPU for network inference and fault injection
- [x] The selector - injector design is significantly faster than generate probability on all position when perform a random error injection
- [x] Accelerate error impact analysis through internal observer metrics rather than use original accuracy metric
**Fine-grained configuration**
- [x] By python code
- [x] By .yaml config file
- [ ] By GUI
**Evaluation fault tolerance policy**
- [x] Selective protection on different level
- [ ] More fault tolerance method may be support later (e.g. fault tolerant retrain, range-based filter)
## Developing
Code in `mrfi/` are tested to ensure correctness. Test case are under `test/`.
Set `logging.basicConfig(level = logging.DEBUG)` can get more runtime logging. It may be helpful when MRFI does not perform as expected.
### Unit Test
```bash
pytest --cov=mrfi --cov-report=html
```
coverage report written to `htmlconv/index.html`
### Update Docs
Modify markdown content files under `docs/`.
```bash
mkdocs build
mkdocs serve
mkdocs gh-deploy
```
### Bug Report
I tried to conduct thorough testing for MRFI, but due to the many features, there may still be potential issues.
If you find a problem, please provide sufficient code and configuration in the issue to analyze the cause, or submit a pull request to fix the problem.
Raw data
{
"_id": null,
"home_page": "",
"name": "MRFI",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "PyTorch,reliability,FaultInjection,FaultTolerance",
"author": "",
"author_email": "fffasttime <fffasttime@outlook.com>",
"download_url": "https://files.pythonhosted.org/packages/46/b8/f82cde4feb62faa2040d0129f85cd4fbbd0556bede62dbe7771b23a60bea/MRFI-2.0.0.tar.gz",
"platform": null,
"description": "# Multi-Resolution Fault Injector\r\n\r\n<p dir=\"auto\" align=\"center\">\r\n<img src=\"docs/assets/logo_name.png\" width=350)>\r\n</p>\r\n\r\n## MRFI Overview\r\n[](https://github.com/fffasttime/MRFI/blob/master/LICENSE)\r\n[](https://github.com/fffasttime/MRFI/actions/workflows/codecov.yml)\r\n[](https://codecov.io/gh/fffasttime/MRFI)\r\n\r\nMulti-Resolution Fault Injector is a **powerful** neural network fault injector tool based on PyTorch.\r\n\r\nCompared with previous network-level fault injection frameworks, the biggest feature is that MRFI can flexibly adjust different injection configurations for different experimental needs. \r\nInjection config and observations on each layer can be set **independently** by one clear config file. \r\nMRFI also provides a large number of commonly used fault injection methods and error models, and allows customization.\r\nIn preliminary experiments, we may not want to face complex experimental configurations. MRFI also provide simple API for course-grained fault injection experiment.\r\n\r\nRead detail usage on [MRFI Documents >>](https://fffasttime.github.io/MRFI/)\r\n\r\n\r\n\r\nFault injection is an important classic method for studying reliability. \r\nThis is crucial for the design and deployment of neural networks for security critical applications.\r\nThe low-level hardware based fault injection methods are time-consuming, \r\nwhile the high-level network model based methods usually neglect details.\r\nTherefore, MRFI is designed to be an efficient and highly configurable multi-resolution network-level fault injection tool.\r\n\r\nLearn more from our [paper of MRFI on Arxiv](https://arxiv.org/pdf/2306.11758.pdf). For other level fault injection tool, see also [Neural-Network-Reliability-Analysis-Toolbox](https://github.com/fffasttime/Neural-Network-Reliability-Analysis-Toolbox).\r\n\r\n## Basic Example\r\n\r\n### Install MRFI\r\n\r\n```bash\r\n# Install by pip\r\npip install MRFI\r\n\r\n# Or download from github\r\ngit clone https://github.com/fffasttime/MRFI.git\r\npip install -r requirements.txt\r\n```\r\n\r\n### A coarse-grained configuation fault inject experiment\r\nThe following code perform a quantized random integer bit flip injection on LeNet, \r\nand find the relation between bit error rate (BER) and classification accuracy.\r\n\r\n```python title=\"LeNet default fault injection\"\r\nfrom dataset.lenet_cifar import make_testloader, LeNet\r\nfrom mrfi import MRFI, EasyConfig\r\nfrom mrfi.experiment import Acc_experiment, Acc_golden, BER_Acc_experiment\r\n\r\ntestloader = make_testloader(1000, batch_size = 128) # test on 1000 cifar-10 images\r\n\r\n# Create fault inject model\r\nfi_model = MRFI(LeNet(trained=True).eval(), \r\n EasyConfig.load_preset('default_fi'))\r\n\r\n################### A Simple acccuracy experiment #####################\r\n# Test accuracy under fault injection with select rate = 1e-3 which specified in \"default_fi\"\r\nprint('FI Acc: ', Acc_experiment(fi_model, testloader))\r\n# Test accuracy w/o fault inject\r\nwith fi_model.golden_run():\r\n print('golden run Acc: ', Acc_experiment(fi_model, testloader))\r\n\r\n######################## BER_Acc_experiment ###########################\r\n# Get selector handler because BER_Acc_experiment needs to modify selection rate in experiment\r\nselector_cfg = fi_model.get_activation_configs('selector')\r\nBER, Acc = BER_Acc_experiment(fi_model, selector_cfg, \r\n make_testloader(1000, batch_size = 128), \r\n [1e-6, 1e-5, 1e-4, 1e-3])\r\nprint('Bit error rate and accuracy: ', BER, Acc)\r\n```\r\n\r\nA possible output should like is:\r\n> FI Acc: 0.597\r\n> \r\n> golden run Acc: 0.638\r\n>\r\n> Bit error rate and accuracy: [1.e-06 1.e-05 1.e-04 1.e-03] [0.637 0.624 0.539 0.247]\r\n\r\nThe content of corresponding EasyConfig file `default_fi.yaml`:\r\n```yaml\r\nfaultinject:\r\n - type: activation\r\n quantization:\r\n method: SymmericQuantization\r\n dynamic_range: auto\r\n bit_width: 16\r\n enabled: True\r\n selector:\r\n method: RandomPositionByRate\r\n poisson: True\r\n rate: 1e-3\r\n error_mode:\r\n method: IntSignBitFlip\r\n bit_width: 16\r\n\r\n module_name: [conv, fc] # Match layers that contain these name\r\n module_type: [Conv2d, Linear] # Or match layers contain these typename\r\n```\r\n\r\nFor detail usage, see [MRFI Doc - Usage](https://fffasttime.github.io/MRFI/usage/).\r\n\r\nExplore MRFI internal functions in [MRFI Doc - function table](https://fffasttime.github.io/MRFI/function_table/).\r\n\r\n## Supported Features\r\n\r\n**Activation injection**\r\n\r\n- [x] Fixed position (Permanent fault)\r\n- [x] Runtime random position (Transient fault)\r\n\r\n**Weight injection**\r\n\r\n- [x] Fixed position (Permanent fault)\r\n- [x] Runtime random position (Transient fault)\r\n\r\n**Injection on quantization model**\r\n\r\n- [x] Posting training quantization\r\n- [x] Dynamic quantization\r\n- [x] Fine-grained quantization parameters config\r\n- [x] Add custom quantization\r\n\r\n**Error mode**\r\n\r\n- [x] Integer bit flip\r\n- [x] Float bit flip\r\n- [x] Stuck-at fault (SetValue)\r\n- [x] Random value\r\n- [x] Add random noise\r\n\r\n**Internal observation & visualize**\r\n\r\n- [x] Activation & Weight observer\r\n- [x] Error propagation observer\r\n- [x] Easy to save and visualize result, work well with `numpy` and `matplotlib`\r\n\r\n**Flexibility**\r\n\r\n- [x] Add custom `error_mode`, `selector`, `quantization` and `observer`\r\n- [x] Distinguish network-level, layer-level, channel-level, neuron-level and bit-level fault tolerance difference\r\n\r\n**Performance**\r\n\r\n- [x] Automatically use GPU for network inference and fault injection\r\n- [x] The selector - injector design is significantly faster than generate probability on all position when perform a random error injection\r\n- [x] Accelerate error impact analysis through internal observer metrics rather than use original accuracy metric\r\n\r\n**Fine-grained configuration**\r\n\r\n- [x] By python code\r\n- [x] By .yaml config file\r\n- [ ] By GUI\r\n\r\n**Evaluation fault tolerance policy**\r\n\r\n- [x] Selective protection on different level\r\n- [ ] More fault tolerance method may be support later (e.g. fault tolerant retrain, range-based filter)\r\n\r\n\r\n## Developing\r\n\r\nCode in `mrfi/` are tested to ensure correctness. Test case are under `test/`.\r\n\r\nSet `logging.basicConfig(level = logging.DEBUG)` can get more runtime logging. It may be helpful when MRFI does not perform as expected.\r\n\r\n### Unit Test\r\n\r\n```bash\r\npytest --cov=mrfi --cov-report=html\r\n```\r\ncoverage report written to `htmlconv/index.html`\r\n\r\n### Update Docs\r\n\r\nModify markdown content files under `docs/`.\r\n```bash\r\nmkdocs build\r\nmkdocs serve\r\nmkdocs gh-deploy\r\n```\r\n\r\n### Bug Report\r\n\r\nI tried to conduct thorough testing for MRFI, but due to the many features, there may still be potential issues. \r\nIf you find a problem, please provide sufficient code and configuration in the issue to analyze the cause, or submit a pull request to fix the problem.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A multi-resolution DNN fault injection tool",
"version": "2.0.0",
"project_urls": {
"Documentation": "https://fffasttime.github.io/MRFI",
"Homepage": "https://github.com/fffasttime/MRFI",
"Repository": "https://github.com/fffasttime/MRFI.git"
},
"split_keywords": [
"pytorch",
"reliability",
"faultinjection",
"faulttolerance"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a3ae1bcaca707fee9ac8baa2676e6598bbcbb6df8f1631e012e19df2ae4e1d12",
"md5": "3a6dd0272af64c27c7310259e80c17fb",
"sha256": "0ad10e0622d3b524a70914151ae55e800513264ca54cf05eb4820f062330e488"
},
"downloads": -1,
"filename": "MRFI-2.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3a6dd0272af64c27c7310259e80c17fb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 25143,
"upload_time": "2023-07-15T15:13:16",
"upload_time_iso_8601": "2023-07-15T15:13:16.113430Z",
"url": "https://files.pythonhosted.org/packages/a3/ae/1bcaca707fee9ac8baa2676e6598bbcbb6df8f1631e012e19df2ae4e1d12/MRFI-2.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "46b8f82cde4feb62faa2040d0129f85cd4fbbd0556bede62dbe7771b23a60bea",
"md5": "80a53fab79d64c27d08b764bae1af9bd",
"sha256": "7d326b86cd1f6f8db76ad8351beb06af8f3dbf1588c278ad289aa20c4a25f05b"
},
"downloads": -1,
"filename": "MRFI-2.0.0.tar.gz",
"has_sig": false,
"md5_digest": "80a53fab79d64c27d08b764bae1af9bd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 1271706,
"upload_time": "2023-07-15T15:13:19",
"upload_time_iso_8601": "2023-07-15T15:13:19.395553Z",
"url": "https://files.pythonhosted.org/packages/46/b8/f82cde4feb62faa2040d0129f85cd4fbbd0556bede62dbe7771b23a60bea/MRFI-2.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-15 15:13:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fffasttime",
"github_project": "MRFI",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "mrfi"
}
