# Distilling Model Failures as Directions in Latent Space
Python package for automatically finding failure modes in your dataset.
The paper: **[Distilling Model Failures as Directions in Latent Space](https://arxiv.org/abs/2206.14754)** and corresponding [blog post](https://gradientscience.org/failure-directions/)
See the [github repository](https://github.com/MadryLab/failure-directions) for documentation and examples!
## Getting Started
Install using pip, or clone our repository.
```
pip install failure-directions
```
Basic usage. Here, `svm_decision_values` are the alignment of the images to the identified failure modes.
```
import failure_directions
# let hparams contain mean and std for dataset
# let loaders contain a dict of train, test, val loaders.
# let split_gts contain the ground truth labels for each split
# let split_preds contain the predictions for each split
# Load CLIP features
clip_processor = failure_directions.CLIPProcessor(ds_mean=hparams['mean'], ds_std=hparams['std'])
clip_features = {}
for split, loader in loaders.items():
clip_features[split] = clip_processor.evaluate_clip_images(loader)
# Fit SVM
svm_fitter = failure_directions.SVMFitter()
svm_fitter.set_preprocess(clip_features['train'])
cv_scores = svm_fitter.fit(preds=split_preds['val'], ys=split_gts['val'], latents=clip_features['val'])
# Get SVM decision values
svm_predictions = {}
svm_decision_values = {}
for split, loader in loaders.items():
mask, dv = svm_fitter.predict(ys=split_gts[split], latents=clip_features[split], compute_metrics=False)
svm_predictions[split] = mask
svm_decision_values[split] = dv
```
## Citation
If use this package, please cite the corresponding paper
```
@inproceedings{jain2022distilling,
title = {Distilling Model Failures as Directions in Latent Space},
author = {Saachi Jain and Hannah Lawrence and Ankur Moitra and Aleksander Madry},
booktitle = {ArXiv preprint arXiv:2206.14754},
year = {2022}
}
```
## Maintainers
[Saachi Jain](https://twitter.com/saachi_jain_)<br>
[Hannah Lawrence](https://twitter.com/HLawrenceCS)
Raw data
{
"_id": null,
"home_page": "https://github.com/MadryLab/failure-directions",
"name": "failure-directions",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "MadryLab",
"author_email": "saachij@mit.edu",
"download_url": "https://files.pythonhosted.org/packages/a5/f3/8414cd55248e1754d30080d2f31077fcbb341876513efca8a73cddc93241/failure_directions-0.1.3.tar.gz",
"platform": null,
"description": "# Distilling Model Failures as Directions in Latent Space\nPython package for automatically finding failure modes in your dataset.\n\nThe paper: **[Distilling Model Failures as Directions in Latent Space](https://arxiv.org/abs/2206.14754)** and corresponding [blog post](https://gradientscience.org/failure-directions/) \n\nSee the [github repository](https://github.com/MadryLab/failure-directions) for documentation and examples!\n\n\n## Getting Started\nInstall using pip, or clone our repository.\n```\npip install failure-directions\n```\n \nBasic usage. Here, `svm_decision_values` are the alignment of the images to the identified failure modes. \n```\nimport failure_directions\n# let hparams contain mean and std for dataset\n# let loaders contain a dict of train, test, val loaders.\n# let split_gts contain the ground truth labels for each split\n# let split_preds contain the predictions for each split\n\n# Load CLIP features\nclip_processor = failure_directions.CLIPProcessor(ds_mean=hparams['mean'], ds_std=hparams['std'])\nclip_features = {}\nfor split, loader in loaders.items():\n clip_features[split] = clip_processor.evaluate_clip_images(loader)\n \n# Fit SVM\nsvm_fitter = failure_directions.SVMFitter()\nsvm_fitter.set_preprocess(clip_features['train'])\ncv_scores = svm_fitter.fit(preds=split_preds['val'], ys=split_gts['val'], latents=clip_features['val'])\n\n# Get SVM decision values\nsvm_predictions = {}\nsvm_decision_values = {}\nfor split, loader in loaders.items():\n mask, dv = svm_fitter.predict(ys=split_gts[split], latents=clip_features[split], compute_metrics=False)\n svm_predictions[split] = mask\n svm_decision_values[split] = dv\n```\n\n\n## Citation\nIf use this package, please cite the corresponding paper\n```\n@inproceedings{jain2022distilling,\n title = {Distilling Model Failures as Directions in Latent Space},\n author = {Saachi Jain and Hannah Lawrence and Ankur Moitra and Aleksander Madry}, \n booktitle = {ArXiv preprint arXiv:2206.14754},\n year = {2022}\n}\n```\n\n## Maintainers\n[Saachi Jain](https://twitter.com/saachi_jain_)<br>\n[Hannah Lawrence](https://twitter.com/HLawrenceCS)\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Failure Directions",
"version": "0.1.3",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "e061fb22e0e7d96301e71f8940b37922",
"sha256": "41d5b86d3842050c4f7c20347b799de4848669c95588e09a9e66858390d14bef"
},
"downloads": -1,
"filename": "failure_directions-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "e061fb22e0e7d96301e71f8940b37922",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 69095,
"upload_time": "2022-12-14T19:28:24",
"upload_time_iso_8601": "2022-12-14T19:28:24.428715Z",
"url": "https://files.pythonhosted.org/packages/a5/f3/8414cd55248e1754d30080d2f31077fcbb341876513efca8a73cddc93241/failure_directions-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-14 19:28:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "MadryLab",
"github_project": "failure-directions",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "failure-directions"
}
JSON
