# graphpatch
`graphpatch` is a library for activation patching on [PyTorch](https://pytorch.org/docs/stable/index.html)
neural network models. You use it by first wrapping your model in a `PatchableGraph` and then running
operations in a context created by
`PatchableGraph.patch()`:
```
model = GPT2LMHeadModel.from_pretrained(
"gpt2-xl",
device_map="auto",
load_in_8bit=True,
torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")
inputs = tokenizer(
"The Eiffel Tower, located in", return_tensors="pt", padding=False
).to(torch.device("cuda"))
# Note that all arguments after the first are forwarded as example inputs
# to the model during compilation; use_cache and return_dict are arguments
# to GPT2LMHeadModel, not graphpatch-specific.
pg = PatchableGraph(model, **inputs, use_cache=False, return_dict=False)
# Applies two patches to the multiplication result within the activation function
# of the MLP in the 18th transformer layer. ProbePatch records the last observed value
# at the given node, while ZeroPatch zeroes out the value seen by downstream computations.
with pg.patch("transformer.h_17.mlp.act.mul_3": [probe := ProbePatch(), ZeroPatch()]):
output = pg(**inputs)
# Patches are applied in order. probe.activation holds the value prior
# to ZeroPatch zeroing it out.
print(probe.activation)
```
`graphpatch` can patch (or record) any intermediate Tensor value without manual modification of the
underlying model’s code. See full documentation [here](https://graphpatch.readthedocs.io/en/latest/).
# Requirements
`graphpatch` requires `torch>=2` as it uses [`torch.compile()`](https://pytorch.org/docs/stable/generated/torch.compile.html#torch-compile) to build the
computational graph it uses for activation patching. As of `torch>=2.1.0`,
Python 3.8–3.11 are supported. `torch==2.0.*` do not support compilation on Python 3.11; you
will get an exception if you try to use `graphpatch` on such a configuration.
`graphpatch` automatically supports models loaded with features supplied by `accelerate` and
`bitsandbytes`. For example, you can easily use `graphpatch` on multiple GPU's and with quantized
inference:
```
model = LlamaForCausalLM.from_pretrained(
model_path, device_map="auto", load_in_8bit=True, torch_dtype=torch.float16
)
pg = PatchableGraph(model, **example_inputs)
```
# Installation
`graphpatch` is available on [PyPI](https://pypi.org/project/graphpatch), and can be installed via `pip`:
```
pip install graphpatch
```
Optionally, you can install `graphpatch` with the "transformers" extra to select known compatible versions of `transformers`, `accelerate`, `bitsandbytes`, and miscellaneous optional requirements of these packages to quickly get started with multi-GPU and quantized inference on real-world models:
```
pip install graphpatch[transformers]
```
# Demos
See the [demos](https://github.com/evan-lloyd/graphpatch/tree/main/demos) for some practical usage examples.
# Documentation
See the full documentation on [Read the Docs](https://graphpatch.readthedocs.io/en/latest/).
Raw data
{
"_id": null,
"home_page": "https://www.graphpatch.dev",
"name": "graphpatch",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<3.12",
"maintainer_email": "",
"keywords": "mechanistic interpretability,interpretability,pytorch,torch,activation patch,ablation,transformer,large language model,llm",
"author": "Evan Lloyd",
"author_email": "evan.t.lloyd@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/88/52/b83e0c15379fccdc765924c7dfd545d929a3f8b239c0de94976c34b43b4a/graphpatch-0.1.0.tar.gz",
"platform": null,
"description": "# graphpatch\n\n`graphpatch` is a library for activation patching on [PyTorch](https://pytorch.org/docs/stable/index.html)\nneural network models. You use it by first wrapping your model in a `PatchableGraph` and then running\noperations in a context created by\n`PatchableGraph.patch()`:\n\n```\nmodel = GPT2LMHeadModel.from_pretrained(\n \"gpt2-xl\",\n device_map=\"auto\",\n load_in_8bit=True,\n torch_dtype=torch.float16\n)\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2-xl\")\ninputs = tokenizer(\n \"The Eiffel Tower, located in\", return_tensors=\"pt\", padding=False\n).to(torch.device(\"cuda\"))\n# Note that all arguments after the first are forwarded as example inputs\n# to the model during compilation; use_cache and return_dict are arguments\n# to GPT2LMHeadModel, not graphpatch-specific.\npg = PatchableGraph(model, **inputs, use_cache=False, return_dict=False)\n# Applies two patches to the multiplication result within the activation function\n# of the MLP in the 18th transformer layer. ProbePatch records the last observed value\n# at the given node, while ZeroPatch zeroes out the value seen by downstream computations.\nwith pg.patch(\"transformer.h_17.mlp.act.mul_3\": [probe := ProbePatch(), ZeroPatch()]):\n output = pg(**inputs)\n# Patches are applied in order. probe.activation holds the value prior\n# to ZeroPatch zeroing it out.\nprint(probe.activation)\n```\n\n`graphpatch` can patch (or record) any intermediate Tensor value without manual modification of the\nunderlying model\u2019s code. See full documentation [here](https://graphpatch.readthedocs.io/en/latest/).\n\n# Requirements\n`graphpatch` requires `torch>=2` as it uses [`torch.compile()`](https://pytorch.org/docs/stable/generated/torch.compile.html#torch-compile) to build the\ncomputational graph it uses for activation patching. As of `torch>=2.1.0`,\nPython 3.8–3.11 are supported. `torch==2.0.*` do not support compilation on Python 3.11; you\nwill get an exception if you try to use `graphpatch` on such a configuration.\n\n`graphpatch` automatically supports models loaded with features supplied by `accelerate` and\n`bitsandbytes`. For example, you can easily use `graphpatch` on multiple GPU's and with quantized\ninference:\n\n```\nmodel = LlamaForCausalLM.from_pretrained(\n model_path, device_map=\"auto\", load_in_8bit=True, torch_dtype=torch.float16\n)\npg = PatchableGraph(model, **example_inputs)\n```\n\n# Installation\n`graphpatch` is available on [PyPI](https://pypi.org/project/graphpatch), and can be installed via `pip`:\n```\npip install graphpatch\n```\n\nOptionally, you can install `graphpatch` with the \"transformers\" extra to select known compatible versions of `transformers`, `accelerate`, `bitsandbytes`, and miscellaneous optional requirements of these packages to quickly get started with multi-GPU and quantized inference on real-world models:\n```\npip install graphpatch[transformers]\n```\n\n# Demos\nSee the [demos](https://github.com/evan-lloyd/graphpatch/tree/main/demos) for some practical usage examples.\n\n# Documentation\nSee the full documentation on [Read the Docs](https://graphpatch.readthedocs.io/en/latest/).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "graphpatch is a library for activation patching on PyTorch neural network models.",
"version": "0.1.0",
"project_urls": {
"Documentation": "https://graphpatch.readthedocs.io/en/latest/index.html",
"Homepage": "https://www.graphpatch.dev",
"Repository": "https://github.com/evan-lloyd/graphpatch"
},
"split_keywords": [
"mechanistic interpretability",
"interpretability",
"pytorch",
"torch",
"activation patch",
"ablation",
"transformer",
"large language model",
"llm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fcc61b5b8db67bdb0a053db163d226768404715e2f97224f20195b1d2ac8e5a2",
"md5": "204eb2d399102bd9ecb7dc823d2efd31",
"sha256": "2fe6aa35a46a62527c8cce34b6f8366992023c1ab02ac5d4e600ea424539997c"
},
"downloads": -1,
"filename": "graphpatch-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "204eb2d399102bd9ecb7dc823d2efd31",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<3.12",
"size": 37715,
"upload_time": "2023-12-07T02:03:39",
"upload_time_iso_8601": "2023-12-07T02:03:39.483578Z",
"url": "https://files.pythonhosted.org/packages/fc/c6/1b5b8db67bdb0a053db163d226768404715e2f97224f20195b1d2ac8e5a2/graphpatch-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8852b83e0c15379fccdc765924c7dfd545d929a3f8b239c0de94976c34b43b4a",
"md5": "1ca6cfde4d8f8bce746bce368021e793",
"sha256": "e52113fa0a2de2d1482e7a3695fb385a67b1ff432067552e828bb522eeb93da1"
},
"downloads": -1,
"filename": "graphpatch-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "1ca6cfde4d8f8bce746bce368021e793",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<3.12",
"size": 34193,
"upload_time": "2023-12-07T02:03:41",
"upload_time_iso_8601": "2023-12-07T02:03:41.518733Z",
"url": "https://files.pythonhosted.org/packages/88/52/b83e0c15379fccdc765924c7dfd545d929a3f8b239c0de94976c34b43b4a/graphpatch-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-07 02:03:41",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "evan-lloyd",
"github_project": "graphpatch",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "graphpatch"
}
JSON
