onnx-diagnostic: investigate onnx models
========================================
.. image:: https://github.com/sdpython/onnx-diagnostic/actions/workflows/documentation.yml/badge.svg
:target: https://github.com/sdpython/onnx-diagnostic/actions/workflows/documentation.yml
.. image:: https://badge.fury.io/py/onnx-diagnostic.svg
:target: http://badge.fury.io/py/onnx-diagnostic
.. image:: https://img.shields.io/badge/license-MIT-blue.svg
:alt: MIT License
:target: https://opensource.org/license/MIT/
.. image:: https://img.shields.io/github/repo-size/sdpython/onnx-diagnostic
:target: https://github.com/sdpython/onnx-diagnostic/
:alt: size
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
.. image:: https://codecov.io/gh/sdpython/onnx-diagnostic/graph/badge.svg?token=91T5ZVIP96
:target: https://codecov.io/gh/sdpython/onnx-diagnostic
The main feature is about `patches <https://github.com/sdpython/onnx-diagnostic/tree/main/onnx_diagnostic/torch_export_patches>`_:
it helps exporting **pytorch models into ONNX**, mostly designed for LLMs using dynamic caches.
Patches can be enabled as follows:
.. code-block:: python
from onnx_diagnostic.torch_export_patches import torch_export_patches
with torch_export_patches(patch_transformers=True) as f:
ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)
# ...
Dynamic shapes are difficult to guess for caches, one function
returns a structure defining all dimensions as dynamic.
You need then to remove those which are not dynamic in your model.
.. code-block:: python
from onnx_diagnostic.export.shape_helper import all_dynamic_shape_from_inputs
dynamic_shapes = all_dynamic_shape_from_inputs(cache)
It also implements tools to investigate, validate exported models (ExportedProgramm, ONNXProgram, ...).
See `documentation of onnx-diagnostic <https://sdpython.github.io/doc/onnx-diagnostic/dev/>`_ and
`torch_export_patches <https://sdpython.github.io/doc/onnx-diagnostic/dev/api/torch_export_patches/index.html#onnx_diagnostic.torch_export_patches.torch_export_patches>`_.
Getting started
+++++++++++++++
::
git clone https://github.com/sdpython/onnx-diagnostic.git
cd onnx-diagnostic
pip install -e .
or
::
pip install onnx-diagnostic
Enlightening Examples
+++++++++++++++++++++
**Where to start to export a model**
* `Export microsoft/phi-2
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_phi2.html>`_
**Torch Export**
* `Use DYNAMIC or AUTO when exporting if dynamic shapes has constraints
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_with_dynamic_shapes_auto.html>`_
* `Find and fix an export issue due to dynamic shapes
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_locate_issue.html>`_
* `Export with DynamicCache and guessed dynamic shapes
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_with_dynamic_cache.html>`_
* `Steel method forward to guess the dynamic shapes (with Tiny-LLM)
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_llm.html>`_
* `Export Tiny-LLM with patches
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_llm_patched.html>`_
**Investigate ONNX models**
* `Find where a model is failing by running submodels
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_model_extract.html>`_
* `Intermediate results with (ONNX) ReferenceEvaluator
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_reference_evaluator.html>`_
* `Intermediate results with onnxruntime
<https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_onnxruntime_evaluator.html>`_
Snapshot of usefuls tools
+++++++++++++++++++++++++
**torch_export_patches**
.. code-block:: python
from onnx_diagnostic.torch_export_patches import torch_export_patches
with torch_export_patches(patch_transformers=True) as f:
ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)
# ...
**all_dynamic_shape_from_inputs**
.. code-block:: python
from onnx_diagnostic.export.shape_helper import all_dynamic_shape_from_inputs
dynamic_shapes = all_dynamic_shape_from_inputs(cache)
**torch_export_rewrite**
.. code-block:: python
from onnx_diagnostic.torch_export_patches import torch_export_rewrite
with torch_export_rewrite(rewrite=[Model.forward]) as f:
ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)
# ...
**string_type**
.. code-block:: python
import torch
from onnx_diagnostic.helpers import string_type
inputs = (
torch.rand((3, 4), dtype=torch.float16),
[torch.rand((5, 6), dtype=torch.float16), torch.rand((5, 6, 7), dtype=torch.float16)],
)
# with shapes
print(string_type(inputs, with_shape=True))
::
>>> (T10s3x4,#2[T10s5x6,T10s5x6x7])
**onnx_dtype_name**
.. code-block:: python
import onnx
from onnx_diagnostic.helpers.onnx_helper import onnx_dtype_name
itype = onnx.TensorProto.BFLOAT16
print(onnx_dtype_name(itype))
print(onnx_dtype_name(7))
::
>>> BFLOAT16
>>> INT64
**max_diff**
.. code-block:: python
import torch
from onnx_diagnostic.helpers import max_diff
print(
max_diff(
(torch.Tensor([1, 2]), (torch.Tensor([1, 2]),)),
(torch.Tensor([1, 2]), (torch.Tensor([1, 2]),)),
)
)
::
>>> {"abs": 0.0, "rel": 0.0, "sum": 0.0, "n": 4.0, "dnan": 0.0}s
**guess_dynamic_shapes**
.. code-block:: python
inputs = [
(torch.randn((5, 6)), torch.randn((1, 6))),
(torch.randn((7, 8)), torch.randn((1, 8))),
]
ds = ModelInputs(model, inputs).guess_dynamic_shapes(auto="dim")
print(ds)
::
>>> (({0: 'dim_0I0', 1: 'dim_0I1'}, {1: 'dim_1I1'}), {})
Raw data
{
"_id": null,
"home_page": "https://github.com/sdpython/onnx-diagnostic",
"name": "onnx-diagnostic",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Xavier Dupr\u00e9",
"author_email": "xavier.dupre@gmail.com",
"download_url": null,
"platform": null,
"description": "onnx-diagnostic: investigate onnx models\n========================================\n\n.. image:: https://github.com/sdpython/onnx-diagnostic/actions/workflows/documentation.yml/badge.svg\n :target: https://github.com/sdpython/onnx-diagnostic/actions/workflows/documentation.yml\n\n.. image:: https://badge.fury.io/py/onnx-diagnostic.svg\n :target: http://badge.fury.io/py/onnx-diagnostic\n\n.. image:: https://img.shields.io/badge/license-MIT-blue.svg\n :alt: MIT License\n :target: https://opensource.org/license/MIT/\n\n.. image:: https://img.shields.io/github/repo-size/sdpython/onnx-diagnostic\n :target: https://github.com/sdpython/onnx-diagnostic/\n :alt: size\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/psf/black\n\n.. image:: https://codecov.io/gh/sdpython/onnx-diagnostic/graph/badge.svg?token=91T5ZVIP96 \n :target: https://codecov.io/gh/sdpython/onnx-diagnostic\n\nThe main feature is about `patches <https://github.com/sdpython/onnx-diagnostic/tree/main/onnx_diagnostic/torch_export_patches>`_:\nit helps exporting **pytorch models into ONNX**, mostly designed for LLMs using dynamic caches.\nPatches can be enabled as follows:\n\n.. code-block:: python\n\n from onnx_diagnostic.torch_export_patches import torch_export_patches\n\n with torch_export_patches(patch_transformers=True) as f:\n ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)\n # ...\n\nDynamic shapes are difficult to guess for caches, one function\nreturns a structure defining all dimensions as dynamic.\nYou need then to remove those which are not dynamic in your model.\n\n.. code-block:: python\n\n from onnx_diagnostic.export.shape_helper import all_dynamic_shape_from_inputs\n\n dynamic_shapes = all_dynamic_shape_from_inputs(cache)\n\nIt also implements tools to investigate, validate exported models (ExportedProgramm, ONNXProgram, ...).\nSee `documentation of onnx-diagnostic <https://sdpython.github.io/doc/onnx-diagnostic/dev/>`_ and\n`torch_export_patches <https://sdpython.github.io/doc/onnx-diagnostic/dev/api/torch_export_patches/index.html#onnx_diagnostic.torch_export_patches.torch_export_patches>`_.\n\nGetting started\n+++++++++++++++\n\n::\n\n git clone https://github.com/sdpython/onnx-diagnostic.git\n cd onnx-diagnostic\n pip install -e .\n\nor\n\n::\n\n pip install onnx-diagnostic\n\nEnlightening Examples\n+++++++++++++++++++++\n\n**Where to start to export a model**\n\n* `Export microsoft/phi-2\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_phi2.html>`_\n\n**Torch Export**\n\n* `Use DYNAMIC or AUTO when exporting if dynamic shapes has constraints\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_with_dynamic_shapes_auto.html>`_\n* `Find and fix an export issue due to dynamic shapes\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_locate_issue.html>`_\n* `Export with DynamicCache and guessed dynamic shapes\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_with_dynamic_cache.html>`_\n* `Steel method forward to guess the dynamic shapes (with Tiny-LLM)\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_llm.html>`_\n* `Export Tiny-LLM with patches\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_export_tiny_llm_patched.html>`_\n\n**Investigate ONNX models**\n\n* `Find where a model is failing by running submodels\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_model_extract.html>`_\n* `Intermediate results with (ONNX) ReferenceEvaluator\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_reference_evaluator.html>`_\n* `Intermediate results with onnxruntime\n <https://sdpython.github.io/doc/onnx-diagnostic/dev/auto_examples/plot_failing_onnxruntime_evaluator.html>`_\n\nSnapshot of usefuls tools\n+++++++++++++++++++++++++\n\n**torch_export_patches**\n\n.. code-block:: python\n\n from onnx_diagnostic.torch_export_patches import torch_export_patches\n\n with torch_export_patches(patch_transformers=True) as f:\n ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)\n # ...\n\n**all_dynamic_shape_from_inputs**\n\n.. code-block:: python\n\n from onnx_diagnostic.export.shape_helper import all_dynamic_shape_from_inputs\n\n dynamic_shapes = all_dynamic_shape_from_inputs(cache)\n\n**torch_export_rewrite**\n\n.. code-block:: python\n\n from onnx_diagnostic.torch_export_patches import torch_export_rewrite\n\n with torch_export_rewrite(rewrite=[Model.forward]) as f:\n ep = torch.export.export(model, args, kwargs=kwargs, dynamic_shapes=dynamic_shapes)\n # ...\n\n**string_type**\n\n.. code-block:: python\n\n import torch\n from onnx_diagnostic.helpers import string_type\n\n inputs = (\n torch.rand((3, 4), dtype=torch.float16),\n [torch.rand((5, 6), dtype=torch.float16), torch.rand((5, 6, 7), dtype=torch.float16)],\n )\n\n # with shapes\n print(string_type(inputs, with_shape=True))\n\n::\n\n >>> (T10s3x4,#2[T10s5x6,T10s5x6x7])\n\n**onnx_dtype_name**\n\n.. code-block:: python\n\n import onnx\n from onnx_diagnostic.helpers.onnx_helper import onnx_dtype_name\n\n itype = onnx.TensorProto.BFLOAT16\n print(onnx_dtype_name(itype))\n print(onnx_dtype_name(7))\n\n::\n\n >>> BFLOAT16\n >>> INT64\n\n**max_diff**\n\n.. code-block:: python\n\n import torch\n from onnx_diagnostic.helpers import max_diff\n\n print(\n max_diff(\n (torch.Tensor([1, 2]), (torch.Tensor([1, 2]),)),\n (torch.Tensor([1, 2]), (torch.Tensor([1, 2]),)),\n )\n )\n\n::\n\n >>> {\"abs\": 0.0, \"rel\": 0.0, \"sum\": 0.0, \"n\": 4.0, \"dnan\": 0.0}s\n\n**guess_dynamic_shapes**\n\n.. code-block:: python\n\n inputs = [\n (torch.randn((5, 6)), torch.randn((1, 6))),\n (torch.randn((7, 8)), torch.randn((1, 8))),\n ]\n ds = ModelInputs(model, inputs).guess_dynamic_shapes(auto=\"dim\")\n print(ds)\n\n::\n\n >>> (({0: 'dim_0I0', 1: 'dim_0I1'}, {1: 'dim_1I1'}), {})\n",
"bugtrack_url": null,
"license": null,
"summary": "Investigate ONNX models",
"version": "0.7.5",
"project_urls": {
"Homepage": "https://github.com/sdpython/onnx-diagnostic"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ccffc0debbe948287f31608a89fa8b17911da99e3c3eec13f78f65eb3488389c",
"md5": "52cb34c083f3e8c77240bec64b18482b",
"sha256": "1b1038a700c1d6ad850cd31d25e53a3baacd83a925efc0d8efbfa9364a73dd01"
},
"downloads": -1,
"filename": "onnx_diagnostic-0.7.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "52cb34c083f3e8c77240bec64b18482b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 370091,
"upload_time": "2025-07-11T18:26:20",
"upload_time_iso_8601": "2025-07-11T18:26:20.245414Z",
"url": "https://files.pythonhosted.org/packages/cc/ff/c0debbe948287f31608a89fa8b17911da99e3c3eec13f78f65eb3488389c/onnx_diagnostic-0.7.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-11 18:26:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sdpython",
"github_project": "onnx-diagnostic",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": []
},
{
"name": "onnx",
"specs": [
[
">=",
"1.16.0"
]
]
},
{
"name": "onnxruntime",
"specs": [
[
">=",
"1.21"
]
]
},
{
"name": "optree",
"specs": []
},
{
"name": "torch",
"specs": [
[
">=",
"2.7"
]
]
},
{
"name": "torch_geometric",
"specs": []
}
],
"lcname": "onnx-diagnostic"
}
JSON
