| Name | maite JSON |
| Version |
0.6.0
JSON |
| download |
| home_page | None |
| Summary | Library of common types, protocols (a.k.a. structural subtypes), and utilities to support AI test and evaluation |
| upload_time | 2024-06-14 18:19:44 |
| maintainer | Justin Goodwin, Michael Yee, Lei Hamilton, Jeff Arena |
| docs_url | None |
| author | Justin Goodwin, Michael Yee, Ryan Soklaski, Lei Hamilton, Jeff Arena, Garrett Botkin, Manasi Sharma |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2023 Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
machine learning
testing
pytest
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# MAITE (Modular AI Trustworthy Engineering)
<p align="center">
<a>
<img src="https://img.shields.io/badge/python-3.8%20‐%203.10-blue.svg" alt="Python version support" />
</a>
<a>
<img src="https://img.shields.io/badge/coverage-%3E90%25-green.svg" alt="Code Coverage" />
<a href="https://github.com/microsoft/pyright/blob/92b4028cd5fd483efcf3f1cdb8597b2d4edd8866/docs/typed-libraries.md#verifying-type-completeness">
<img src="https://img.shields.io/badge/type%20completeness-100%25-green.svg" alt="Type-Completeness Score" />
<a href="https://hypothesis.readthedocs.io/">
<img src="https://img.shields.io/badge/hypothesis-tested-brightgreen.svg" alt="Tested with Hypothesis" />
</a>
</p>
<p align="center">
A toolbox of common types, protocols, and tooling to support AI test and evaluation workflows.
</p>
<p align="center">
Check out the <a href="https://mit-ll-ai-technology.github.io/maite/">documentation</a> and
<a href="https://github.com/mit-ll-ai-technology/maite/tree/main/examples">examples</a> for more information.
</p>
</p>
MAITE is a library of common types, protocols (a.k.a. structural subtypes), and utilities for the test and evaluation (T&E) of supervised machine learning models. It is being developed under the [Joint AI T&E Infrastructure Capability (JATIC)](https://gitlab.jatic.net/home/) program. Its goal is to streamline the development of JATIC Python projects by ensuring seamless, synergistic workflows when working with MAITE-conforming Python packages for different T&E tasks. To this end, MAITE seeks to eliminate redundancies that would otherwise be shared across – and burden – separate efforts in machine learning test and evaluation. MAITE is designed to be a low-dependency, frequently-improved Python package that is installed by JATIC projects. The following is a brief overview of the current state of its submodules.
## Installation
### From Python Package Index (PyPI)
To install from the Python Package Index (PyPI), run:
```console
pip install maite
```
> :information_source: You can install MAITE for a given release tag, e.g. `v0.4.0`, by running:
>
>```console
>$ pip install git+ssh://git@github.com/mit-ll-ai-technology/maite.git@v0.4.0
>```
### From Source
To clone this repository and install from source, run:
```console
$ git clone https://github.com/mit-ll-ai-technology/maite
$ cd maite
$ pip install .
```
## maite.protocols
*Common types for machine learning test and evaluation*
The `protocols` subpackage defines common types – such as an inference-mode object detector – to be leveraged across JATIC projects. These are specifically designed to be [Python protocol classes](https://peps.python.org/pep-0544/), which support structural subtyping. As a result, developers and users can satisfy MAITE-typed interfaces without having to explicitly subclass. This ability helps to promote common interfaces across JATIC projects without introducing explicit inter-dependencies between them.
### ArrayLike
One example of a MAITE protocol class is `ArrayLike`. An `ArrayLike` defines a common interface for objects that can be manipulated as arrays, regardless of the specific implementation.
This allows code to be written in a more generic way, allowing it to work with different array-like objects without having to worry
about the details of the specific implementation. With an `ArrayLike` protocol, vendors can write functions and algorithms that
operate on arrays without JATIC defining the specific implementation of arrays to use.
```python
from maite.protocols import ArrayLike
# ArrayLike requires objects to implement `__array__`
assert not isinstance([1, 2, 3], ArrayLike)
# NumPy ndarray objects satisfy protocol
import numpy as np
np_array = np.zeros((10, 10, 3), dtype=np.uint8)
assert isinstance(np_array, ArrayLike)
# PyTorch Tensor objects satisfy protocol
import torch as tr
array = tr.as_tensor(np_array)
assert isinstance(array, ArrayLike)
```
## maite.testing
*Support for rigorous software testing*
The `testing` subpackage is designed to help developers create a rigorous automated test suite for their project. These include:
- Pytest fixtures for initializing test functions with common models, datasets, and other inputs that are useful for testing machine learning code.
- Functions running static type checking tests using [pyright](https://github.com/microsoft/pyright) in a pytest test suite, including scans of both source code and example documentation code blocks.
- [Hypothesis strategies](https://hypothesis.readthedocs.io/en/latest/) for driving property-based tests of interfaces that leverage MAITE protocols.
### Pyright Static Type Checking in Code
```python
>>> def f(x: str):
... return 1 + x
>>> pyright_analyze(f)
{'version': '1.1.281',
'time': '1669686515154',
'generalDiagnostics': [{'file': 'source.py',
'severity': 'error',
'message': 'Operator "+" not supported for types "Literal[1]" and "str"\n\xa0\xa0Operator "+" not supported for types "Literal[1]" and "str"',
'range': {'start': {'line': 1, 'character': 11},
'end': {'line': 1, 'character': 16}},
'rule': 'reportGeneralTypeIssues'}],
'summary': {'filesAnalyzed': 20,
'errorCount': 1,
'warningCount': 0,
'informationCount': 0,
'timeInSec': 0.319}}
```
## maite.utils
*General utilities*
- Functions for validating the types and values of user arguments, with explicit and consistent user-error messages, that raise MAITE-customized exceptions.
- Specialized PyTorch utilities to help facilitate safe and ergonomic code patterns for manipulating stateful torch objects
- Other quality assurance and convenience functions that may be widely useful across projects
## Disclaimer
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
© 2024 MASSACHUSETTS INSTITUTE OF TECHNOLOGY
* Subject to FAR 52.227-11 – Patent Rights – Ownership by the Contractor (May 2014)
* SPDX-License-Identifier: MIT
This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
The software/firmware is provided to you on an As-Is basis
Raw data
{
"_id": null,
"home_page": null,
"name": "maite",
"maintainer": "Justin Goodwin, Michael Yee, Lei Hamilton, Jeff Arena",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "machine learning, testing, pytest",
"author": "Justin Goodwin, Michael Yee, Ryan Soklaski, Lei Hamilton, Jeff Arena, Garrett Botkin, Manasi Sharma",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ac/45/d183a798a4bbb4ecee4fa642df8526a9fd0c004e945926d11fa810b17e15/maite-0.6.0.tar.gz",
"platform": null,
"description": "# MAITE (Modular AI Trustworthy Engineering)\n\n<p align=\"center\">\n <a>\n <img src=\"https://img.shields.io/badge/python-3.8%20‐%203.10-blue.svg\" alt=\"Python version support\" />\n </a>\n <a>\n <img src=\"https://img.shields.io/badge/coverage-%3E90%25-green.svg\" alt=\"Code Coverage\" />\n <a href=\"https://github.com/microsoft/pyright/blob/92b4028cd5fd483efcf3f1cdb8597b2d4edd8866/docs/typed-libraries.md#verifying-type-completeness\">\n <img src=\"https://img.shields.io/badge/type%20completeness-100%25-green.svg\" alt=\"Type-Completeness Score\" />\n <a href=\"https://hypothesis.readthedocs.io/\">\n <img src=\"https://img.shields.io/badge/hypothesis-tested-brightgreen.svg\" alt=\"Tested with Hypothesis\" />\n </a>\n </p>\n\n <p align=\"center\">\n A toolbox of common types, protocols, and tooling to support AI test and evaluation workflows.\n </p>\n\n <p align=\"center\">\n Check out the <a href=\"https://mit-ll-ai-technology.github.io/maite/\">documentation</a> and \n <a href=\"https://github.com/mit-ll-ai-technology/maite/tree/main/examples\">examples</a> for more information.\n </p>\n</p>\n\nMAITE is a library of common types, protocols (a.k.a. structural subtypes), and utilities for the test and evaluation (T&E) of supervised machine learning models. It is being developed under the [Joint AI T&E Infrastructure Capability (JATIC)](https://gitlab.jatic.net/home/) program. Its goal is to streamline the development of JATIC Python projects by ensuring seamless, synergistic workflows when working with MAITE-conforming Python packages for different T&E tasks. To this end, MAITE seeks to eliminate redundancies that would otherwise be shared across \u2013 and burden \u2013\u00a0separate efforts in machine learning test and evaluation. MAITE\u00a0is designed to be a low-dependency, frequently-improved Python package that is installed by JATIC projects. The following is a brief overview of the current state of its submodules.\n\n## Installation\n\n### From Python Package Index (PyPI)\nTo install from the Python Package Index (PyPI), run:\n\n```console\npip install maite\n```\n\n> :information_source: You can install MAITE for a given release tag, e.g. `v0.4.0`, by running:\n>\n>```console\n>$ pip install git+ssh://git@github.com/mit-ll-ai-technology/maite.git@v0.4.0\n>```\n\n### From Source\n\nTo clone this repository and install from source, run:\n\n```console\n$ git clone https://github.com/mit-ll-ai-technology/maite\n$ cd maite\n$ pip install .\n```\n\n## maite.protocols\n\n*Common types for machine learning test and evaluation*\n\nThe `protocols` subpackage defines common types \u2013 such as an inference-mode object detector \u2013 to be leveraged across JATIC projects. These are specifically designed to be [Python protocol classes](https://peps.python.org/pep-0544/), which support structural subtyping. As a result, developers and users can satisfy MAITE-typed interfaces without having to explicitly subclass. This ability helps to promote common interfaces across JATIC projects without introducing explicit inter-dependencies between them.\n\n### ArrayLike\n\nOne example of a MAITE protocol class is `ArrayLike`. An `ArrayLike` defines a common interface for objects that can be manipulated as arrays, regardless of the specific implementation.\nThis allows code to be written in a more generic way, allowing it to work with different array-like objects without having to worry\nabout the details of the specific implementation. With an `ArrayLike` protocol, vendors can write functions and algorithms that\noperate on arrays without JATIC defining the specific implementation of arrays to use.\n\n```python\nfrom maite.protocols import ArrayLike\n\n# ArrayLike requires objects to implement `__array__`\nassert not isinstance([1, 2, 3], ArrayLike)\n\n# NumPy ndarray objects satisfy protocol\nimport numpy as np\nnp_array = np.zeros((10, 10, 3), dtype=np.uint8)\nassert isinstance(np_array, ArrayLike)\n\n# PyTorch Tensor objects satisfy protocol \nimport torch as tr\narray = tr.as_tensor(np_array)\nassert isinstance(array, ArrayLike)\n```\n\n## maite.testing\n\n*Support for rigorous software testing*\n\nThe `testing` subpackage is designed to help developers create a rigorous automated test suite for their project. These include:\n\n- Pytest fixtures for initializing test functions with common models, datasets, and other inputs that are useful for testing machine learning code.\n- Functions running static type checking tests using [pyright](https://github.com/microsoft/pyright) in a pytest test suite, including scans of both source code and example documentation code blocks.\n- [Hypothesis strategies](https://hypothesis.readthedocs.io/en/latest/) for driving property-based tests of interfaces that leverage MAITE protocols.\n\n### Pyright Static Type Checking in Code\n\n```python\n>>> def f(x: str):\n... return 1 + x\n>>> pyright_analyze(f)\n{'version': '1.1.281',\n 'time': '1669686515154',\n 'generalDiagnostics': [{'file': 'source.py',\n 'severity': 'error',\n 'message': 'Operator \"+\" not supported for types \"Literal[1]\" and \"str\"\\n\\xa0\\xa0Operator \"+\" not supported for types \"Literal[1]\" and \"str\"',\n 'range': {'start': {'line': 1, 'character': 11},\n 'end': {'line': 1, 'character': 16}},\n 'rule': 'reportGeneralTypeIssues'}],\n 'summary': {'filesAnalyzed': 20,\n 'errorCount': 1,\n 'warningCount': 0,\n 'informationCount': 0,\n 'timeInSec': 0.319}}\n```\n\n## maite.utils\n\n*General utilities*\n\n- Functions for validating the types and values of user arguments, with explicit and consistent user-error messages, that raise MAITE-customized exceptions.\n- Specialized PyTorch utilities to help facilitate safe and ergonomic code patterns for manipulating stateful torch objects\n- Other quality assurance and convenience functions that may be widely useful across projects\n\n## Disclaimer\n\nDISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.\n\n\u00a9 2024 MASSACHUSETTS INSTITUTE OF TECHNOLOGY\n\n* Subject to FAR 52.227-11 \u2013 Patent Rights \u2013 Ownership by the Contractor (May 2014)\n* SPDX-License-Identifier: MIT\n\nThis material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.\n\nThe software/firmware is provided to you on an As-Is basis\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2023 Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Library of common types, protocols (a.k.a. structural subtypes), and utilities to support AI test and evaluation",
"version": "0.6.0",
"project_urls": {
"Bug Reports": "https://github.com/mit-ll-ai-technology/maite/issues",
"Homepage": "https://github.com/mit-ll-ai-technology/maite",
"Source": "https://github.com/mit-ll-ai-technology/maite"
},
"split_keywords": [
"machine learning",
" testing",
" pytest"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8d237d86b503866bfaee036107e7eaf8c37cf30e938eb5b4c8c23daa0154291c",
"md5": "afe4f82cf196b42c9ba18020241af46b",
"sha256": "518bd5071c4a619687c72b37e4512e655533d2ac82ba94ece4864eade2f755c9"
},
"downloads": -1,
"filename": "maite-0.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "afe4f82cf196b42c9ba18020241af46b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 60179,
"upload_time": "2024-06-14T18:19:42",
"upload_time_iso_8601": "2024-06-14T18:19:42.816052Z",
"url": "https://files.pythonhosted.org/packages/8d/23/7d86b503866bfaee036107e7eaf8c37cf30e938eb5b4c8c23daa0154291c/maite-0.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ac45d183a798a4bbb4ecee4fa642df8526a9fd0c004e945926d11fa810b17e15",
"md5": "f3a8eb17fa38c630f4f49f939d433bc9",
"sha256": "9e2dd28dbb38725ed63ce9e22d4b3299baa357ff290609ed1e4305ccc3d710f5"
},
"downloads": -1,
"filename": "maite-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "f3a8eb17fa38c630f4f49f939d433bc9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1716247,
"upload_time": "2024-06-14T18:19:44",
"upload_time_iso_8601": "2024-06-14T18:19:44.774753Z",
"url": "https://files.pythonhosted.org/packages/ac/45/d183a798a4bbb4ecee4fa642df8526a9fd0c004e945926d11fa810b17e15/maite-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-14 18:19:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mit-ll-ai-technology",
"github_project": "maite",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "maite"
}
