| Name | airt-utils JSON |
| Version |
0.1.6
JSON |
| download |
| home_page | None |
| Summary | Utilities for experiment-centric AI Red Teaming workflows |
| upload_time | 2024-08-28 14:46:37 |
| maintainer | None |
| docs_url | None |
| author | Joe Lucas |
| requires_python | <4.0,>=3.10 |
| license | MIT |
| keywords |
mlflow
logging
ai-red-team
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# airt_utils
For research-centric AI Red Teaming operations, there are best practices we can borrow from the evolution of MLOps. For example, both workflows involve multiple researchers focused on optimizations against open-ended research questions. At some point, the work from these researchers must be compared, potentially aggregated, and the best solution iterated upon. Often there is also a reporting phase where stakeholders need to be able to access R&D artifacts.
`airt_utils` is a collection of Python utilities for tracking AI Red Teaming experiments and operations, starting with the `airt_run` decorator that seamlessly integrates your functions with MLflow. It provides automatic logging of parameters, metrics, and artifacts, supporting both synchronous and asynchronous functions. This makes it ideal for a wide range of machine learning and data science workflows.
## Features
- 🚀 Easy integration with MLflow through the simple `airt_run` decorator
- 🔄 Support for both synchronous and asynchronous functions
- ⏱️ Optional timeout functionality
- 📊 Automatic logging of function parameters and return values
- 📁 Artifact logging support
- 🏷️ Custom tagging for runs
- 📈 Support for custom and system metrics
- 🔁 Retry mechanism for improved reliability
- 🛠️ Highly configurable to suit various use cases
## Installation
Install `airt_utils` using pip:
```bash
pip install airt_utils
```
## Quick Start
Here's a simple example of how to use the `airt_run` decorator:
```python
from airt_utils import airt_run
import random
@airt_run(
experiment_name="quick_start_example",
params=["n"],
artifacts=["output.txt"],
)
def generate_random_numbers(n):
numbers = [random.randint(1, 100) for _ in range(n)]
with open("output.txt", "w") as f:
f.write(str(numbers))
return sum(numbers)
result = generate_random_numbers(10)
print(f"Sum of generated numbers: {result}")
```
This example will:
1. Create an MLflow experiment named "quick_start_example"
2. Log the parameter `n`
3. Run the function and log its return value
4. Log the generated "output.txt" file as an artifact
## Advanced Usage
### Asynchronous Functions
`airt_run` supports async functions out of the box:
```python
import asyncio
from airt_utils import airt_run
@airt_run(
experiment_name="async_example",
timeout_seconds=5
)
async def async_operation():
await asyncio.sleep(2)
return "Async operation completed"
result = async_operation()
print(result)
```
### Custom Metrics
You can define custom metrics to be logged:
```python
from airt_utils import airt_run
def mean_metric(result):
return sum(result) / len(result)
@airt_run(
experiment_name="custom_metrics_example",
custom_metrics={"mean": mean_metric}
)
def process_data(data):
return [x * 2 for x in data]
result = process_data([1, 2, 3, 4, 5])
```
This will log the mean of the processed data as a custom metric.
## Configuration
`airt_run` offers various configuration options:
- `experiment_name`: Name of the MLflow experiment
- `params`: List of function parameters to log
- `artifacts`: List of files or directories to log as artifacts (protip: log entire configuration directories and debug logs for target applications)
- `tracking_uri`: MLflow tracking server URI
- `timeout_seconds`: Maximum execution time for the function
- `tags`: Dictionary of tags to apply to the run
- `log_system_metrics`: Whether to log system metrics
- `retry_attempts`: Number of retry attempts for MLflow operations
- `retry_delay`: Delay between retry attempts
- `custom_metrics`: Dictionary of custom metrics to compute and log
Raw data
{
"_id": null,
"home_page": null,
"name": "airt-utils",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "mlflow, logging, ai-red-team",
"author": "Joe Lucas",
"author_email": "joe@joetl.com",
"download_url": "https://files.pythonhosted.org/packages/f9/a5/e89ce3f7b4c2850bfff1afc036f5ffc58de55f2ced96d00397f28c240d8a/airt_utils-0.1.6.tar.gz",
"platform": null,
"description": "# airt_utils\n\nFor research-centric AI Red Teaming operations, there are best practices we can borrow from the evolution of MLOps. For example, both workflows involve multiple researchers focused on optimizations against open-ended research questions. At some point, the work from these researchers must be compared, potentially aggregated, and the best solution iterated upon. Often there is also a reporting phase where stakeholders need to be able to access R&D artifacts.\n\n`airt_utils` is a collection of Python utilities for tracking AI Red Teaming experiments and operations, starting with the `airt_run` decorator that seamlessly integrates your functions with MLflow. It provides automatic logging of parameters, metrics, and artifacts, supporting both synchronous and asynchronous functions. This makes it ideal for a wide range of machine learning and data science workflows.\n\n## Features\n\n- \ud83d\ude80 Easy integration with MLflow through the simple `airt_run` decorator\n- \ud83d\udd04 Support for both synchronous and asynchronous functions\n- \u23f1\ufe0f Optional timeout functionality\n- \ud83d\udcca Automatic logging of function parameters and return values\n- \ud83d\udcc1 Artifact logging support\n- \ud83c\udff7\ufe0f Custom tagging for runs\n- \ud83d\udcc8 Support for custom and system metrics\n- \ud83d\udd01 Retry mechanism for improved reliability\n- \ud83d\udee0\ufe0f Highly configurable to suit various use cases\n\n## Installation\n\nInstall `airt_utils` using pip:\n\n```bash\npip install airt_utils\n```\n\n## Quick Start\n\nHere's a simple example of how to use the `airt_run` decorator:\n\n```python\nfrom airt_utils import airt_run\nimport random\n\n@airt_run(\n experiment_name=\"quick_start_example\",\n params=[\"n\"],\n artifacts=[\"output.txt\"],\n)\ndef generate_random_numbers(n):\n numbers = [random.randint(1, 100) for _ in range(n)]\n with open(\"output.txt\", \"w\") as f:\n f.write(str(numbers))\n return sum(numbers)\n\nresult = generate_random_numbers(10)\nprint(f\"Sum of generated numbers: {result}\")\n```\n\nThis example will:\n1. Create an MLflow experiment named \"quick_start_example\"\n2. Log the parameter `n`\n3. Run the function and log its return value\n4. Log the generated \"output.txt\" file as an artifact\n\n## Advanced Usage\n\n### Asynchronous Functions\n\n`airt_run` supports async functions out of the box:\n\n```python\nimport asyncio\nfrom airt_utils import airt_run\n\n@airt_run(\n experiment_name=\"async_example\",\n timeout_seconds=5\n)\nasync def async_operation():\n await asyncio.sleep(2)\n return \"Async operation completed\"\n\nresult = async_operation()\nprint(result)\n```\n\n### Custom Metrics\n\nYou can define custom metrics to be logged:\n\n```python\nfrom airt_utils import airt_run\n\ndef mean_metric(result):\n return sum(result) / len(result)\n\n@airt_run(\n experiment_name=\"custom_metrics_example\",\n custom_metrics={\"mean\": mean_metric}\n)\ndef process_data(data):\n return [x * 2 for x in data]\n\nresult = process_data([1, 2, 3, 4, 5])\n```\n\nThis will log the mean of the processed data as a custom metric.\n\n## Configuration\n\n`airt_run` offers various configuration options:\n\n- `experiment_name`: Name of the MLflow experiment\n- `params`: List of function parameters to log\n- `artifacts`: List of files or directories to log as artifacts (protip: log entire configuration directories and debug logs for target applications)\n- `tracking_uri`: MLflow tracking server URI\n- `timeout_seconds`: Maximum execution time for the function\n- `tags`: Dictionary of tags to apply to the run\n- `log_system_metrics`: Whether to log system metrics\n- `retry_attempts`: Number of retry attempts for MLflow operations\n- `retry_delay`: Delay between retry attempts\n- `custom_metrics`: Dictionary of custom metrics to compute and log\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Utilities for experiment-centric AI Red Teaming workflows",
"version": "0.1.6",
"project_urls": null,
"split_keywords": [
"mlflow",
" logging",
" ai-red-team"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bf0f9f0a925cc4080ed16919d11293a9f84a1e1a9de82eff1ebf30e675e5de75",
"md5": "5781046858e08f509300f561cfe803ca",
"sha256": "7987093d76e962f799c4ba4dd0f93612f02586f56ba76bb3c83b5ca2858ad3c7"
},
"downloads": -1,
"filename": "airt_utils-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5781046858e08f509300f561cfe803ca",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 6667,
"upload_time": "2024-08-28T14:46:36",
"upload_time_iso_8601": "2024-08-28T14:46:36.091225Z",
"url": "https://files.pythonhosted.org/packages/bf/0f/9f0a925cc4080ed16919d11293a9f84a1e1a9de82eff1ebf30e675e5de75/airt_utils-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f9a5e89ce3f7b4c2850bfff1afc036f5ffc58de55f2ced96d00397f28c240d8a",
"md5": "7f46948074756361a91490c67d8289da",
"sha256": "b4eea618eef634fd53fec7594813ba53035a2fba0914ea98d4aeea50818ea1a0"
},
"downloads": -1,
"filename": "airt_utils-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "7f46948074756361a91490c67d8289da",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 5442,
"upload_time": "2024-08-28T14:46:37",
"upload_time_iso_8601": "2024-08-28T14:46:37.449206Z",
"url": "https://files.pythonhosted.org/packages/f9/a5/e89ce3f7b4c2850bfff1afc036f5ffc58de55f2ced96d00397f28c240d8a/airt_utils-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-28 14:46:37",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "airt-utils"
}
