| Name | auditora JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | None |
| Summary | Add your description here |
| upload_time | 2025-10-06 15:27:44 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.12 |
| license | MIT License Copyright (c) 2025 Mohamed Sofiene KADRI Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
ai
auditing
framework
monitoring
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Auditora: Non-invasive observability for LLM systems and data pipelines
[](https://badge.fury.io/py/levelapp)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
## Overview
A lightweight non-invasive instrumentation framework for data processing pipelines
and LLM-powered systems.<br/>
<br/>
Auditora provides:<br/>
✅ Context-aware architecture using Python's contextvars<br/>
✅ Thread and async-safe execution contexts<br/>
✅ Pluggable Adats (Session, Monitor, Report)<br/>
✅ Intelligent Sentinel decorator with sync/async detection<br/>
✅ Proper token-based context management via Bifrost<br/>
✅ PyPI-ready packaging with UV<br/>
<br/>
## Context
Auditora leverages Python's **Context Variables** (introduced in Python 3.7) to provide
execution-context-local storage that ensures complete isolation across threads,
async coroutines, and nested contexts. Unlike traditional thread-local storage,
Auditora's context management works seamlessly with both synchronous and asynchronous code,
making it ideal for modern LLM applications and data processing pipelines.
## Features
- **Paragon**: Context variable storage manager with proper token-based context stack management
- **Adats**: Pluggable session, monitor, and report components with LLM-specific utilities
- **Bifrost**: Dual sync/async context managers for clean setup and teardown
- **Sentinel**: Intelligent decorator that automatically detects sync/async functions
- **Seamless Integration**: Use global `session`, `monitor`, `report` objects without parameters
- **LLM-Optimized**: Built-in support for LLM API calls, evaluation metrics, and multi-agent systems
- **Nested Context Support**: Proper context stack management enables complex evaluation scenarios
- **Thread & Async Safe**: Works correctly in multi-threaded and async/await environments
## Architecture
- **Paragon** (The Guardian): Context variable storage manager that maintains isolated execution contexts using Python's `contextvars`
- **Adats** (The Weavers): Pluggable components for state management (`Session`), performance monitoring (`Monitor`), and structured reporting (`Report`)
- **Bifrost** (The Bridge): Dual sync/async context managers that provide clean context boundaries with proper token-based restoration
- **Sentinel** (The Watcher): Intelligent decorator that automatically wraps functions with appropriate context management based on sync/async detection
## Installation
```bash
pip install auditora
```
## Quick Start
### Basic Usage
```Python
from auditora import sentinel, session, monitor, report
@sentinel()
def evaluate_llm_response(response: str):
report.info("Starting evaluation")
session.set('response', response)
# Simulate evaluation
score = len(response) / 100.0
monitor.increment_metric("coherence_score", score)
report.log_evaluation_result("coherence", score, threshold=0.5)
return score
# Usage
result = evaluate_llm_response("This is a sample LLM response.")
print(f"Session ID: {evaluate_llm_response._session.session_id}")
```
### Async Support
```Python
import asyncio
from auditora import sentinel, session, monitor, report
@sentinel()
async def async_llm_evaluation(query: str):
report.info(f"Processing async query: {query}")
# Simulate async LLM call
await asyncio.sleep(0.1)
session.set('query', query)
monitor.track('async_processing_completed', query=query)
report.log_llm_call(
model="gpt-4",
prompt_tokens=len(query.split()),
completion_tokens=50,
response_time=0.15
)
return {"status": "completed", "query": query}
# Run async function
asyncio.run(async_llm_evaluation("What is the meaning of life?"))
```
### Nested Context
```Python
from auditora import sentinel, bifrost_sync, session, monitor, report
@sentinel(session_id="inner_eval")
def inner_evaluation():
session.set('inner_data', 'sub_evaluation')
report.info("Inner evaluation running")
@sentinel(session_id="outer_eval")
def outer_evaluation():
session.set('outer_data', 'main_evaluation')
# Nested sub-evaluation call with its proper context management
inner_evaluation()
# Context automatically restored to outer evaluation
report.info(f"Back to outer context: {session.get('outer_data')}")
```
## Advanced Configuration
```Python
from auditora.adata.session import DefaultSession
class CustomSession(DefaultSession):
def __init__(self, session_id: str = None):
super().__init__(session_id)
self.custom_counter = 0
def increment_counter(self):
self.custom_counter += 1
@sentinel(session=CustomSession("custom_session"))
def custom_evaluation():
session.increment_counter()
report.info(f"Counter: {session.custom_counter}")
```
### Session ID Management
```Python
@sentinel(session_id="my_unique_session_123")
def tracked_evaluation():
report.info(f"Running in session: {session.session_id}")
```
## Design Philosophy
Auditora follows the principle of non-invasive observability:
- **Zero parameter pollution**: Functions don't need context parameters
- **Automatic context management**: No manual setup/teardown required
- **Transparent integration**: Existing code works with minimal changes
- **Runtime safety**: Clear error messages when used incorrectly
- **Performance conscious**: Minimal overhead for monitoring operations
## Requirements
- Python 3.7+
- No external dependencies
## Use Cases
- **LLM Evaluation Pipelines**: Track metrics, log API calls, manage evaluation state
- **Multi-Agent Systems**: Monitor agent interactions and coordination
- **Data Processing Workflows**: Trace pipeline stages and performance metrics
- **API Monitoring**: Log structured metrics for LLM-powered endpoints
- **Testing and Debugging**: Inspect context objects externally for validation
## Acknowledgments
- Powered by [Norma](https://norma.dev).
## License
This project is licensed under the MIT License - see the [LICENCE](LICENCE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "auditora",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "ai, auditing, framework, monitoring",
"author": null,
"author_email": "Mohamed Sofiene KADRI <ms.kadri.dev@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/46/64/c7212d9d6aa111d971e15b07010a01d515fdb81bd23622deba7baf91bcef/auditora-0.1.0.tar.gz",
"platform": null,
"description": "# Auditora: Non-invasive observability for LLM systems and data pipelines\n\n[](https://badge.fury.io/py/levelapp) \n[](https://opensource.org/licenses/MIT) \n[](https://www.python.org/downloads/)\n\n## Overview\nA lightweight non-invasive instrumentation framework for data processing pipelines \nand LLM-powered systems.<br/> \n<br/> \nAuditora provides:<br/> \n\u2705 Context-aware architecture using Python's contextvars<br/> \n\u2705 Thread and async-safe execution contexts<br/> \n\u2705 Pluggable Adats (Session, Monitor, Report)<br/> \n\u2705 Intelligent Sentinel decorator with sync/async detection<br/> \n\u2705 Proper token-based context management via Bifrost<br/> \n\u2705 PyPI-ready packaging with UV<br/> \n<br/> \n## Context\nAuditora leverages Python's **Context Variables** (introduced in Python 3.7) to provide \nexecution-context-local storage that ensures complete isolation across threads, \nasync coroutines, and nested contexts. Unlike traditional thread-local storage, \nAuditora's context management works seamlessly with both synchronous and asynchronous code, \nmaking it ideal for modern LLM applications and data processing pipelines.\n\n## Features\n\n- **Paragon**: Context variable storage manager with proper token-based context stack management\n- **Adats**: Pluggable session, monitor, and report components with LLM-specific utilities\n- **Bifrost**: Dual sync/async context managers for clean setup and teardown\n- **Sentinel**: Intelligent decorator that automatically detects sync/async functions\n- **Seamless Integration**: Use global `session`, `monitor`, `report` objects without parameters\n- **LLM-Optimized**: Built-in support for LLM API calls, evaluation metrics, and multi-agent systems\n- **Nested Context Support**: Proper context stack management enables complex evaluation scenarios\n- **Thread & Async Safe**: Works correctly in multi-threaded and async/await environments\n\n## Architecture\n\n- **Paragon** (The Guardian): Context variable storage manager that maintains isolated execution contexts using Python's `contextvars`\n- **Adats** (The Weavers): Pluggable components for state management (`Session`), performance monitoring (`Monitor`), and structured reporting (`Report`)\n- **Bifrost** (The Bridge): Dual sync/async context managers that provide clean context boundaries with proper token-based restoration\n- **Sentinel** (The Watcher): Intelligent decorator that automatically wraps functions with appropriate context management based on sync/async detection\n\n## Installation\n\n```bash\n pip install auditora\n```\n\n## Quick Start\n### Basic Usage\n```Python\nfrom auditora import sentinel, session, monitor, report\n\n@sentinel()\ndef evaluate_llm_response(response: str):\n report.info(\"Starting evaluation\")\n session.set('response', response)\n \n # Simulate evaluation\n score = len(response) / 100.0\n monitor.increment_metric(\"coherence_score\", score)\n report.log_evaluation_result(\"coherence\", score, threshold=0.5)\n \n return score\n\n# Usage\nresult = evaluate_llm_response(\"This is a sample LLM response.\")\nprint(f\"Session ID: {evaluate_llm_response._session.session_id}\")\n```\n\n### Async Support\n```Python\nimport asyncio\nfrom auditora import sentinel, session, monitor, report\n\n@sentinel()\nasync def async_llm_evaluation(query: str):\n report.info(f\"Processing async query: {query}\")\n \n # Simulate async LLM call\n await asyncio.sleep(0.1)\n \n session.set('query', query)\n monitor.track('async_processing_completed', query=query)\n report.log_llm_call(\n model=\"gpt-4\",\n prompt_tokens=len(query.split()),\n completion_tokens=50,\n response_time=0.15\n )\n \n return {\"status\": \"completed\", \"query\": query}\n\n# Run async function\nasyncio.run(async_llm_evaluation(\"What is the meaning of life?\"))\n```\n\n### Nested Context\n```Python\nfrom auditora import sentinel, bifrost_sync, session, monitor, report\n\n\n@sentinel(session_id=\"inner_eval\")\ndef inner_evaluation():\n session.set('inner_data', 'sub_evaluation')\n report.info(\"Inner evaluation running\")\n\n@sentinel(session_id=\"outer_eval\")\ndef outer_evaluation():\n session.set('outer_data', 'main_evaluation')\n \n # Nested sub-evaluation call with its proper context management\n inner_evaluation()\n \n # Context automatically restored to outer evaluation\n report.info(f\"Back to outer context: {session.get('outer_data')}\")\n```\n\n## Advanced Configuration\n```Python\nfrom auditora.adata.session import DefaultSession\n\nclass CustomSession(DefaultSession):\n def __init__(self, session_id: str = None):\n super().__init__(session_id)\n self.custom_counter = 0\n \n def increment_counter(self):\n self.custom_counter += 1\n\n@sentinel(session=CustomSession(\"custom_session\"))\ndef custom_evaluation():\n session.increment_counter()\n report.info(f\"Counter: {session.custom_counter}\")\n```\n\n### Session ID Management\n```Python\n@sentinel(session_id=\"my_unique_session_123\")\ndef tracked_evaluation():\n report.info(f\"Running in session: {session.session_id}\")\n```\n\n## Design Philosophy\nAuditora follows the principle of non-invasive observability:\n\n- **Zero parameter pollution**: Functions don't need context parameters\n- **Automatic context management**: No manual setup/teardown required\n- **Transparent integration**: Existing code works with minimal changes\n- **Runtime safety**: Clear error messages when used incorrectly\n- **Performance conscious**: Minimal overhead for monitoring operations\n\n## Requirements\n- Python 3.7+\n- No external dependencies\n\n## Use Cases\n- **LLM Evaluation Pipelines**: Track metrics, log API calls, manage evaluation state\n- **Multi-Agent Systems**: Monitor agent interactions and coordination\n- **Data Processing Workflows**: Trace pipeline stages and performance metrics\n- **API Monitoring**: Log structured metrics for LLM-powered endpoints\n- **Testing and Debugging**: Inspect context objects externally for validation\n\n## Acknowledgments\n\n- Powered by [Norma](https://norma.dev).\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENCE](LICENCE) file for details.\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2025 Mohamed Sofiene KADRI Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Add your description here",
"version": "0.1.0",
"project_urls": {
"Repository": "https://github.com/KadriSof/Auditora"
},
"split_keywords": [
"ai",
" auditing",
" framework",
" monitoring"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "64c0fc613e91a5dc4df6085850ad5a0a70dfaa68a51f32ede5ce1f39b733990b",
"md5": "a3ba0833e12700d02fd4fff6c47b7582",
"sha256": "8e7cf782f04d54368a9d5d4ccb6ffd462ebd70d8969e742f91e2913f087fd8f7"
},
"downloads": -1,
"filename": "auditora-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a3ba0833e12700d02fd4fff6c47b7582",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 12595,
"upload_time": "2025-10-06T15:27:42",
"upload_time_iso_8601": "2025-10-06T15:27:42.008460Z",
"url": "https://files.pythonhosted.org/packages/64/c0/fc613e91a5dc4df6085850ad5a0a70dfaa68a51f32ede5ce1f39b733990b/auditora-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4664c7212d9d6aa111d971e15b07010a01d515fdb81bd23622deba7baf91bcef",
"md5": "20b82a87e7b00561521d62963440192a",
"sha256": "30ef67892bf5760e177c26e6a716113963f3e85dfe5dba379975f8eb49e443a7"
},
"downloads": -1,
"filename": "auditora-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "20b82a87e7b00561521d62963440192a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 12151,
"upload_time": "2025-10-06T15:27:44",
"upload_time_iso_8601": "2025-10-06T15:27:44.127774Z",
"url": "https://files.pythonhosted.org/packages/46/64/c7212d9d6aa111d971e15b07010a01d515fdb81bd23622deba7baf91bcef/auditora-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-06 15:27:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "KadriSof",
"github_project": "Auditora",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "auditora"
}
