| Name | rogue-ai-sdk JSON |
| Version |
0.1.13
JSON |
| download |
| home_page | None |
| Summary | Python SDK for Rogue Agent Evaluator |
| upload_time | 2025-10-22 13:33:59 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | Qualifire OSS License
MIT License
Copyright (c) 2025 — Qualifire ltd
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"Commons Clause" License Condition v1.0
The Software is provided to you by the Licensor under the License (defined below), subject to the following condition:
Without limiting other conditions in the License, the grant of rights under the License will not include, and the License does not grant to you, the right to Sell the Software.
For purposes of the foregoing, "Sell" means practicing any or all of the rights granted to you under the License to provide the Software to third parties, for a fee or other consideration (including without limitation fees for hosting or consulting/support services related to the Software), as part of a product or service whose value derives, entirely or substantially, from the functionality of the Software. Any license notice or attribution required by the License must also include this Commons Clause License Condition notice.
Software: All Rogue associated files (including all files in the GitHub repository "rogue" and in the pypi package "rogue-ai").
License: MIT
Licensor: Qualifire ltd |
| keywords |
agent
evaluator
python
rogue
sdk
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Rogue Agent Evaluator Python SDK
A comprehensive Python SDK for interacting with the Rogue Agent Evaluator API.
## Installation
```bash
pip install rogue-sdk
```
## Quick Start
```python
import asyncio
from rogue_sdk import RogueSDK, RogueClientConfig, AuthType, ScenarioType
async def main():
# Configure the SDK
config = RogueClientConfig(base_url="http://localhost:8000")
async with RogueSDK(config) as client:
# Quick evaluation
result = await client.quick_evaluate(
agent_url="http://localhost:3000",
scenarios=[
"The agent should be polite",
"The agent should not give discounts"
]
)
print(f"Evaluation completed: {result.status}")
print(f"Results: {len(result.results)} scenarios evaluated")
if __name__ == "__main__":
asyncio.run(main())
```
## Features
- **HTTP Client**: Full REST API support with automatic retries
- **WebSocket Client**: Real-time updates during evaluations
- **Type Safety**: Comprehensive type definitions with Pydantic
- **Async/Await**: Modern Python async support
- **Error Handling**: Robust error handling and retry logic
- **High-level Methods**: Convenient methods for common operations
## API Reference
### RogueSDK
Main SDK class that combines HTTP and WebSocket functionality.
#### Configuration
```python
from rogue_sdk import RogueClientConfig
config = RogueClientConfig(
base_url="http://localhost:8000",
api_key="your-api-key", # Optional
timeout=30.0, # Request timeout in seconds
retries=3 # Number of retry attempts
)
```
#### Basic Operations
```python
async with RogueSDK(config) as client:
# Health check
health = await client.health()
# Create evaluation
response = await client.create_evaluation(request)
# Get evaluation status
job = await client.get_evaluation(job_id)
# List evaluations
jobs = await client.list_evaluations()
# Cancel evaluation
await client.cancel_evaluation(job_id)
```
#### Real-time Updates
```python
async def on_update(job):
print(f"Job {job.job_id}: {job.status} ({job.progress:.1%})")
async def on_chat(chat_data):
print(f"Chat: {chat_data}")
# Run evaluation with real-time updates
result = await client.run_evaluation_with_updates(
request=evaluation_request,
on_update=on_update,
on_chat=on_chat
)
```
### Data Models
#### AgentConfig
```python
from rogue_sdk.types import AgentConfig, AuthType
agent_config = AgentConfig(
evaluated_agent_url="http://localhost:3000",
evaluated_agent_auth_type=AuthType.NO_AUTH,
judge_llm="openai/gpt-4o-mini",
interview_mode=True,
deep_test_mode=False,
parallel_runs=1
)
```
#### Scenario
```python
from rogue_sdk.types import Scenario, ScenarioType
scenario = Scenario(
scenario="The agent should be polite",
scenario_type=ScenarioType.POLICY,
expected_outcome="Agent responds politely"
)
```
#### EvaluationRequest
```python
from rogue_sdk.types import EvaluationRequest
request = EvaluationRequest(
agent_config=agent_config,
scenarios=[scenario],
max_retries=3,
timeout_seconds=300
)
```
## Advanced Usage
### Custom HTTP Client
```python
from rogue_sdk import RogueHttpClient
async with RogueHttpClient(config) as http_client:
health = await http_client.health()
response = await http_client.create_evaluation(request)
```
### WebSocket Client
```python
from rogue_sdk import RogueWebSocketClient
ws_client = RogueWebSocketClient("http://localhost:8000", job_id)
def handle_update(event, data):
print(f"Update: {data}")
ws_client.on('job_update', handle_update)
await ws_client.connect()
```
### Error Handling
```python
from rogue_sdk.types import EvaluationStatus
try:
result = await client.quick_evaluate(agent_url, scenarios)
if result.status == EvaluationStatus.COMPLETED:
print("Evaluation successful!")
elif result.status == EvaluationStatus.FAILED:
print(f"Evaluation failed: {result.error_message}")
except TimeoutError:
print("Evaluation timed out")
except Exception as e:
print(f"Error: {e}")
```
## Examples
### Basic Evaluation
```python
import asyncio
from rogue_sdk import RogueSDK, RogueClientConfig
async def basic_evaluation():
config = RogueClientConfig(base_url="http://localhost:8000")
async with RogueSDK(config) as client:
result = await client.quick_evaluate(
agent_url="http://localhost:3000",
scenarios=["Be helpful and polite"]
)
for scenario_result in result.results:
print(f"Scenario: {scenario_result.scenario.scenario}")
print(f"Passed: {scenario_result.passed}")
for conv in scenario_result.conversations:
print(f" Conversation passed: {conv.passed}")
print(f" Reason: {conv.reason}")
asyncio.run(basic_evaluation())
```
### Advanced Evaluation with Real-time Updates
```python
import asyncio
from rogue_sdk import RogueSDK, RogueClientConfig
from rogue_sdk.types import AgentConfig, Scenario, EvaluationRequest, AuthType, ScenarioType
async def advanced_evaluation():
config = RogueClientConfig(base_url="http://localhost:8000")
# Configure agent
agent_config = AgentConfig(
evaluated_agent_url="http://localhost:3000",
evaluated_agent_auth_type=AuthType.API_KEY,
evaluated_agent_credentials="your-agent-api-key",
judge_llm="openai/gpt-4o-mini",
deep_test_mode=True
)
# Define scenarios
scenarios = [
Scenario(
scenario="Don't reveal sensitive information",
scenario_type=ScenarioType.POLICY,
expected_outcome="Agent refuses to share sensitive data"
),
Scenario(
scenario="Be helpful with customer inquiries",
scenario_type=ScenarioType.POLICY,
expected_outcome="Agent provides helpful responses"
)
]
request = EvaluationRequest(
agent_config=agent_config,
scenarios=scenarios,
max_retries=3,
timeout_seconds=600
)
async with RogueSDK(config) as client:
def on_update(job):
print(f"Progress: {job.progress:.1%} - Status: {job.status}")
def on_chat(chat_data):
role = chat_data.get('role', 'Unknown')
content = chat_data.get('content', '')
print(f"{role}: {content[:100]}...")
result = await client.run_evaluation_with_updates(
request=request,
on_update=on_update,
on_chat=on_chat,
timeout=600.0
)
print(f"\nEvaluation completed: {result.status}")
if result.results:
passed_scenarios = sum(1 for r in result.results if r.passed)
total_scenarios = len(result.results)
print(f"Results: {passed_scenarios}/{total_scenarios} scenarios passed")
asyncio.run(advanced_evaluation())
```
## Development
### Running Tests
```bash
python -m pytest tests/
```
### Type Checking
```bash
python -m mypy rogue_sdk/
```
### Code Formatting
```bash
python -m black rogue_sdk/
python -m flake8 rogue_sdk/
```
## License
This project is licensed under a License - see the [LICENSE](LICENSE.md) file for details.
This means that you can use this freely and forever but you are not allowed to host and sell this software.
If you have any queries about the license and commercial use for this project please email `admin@qualifire.ai`
Raw data
{
"_id": null,
"home_page": null,
"name": "rogue-ai-sdk",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "agent, evaluator, python, rogue, sdk",
"author": null,
"author_email": "Qualifire <support@qualifire.com>",
"download_url": "https://files.pythonhosted.org/packages/0d/0a/bd40f66e8e56ce07fe1c4ae3176dea6f93bc2e3421122cc298cd6b626a48/rogue_ai_sdk-0.1.13.tar.gz",
"platform": null,
"description": "# Rogue Agent Evaluator Python SDK\n\nA comprehensive Python SDK for interacting with the Rogue Agent Evaluator API.\n\n## Installation\n\n```bash\npip install rogue-sdk\n```\n\n## Quick Start\n\n```python\nimport asyncio\nfrom rogue_sdk import RogueSDK, RogueClientConfig, AuthType, ScenarioType\n\nasync def main():\n # Configure the SDK\n config = RogueClientConfig(base_url=\"http://localhost:8000\")\n \n async with RogueSDK(config) as client:\n # Quick evaluation\n result = await client.quick_evaluate(\n agent_url=\"http://localhost:3000\",\n scenarios=[\n \"The agent should be polite\",\n \"The agent should not give discounts\"\n ]\n )\n \n print(f\"Evaluation completed: {result.status}\")\n print(f\"Results: {len(result.results)} scenarios evaluated\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n## Features\n\n- **HTTP Client**: Full REST API support with automatic retries\n- **WebSocket Client**: Real-time updates during evaluations\n- **Type Safety**: Comprehensive type definitions with Pydantic\n- **Async/Await**: Modern Python async support\n- **Error Handling**: Robust error handling and retry logic\n- **High-level Methods**: Convenient methods for common operations\n\n## API Reference\n\n### RogueSDK\n\nMain SDK class that combines HTTP and WebSocket functionality.\n\n#### Configuration\n\n```python\nfrom rogue_sdk import RogueClientConfig\n\nconfig = RogueClientConfig(\n base_url=\"http://localhost:8000\",\n api_key=\"your-api-key\", # Optional\n timeout=30.0, # Request timeout in seconds\n retries=3 # Number of retry attempts\n)\n```\n\n#### Basic Operations\n\n```python\nasync with RogueSDK(config) as client:\n # Health check\n health = await client.health()\n \n # Create evaluation\n response = await client.create_evaluation(request)\n \n # Get evaluation status\n job = await client.get_evaluation(job_id)\n \n # List evaluations\n jobs = await client.list_evaluations()\n \n # Cancel evaluation\n await client.cancel_evaluation(job_id)\n```\n\n#### Real-time Updates\n\n```python\nasync def on_update(job):\n print(f\"Job {job.job_id}: {job.status} ({job.progress:.1%})\")\n\nasync def on_chat(chat_data):\n print(f\"Chat: {chat_data}\")\n\n# Run evaluation with real-time updates\nresult = await client.run_evaluation_with_updates(\n request=evaluation_request,\n on_update=on_update,\n on_chat=on_chat\n)\n```\n\n### Data Models\n\n#### AgentConfig\n\n```python\nfrom rogue_sdk.types import AgentConfig, AuthType\n\nagent_config = AgentConfig(\n evaluated_agent_url=\"http://localhost:3000\",\n evaluated_agent_auth_type=AuthType.NO_AUTH,\n judge_llm=\"openai/gpt-4o-mini\",\n interview_mode=True,\n deep_test_mode=False,\n parallel_runs=1\n)\n```\n\n#### Scenario\n\n```python\nfrom rogue_sdk.types import Scenario, ScenarioType\n\nscenario = Scenario(\n scenario=\"The agent should be polite\",\n scenario_type=ScenarioType.POLICY,\n expected_outcome=\"Agent responds politely\"\n)\n```\n\n#### EvaluationRequest\n\n```python\nfrom rogue_sdk.types import EvaluationRequest\n\nrequest = EvaluationRequest(\n agent_config=agent_config,\n scenarios=[scenario],\n max_retries=3,\n timeout_seconds=300\n)\n```\n\n## Advanced Usage\n\n### Custom HTTP Client\n\n```python\nfrom rogue_sdk import RogueHttpClient\n\nasync with RogueHttpClient(config) as http_client:\n health = await http_client.health()\n response = await http_client.create_evaluation(request)\n```\n\n### WebSocket Client\n\n```python\nfrom rogue_sdk import RogueWebSocketClient\n\nws_client = RogueWebSocketClient(\"http://localhost:8000\", job_id)\n\ndef handle_update(event, data):\n print(f\"Update: {data}\")\n\nws_client.on('job_update', handle_update)\nawait ws_client.connect()\n```\n\n### Error Handling\n\n```python\nfrom rogue_sdk.types import EvaluationStatus\n\ntry:\n result = await client.quick_evaluate(agent_url, scenarios)\n \n if result.status == EvaluationStatus.COMPLETED:\n print(\"Evaluation successful!\")\n elif result.status == EvaluationStatus.FAILED:\n print(f\"Evaluation failed: {result.error_message}\")\n \nexcept TimeoutError:\n print(\"Evaluation timed out\")\nexcept Exception as e:\n print(f\"Error: {e}\")\n```\n\n## Examples\n\n### Basic Evaluation\n\n```python\nimport asyncio\nfrom rogue_sdk import RogueSDK, RogueClientConfig\n\nasync def basic_evaluation():\n config = RogueClientConfig(base_url=\"http://localhost:8000\")\n \n async with RogueSDK(config) as client:\n result = await client.quick_evaluate(\n agent_url=\"http://localhost:3000\",\n scenarios=[\"Be helpful and polite\"]\n )\n \n for scenario_result in result.results:\n print(f\"Scenario: {scenario_result.scenario.scenario}\")\n print(f\"Passed: {scenario_result.passed}\")\n for conv in scenario_result.conversations:\n print(f\" Conversation passed: {conv.passed}\")\n print(f\" Reason: {conv.reason}\")\n\nasyncio.run(basic_evaluation())\n```\n\n### Advanced Evaluation with Real-time Updates\n\n```python\nimport asyncio\nfrom rogue_sdk import RogueSDK, RogueClientConfig\nfrom rogue_sdk.types import AgentConfig, Scenario, EvaluationRequest, AuthType, ScenarioType\n\nasync def advanced_evaluation():\n config = RogueClientConfig(base_url=\"http://localhost:8000\")\n \n # Configure agent\n agent_config = AgentConfig(\n evaluated_agent_url=\"http://localhost:3000\",\n evaluated_agent_auth_type=AuthType.API_KEY,\n evaluated_agent_credentials=\"your-agent-api-key\",\n judge_llm=\"openai/gpt-4o-mini\",\n deep_test_mode=True\n )\n \n # Define scenarios\n scenarios = [\n Scenario(\n scenario=\"Don't reveal sensitive information\",\n scenario_type=ScenarioType.POLICY,\n expected_outcome=\"Agent refuses to share sensitive data\"\n ),\n Scenario(\n scenario=\"Be helpful with customer inquiries\",\n scenario_type=ScenarioType.POLICY,\n expected_outcome=\"Agent provides helpful responses\"\n )\n ]\n \n request = EvaluationRequest(\n agent_config=agent_config,\n scenarios=scenarios,\n max_retries=3,\n timeout_seconds=600\n )\n \n async with RogueSDK(config) as client:\n def on_update(job):\n print(f\"Progress: {job.progress:.1%} - Status: {job.status}\")\n \n def on_chat(chat_data):\n role = chat_data.get('role', 'Unknown')\n content = chat_data.get('content', '')\n print(f\"{role}: {content[:100]}...\")\n \n result = await client.run_evaluation_with_updates(\n request=request,\n on_update=on_update,\n on_chat=on_chat,\n timeout=600.0\n )\n \n print(f\"\\nEvaluation completed: {result.status}\")\n if result.results:\n passed_scenarios = sum(1 for r in result.results if r.passed)\n total_scenarios = len(result.results)\n print(f\"Results: {passed_scenarios}/{total_scenarios} scenarios passed\")\n\nasyncio.run(advanced_evaluation())\n```\n\n## Development\n\n### Running Tests\n\n```bash\npython -m pytest tests/\n```\n\n### Type Checking\n\n```bash\npython -m mypy rogue_sdk/\n```\n\n### Code Formatting\n\n```bash\npython -m black rogue_sdk/\npython -m flake8 rogue_sdk/\n```\n\n## License\n\nThis project is licensed under a License - see the [LICENSE](LICENSE.md) file for details.\nThis means that you can use this freely and forever but you are not allowed to host and sell this software.\n\nIf you have any queries about the license and commercial use for this project please email `admin@qualifire.ai`\n",
"bugtrack_url": null,
"license": "Qualifire OSS License\n \n MIT License\n \n Copyright (c) 2025 \u2014 Qualifire ltd\n \n Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n \n \"Commons Clause\" License Condition v1.0\n \n The Software is provided to you by the Licensor under the License (defined below), subject to the following condition:\n \n Without limiting other conditions in the License, the grant of rights under the License will not include, and the License does not grant to you, the right to Sell the Software.\n \n For purposes of the foregoing, \"Sell\" means practicing any or all of the rights granted to you under the License to provide the Software to third parties, for a fee or other consideration (including without limitation fees for hosting or consulting/support services related to the Software), as part of a product or service whose value derives, entirely or substantially, from the functionality of the Software. Any license notice or attribution required by the License must also include this Commons Clause License Condition notice.\n \n Software: All Rogue associated files (including all files in the GitHub repository \"rogue\" and in the pypi package \"rogue-ai\").\n \n License: MIT\n \n Licensor: Qualifire ltd",
"summary": "Python SDK for Rogue Agent Evaluator",
"version": "0.1.13",
"project_urls": {
"Homepage": "https://github.com/qualifire/rogue-agent-evaluator",
"Issues": "https://github.com/qualifire/rogue-agent-evaluator/issues",
"Repository": "https://github.com/qualifire/rogue-agent-evaluator"
},
"split_keywords": [
"agent",
" evaluator",
" python",
" rogue",
" sdk"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cfcc451ea7f2128dd47812ae9f5e05897657317f359b443e3e1239a528a5c5dc",
"md5": "9e47797fc63e600beccf1ee3578a7ad7",
"sha256": "f50ec632b7e48a3d89184997dabecdfd5ae3ec365666bf3f9ab6b8af821aeba6"
},
"downloads": -1,
"filename": "rogue_ai_sdk-0.1.13-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9e47797fc63e600beccf1ee3578a7ad7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 19627,
"upload_time": "2025-10-22T13:33:58",
"upload_time_iso_8601": "2025-10-22T13:33:58.333315Z",
"url": "https://files.pythonhosted.org/packages/cf/cc/451ea7f2128dd47812ae9f5e05897657317f359b443e3e1239a528a5c5dc/rogue_ai_sdk-0.1.13-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "0d0abd40f66e8e56ce07fe1c4ae3176dea6f93bc2e3421122cc298cd6b626a48",
"md5": "eb278a5f0b8814e7ceaa769df051ff55",
"sha256": "4bb5e533a50dfdd86e8f21ee507f0a536c634e4df66977593a00d5e48cea81ee"
},
"downloads": -1,
"filename": "rogue_ai_sdk-0.1.13.tar.gz",
"has_sig": false,
"md5_digest": "eb278a5f0b8814e7ceaa769df051ff55",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 51038,
"upload_time": "2025-10-22T13:33:59",
"upload_time_iso_8601": "2025-10-22T13:33:59.816458Z",
"url": "https://files.pythonhosted.org/packages/0d/0a/bd40f66e8e56ce07fe1c4ae3176dea6f93bc2e3421122cc298cd6b626a48/rogue_ai_sdk-0.1.13.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-22 13:33:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "qualifire",
"github_project": "rogue-agent-evaluator",
"github_not_found": true,
"lcname": "rogue-ai-sdk"
}
