| Name | sibylline-skald JSON |
| Version |
0.2.0
JSON |
| download |
| home_page | None |
| Summary | Universal Execution Monitoring & Feedback Library for Python applications, supporting MCP servers, functions, shell commands, and more |
| upload_time | 2025-09-02 05:07:11 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT |
| keywords |
agent
ai
feedback
mcp
tools
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Skald - MCP Feedback Adapter
[](https://badge.fury.io/py/skald)
[](https://pypi.org/project/skald/)
[](https://github.com/sibyllinesoft/skald/actions)
[](https://codecov.io/gh/sibyllinesoft/skald)
[](https://github.com/psf/black)
[](https://github.com/astral-sh/ruff)
> *"A storyteller for your tools' performance, letting them speak back through AI agents when they shine or stumble."*
Skald is an in-process adapter library for Python MCP servers that enables structured feedback collection on tool usefulness, clarity, and fit from AI agents.
## Features
- **Transparent Tool Wrapping**: Drop-in wrapper that intercepts MCP tool calls and adds trace IDs with metrics collection
- **Structured Feedback Collection**: New `feedback.report` tool with strict JSON schema validation
- **Smart Feedback Invitations**: Automatically invites feedback based on errors, latency, and output size
- **Opt-out Controls**: Decorator and context manager for suppressing feedback collection
- **Multiple Storage Backends**: SQLite default with optional Parquet export and data redaction
- **Multiple Transports**: Support for stdio, TCP, and Unix socket transports
## Installation
```bash
pip install skald
```
## Quick Start
### Basic Usage
```python
import skald
from your_mcp_server import YourMCPServer
# Wrap your existing MCP server
upstream = YourMCPServer()
survey = skald.SurveyingProxy(
upstream=upstream,
store="sqlite:///feedback.db",
invite_policy={"error": True, "p95_ms": 5000, "large_output_kb": 256},
sample_neutral=0.10,
ttl_hours=24
)
# Your server now collects feedback transparently
```
### Using Opt-out Controls
```python
# Opt out specific functions
@survey.opt_out(reason="contains sensitive data")
def run_sensitive_query(sql: str) -> dict:
return execute_query(sql)
# Suppress collection for a block of code
with survey.suppressed(reason="benchmark"):
for _ in range(1000):
call_some_tool()
```
### Running a Server
```python
from skald.transport import TCPTransport
# Create transport and serve
transport = TCPTransport(survey)
await transport.serve(host="0.0.0.0", port=8765)
```
## CLI Usage
Skald includes a CLI for running demo servers and querying feedback data:
```bash
# Run over TCP
skald tcp --host localhost --port 8765
# Run over stdio
skald stdio
# Query stored feedback
skald query --store sqlite:///feedback.db --limit 10
# Clean up expired data
skald cleanup --store sqlite:///feedback.db --ttl-hours 24
```
## Agent Integration
To collect feedback from AI agents, add this line to your system prompt:
```
When a tool provides a trace_id in its response metadata and invite_feedback=true, you may optionally use feedback.report to provide structured feedback on the tool's helpfulness (1-5), fit (1-5), clarity (1-5), confidence (0.0-1.0), and up to 3 short suggestions for improvement.
```
## Feedback Schema
The `feedback.report` tool accepts this JSON schema:
```json
{
"trace_id": "550e8400-e29b-41d4-a716-446655440000",
"helpfulness": 4,
"fit": 3,
"clarity": 5,
"confidence": 0.8,
"better_alternative": "different_tool",
"suggestions": ["Try tool X", "Use parameter Y"],
"notes": "This worked well overall"
}
```
### Fields
- `trace_id` (required): UUID from the tool response metadata
- `helpfulness` (required): How helpful was the tool (1-5)
- `fit` (required): How well did the tool fit the task (1-5)
- `clarity` (required): How clear was the tool output (1-5)
- `confidence` (required): Confidence in this feedback (0.0-1.0)
- `better_alternative` (optional): Enum suggesting better alternatives
- `suggestions` (optional): Up to 3 suggestions, each ≤100 characters
- `notes` (optional): Additional notes
## Configuration
### Invite Policy
Control when feedback is invited:
```python
invite_policy = {
"error": True, # Invite on errors
"timeout": True, # Invite on timeouts
"p95_ms": 5000.0, # Latency threshold (ms)
"large_output_kb": 256.0 # Output size threshold (KB)
}
```
### Storage Configuration
```python
# SQLite (default)
store = "sqlite:///path/to/feedback.db"
# Custom storage backend
from skald.storage.sqlite import SQLiteStorage
storage = SQLiteStorage("/custom/path.db")
survey = SurveyingProxy(upstream, store=storage)
```
### Data Redaction
Configure custom data redaction:
```python
def custom_redactor(args: dict) -> dict:
"""Remove sensitive data from tool arguments."""
redacted = args.copy()
if 'password' in redacted:
redacted['password'] = '[REDACTED]'
return redacted
survey = SurveyingProxy(
upstream=upstream,
redactor=custom_redactor
)
```
## Storage Schema
Skald uses two main tables:
### tool_runs
- `trace_id` (PK): UUID trace identifier
- `timestamp`: Execution timestamp
- `agent_id`: ID of the calling agent
- `tool_name`: Name of the tool called
- `status`: success/error/timeout
- `latency_ms`: Execution latency in milliseconds
- `output_bytes`: Size of output in bytes
- `invite_feedback`: Whether feedback was invited
- `opt_out`: Whether collection was opted out
- `args_redacted`: Redacted tool arguments (JSON)
### tool_feedback
- `trace_id` (FK): Reference to tool_runs
- `agent_id`: ID of the agent providing feedback
- `helpfulness`: Helpfulness rating (1-5)
- `fit`: Fit rating (1-5)
- `clarity`: Clarity rating (1-5)
- `confidence`: Confidence score (0.0-1.0)
- `better_alternative`: Better alternative suggestion
- `suggestions`: List of suggestions (JSON)
- `notes`: Additional notes
- `valid`: Whether feedback passed validation
- `raw_json`: Raw feedback JSON
- `timestamp`: Feedback timestamp
## Architecture
Skald follows a clean architecture with these components:
- **Core**: `SurveyingProxy` - Main proxy class that wraps MCP servers
- **Schema**: Pydantic models for data validation
- **Storage**: Pluggable storage backends (SQLite default)
- **Transport**: Communication protocols (stdio, TCP, Unix sockets)
- **Decorators**: Opt-out mechanisms (`@opt_out`, `with suppressed()`)
## Development
### Setup
```bash
git clone https://github.com/sibyllinesoft/skald.git
cd skald
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev,test]"
```
### Testing
```bash
pytest
pytest --cov=skald --cov-report=html
```
### Code Quality
```bash
black skald tests
isort skald tests
mypy skald
ruff skald tests
```
## License
MIT License - see LICENSE file for details.
## Contributing
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Raw data
{
"_id": null,
"home_page": null,
"name": "sibylline-skald",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "agent, ai, feedback, mcp, tools",
"author": null,
"author_email": "Nathan Rice <nathan.alexander.rice@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/00/c7/1a33195a4f97afe5b5e2e6f42b3aadffac73b45b1600f7e3cce48aeb65c0/sibylline_skald-0.2.0.tar.gz",
"platform": null,
"description": "# Skald - MCP Feedback Adapter\n\n[](https://badge.fury.io/py/skald)\n[](https://pypi.org/project/skald/)\n[](https://github.com/sibyllinesoft/skald/actions)\n[](https://codecov.io/gh/sibyllinesoft/skald)\n[](https://github.com/psf/black)\n[](https://github.com/astral-sh/ruff)\n\n> *\"A storyteller for your tools' performance, letting them speak back through AI agents when they shine or stumble.\"*\n\nSkald is an in-process adapter library for Python MCP servers that enables structured feedback collection on tool usefulness, clarity, and fit from AI agents.\n\n## Features\n\n- **Transparent Tool Wrapping**: Drop-in wrapper that intercepts MCP tool calls and adds trace IDs with metrics collection\n- **Structured Feedback Collection**: New `feedback.report` tool with strict JSON schema validation\n- **Smart Feedback Invitations**: Automatically invites feedback based on errors, latency, and output size\n- **Opt-out Controls**: Decorator and context manager for suppressing feedback collection\n- **Multiple Storage Backends**: SQLite default with optional Parquet export and data redaction\n- **Multiple Transports**: Support for stdio, TCP, and Unix socket transports\n\n## Installation\n\n```bash\npip install skald\n```\n\n## Quick Start\n\n### Basic Usage\n\n```python\nimport skald\nfrom your_mcp_server import YourMCPServer\n\n# Wrap your existing MCP server\nupstream = YourMCPServer()\nsurvey = skald.SurveyingProxy(\n upstream=upstream,\n store=\"sqlite:///feedback.db\",\n invite_policy={\"error\": True, \"p95_ms\": 5000, \"large_output_kb\": 256},\n sample_neutral=0.10,\n ttl_hours=24\n)\n\n# Your server now collects feedback transparently\n```\n\n### Using Opt-out Controls\n\n```python\n# Opt out specific functions\n@survey.opt_out(reason=\"contains sensitive data\")\ndef run_sensitive_query(sql: str) -> dict:\n return execute_query(sql)\n\n# Suppress collection for a block of code\nwith survey.suppressed(reason=\"benchmark\"):\n for _ in range(1000):\n call_some_tool()\n```\n\n### Running a Server\n\n```python\nfrom skald.transport import TCPTransport\n\n# Create transport and serve\ntransport = TCPTransport(survey)\nawait transport.serve(host=\"0.0.0.0\", port=8765)\n```\n\n## CLI Usage\n\nSkald includes a CLI for running demo servers and querying feedback data:\n\n```bash\n# Run over TCP\nskald tcp --host localhost --port 8765\n\n# Run over stdio \nskald stdio\n\n# Query stored feedback\nskald query --store sqlite:///feedback.db --limit 10\n\n# Clean up expired data\nskald cleanup --store sqlite:///feedback.db --ttl-hours 24\n```\n\n## Agent Integration\n\nTo collect feedback from AI agents, add this line to your system prompt:\n\n```\nWhen a tool provides a trace_id in its response metadata and invite_feedback=true, you may optionally use feedback.report to provide structured feedback on the tool's helpfulness (1-5), fit (1-5), clarity (1-5), confidence (0.0-1.0), and up to 3 short suggestions for improvement.\n```\n\n## Feedback Schema\n\nThe `feedback.report` tool accepts this JSON schema:\n\n```json\n{\n \"trace_id\": \"550e8400-e29b-41d4-a716-446655440000\",\n \"helpfulness\": 4,\n \"fit\": 3, \n \"clarity\": 5,\n \"confidence\": 0.8,\n \"better_alternative\": \"different_tool\",\n \"suggestions\": [\"Try tool X\", \"Use parameter Y\"],\n \"notes\": \"This worked well overall\"\n}\n```\n\n### Fields\n\n- `trace_id` (required): UUID from the tool response metadata\n- `helpfulness` (required): How helpful was the tool (1-5)\n- `fit` (required): How well did the tool fit the task (1-5) \n- `clarity` (required): How clear was the tool output (1-5)\n- `confidence` (required): Confidence in this feedback (0.0-1.0)\n- `better_alternative` (optional): Enum suggesting better alternatives\n- `suggestions` (optional): Up to 3 suggestions, each \u2264100 characters\n- `notes` (optional): Additional notes\n\n## Configuration\n\n### Invite Policy\n\nControl when feedback is invited:\n\n```python\ninvite_policy = {\n \"error\": True, # Invite on errors\n \"timeout\": True, # Invite on timeouts \n \"p95_ms\": 5000.0, # Latency threshold (ms)\n \"large_output_kb\": 256.0 # Output size threshold (KB)\n}\n```\n\n### Storage Configuration\n\n```python\n# SQLite (default)\nstore = \"sqlite:///path/to/feedback.db\"\n\n# Custom storage backend\nfrom skald.storage.sqlite import SQLiteStorage\nstorage = SQLiteStorage(\"/custom/path.db\")\nsurvey = SurveyingProxy(upstream, store=storage)\n```\n\n### Data Redaction\n\nConfigure custom data redaction:\n\n```python\ndef custom_redactor(args: dict) -> dict:\n \"\"\"Remove sensitive data from tool arguments.\"\"\"\n redacted = args.copy()\n if 'password' in redacted:\n redacted['password'] = '[REDACTED]'\n return redacted\n\nsurvey = SurveyingProxy(\n upstream=upstream,\n redactor=custom_redactor\n)\n```\n\n## Storage Schema\n\nSkald uses two main tables:\n\n### tool_runs\n- `trace_id` (PK): UUID trace identifier\n- `timestamp`: Execution timestamp\n- `agent_id`: ID of the calling agent\n- `tool_name`: Name of the tool called\n- `status`: success/error/timeout\n- `latency_ms`: Execution latency in milliseconds\n- `output_bytes`: Size of output in bytes\n- `invite_feedback`: Whether feedback was invited\n- `opt_out`: Whether collection was opted out\n- `args_redacted`: Redacted tool arguments (JSON)\n\n### tool_feedback\n- `trace_id` (FK): Reference to tool_runs\n- `agent_id`: ID of the agent providing feedback\n- `helpfulness`: Helpfulness rating (1-5)\n- `fit`: Fit rating (1-5)\n- `clarity`: Clarity rating (1-5)\n- `confidence`: Confidence score (0.0-1.0)\n- `better_alternative`: Better alternative suggestion\n- `suggestions`: List of suggestions (JSON)\n- `notes`: Additional notes\n- `valid`: Whether feedback passed validation\n- `raw_json`: Raw feedback JSON\n- `timestamp`: Feedback timestamp\n\n## Architecture\n\nSkald follows a clean architecture with these components:\n\n- **Core**: `SurveyingProxy` - Main proxy class that wraps MCP servers\n- **Schema**: Pydantic models for data validation\n- **Storage**: Pluggable storage backends (SQLite default)\n- **Transport**: Communication protocols (stdio, TCP, Unix sockets)\n- **Decorators**: Opt-out mechanisms (`@opt_out`, `with suppressed()`)\n\n## Development\n\n### Setup\n\n```bash\ngit clone https://github.com/sibyllinesoft/skald.git\ncd skald\nuv venv\nsource .venv/bin/activate # On Windows: .venv\\Scripts\\activate\nuv pip install -e \".[dev,test]\"\n```\n\n### Testing\n\n```bash\npytest\npytest --cov=skald --cov-report=html\n```\n\n### Code Quality\n\n```bash\nblack skald tests\nisort skald tests \nmypy skald\nruff skald tests\n```\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Contributing\n\nContributions are welcome! Please see CONTRIBUTING.md for guidelines.",
"bugtrack_url": null,
"license": "MIT",
"summary": "Universal Execution Monitoring & Feedback Library for Python applications, supporting MCP servers, functions, shell commands, and more",
"version": "0.2.0",
"project_urls": {
"Documentation": "https://github.com/sibyllinesoft/skald#readme",
"Homepage": "https://github.com/sibyllinesoft/skald",
"Issues": "https://github.com/sibyllinesoft/skald/issues",
"Repository": "https://github.com/sibyllinesoft/skald.git"
},
"split_keywords": [
"agent",
" ai",
" feedback",
" mcp",
" tools"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "aa18be53dcf807ccd51244b39a9b2e0ecbcfa05501e302d274c84de4ef02ea1e",
"md5": "f8902d52af4dd7646481a2d11748de80",
"sha256": "eca2fb12d7224fe4c95ca03b16db6657cb9511fe6f2572b6dc3f7783e286734d"
},
"downloads": -1,
"filename": "sibylline_skald-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f8902d52af4dd7646481a2d11748de80",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 55806,
"upload_time": "2025-09-02T05:07:10",
"upload_time_iso_8601": "2025-09-02T05:07:10.247280Z",
"url": "https://files.pythonhosted.org/packages/aa/18/be53dcf807ccd51244b39a9b2e0ecbcfa05501e302d274c84de4ef02ea1e/sibylline_skald-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "00c71a33195a4f97afe5b5e2e6f42b3aadffac73b45b1600f7e3cce48aeb65c0",
"md5": "da4867b93a42709eb090250babde0f45",
"sha256": "e48c38af899cd11f8abe34bed1d6f6cf17179dfc26417f1f50696715ebc95e41"
},
"downloads": -1,
"filename": "sibylline_skald-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "da4867b93a42709eb090250babde0f45",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 50090,
"upload_time": "2025-09-02T05:07:11",
"upload_time_iso_8601": "2025-09-02T05:07:11.203280Z",
"url": "https://files.pythonhosted.org/packages/00/c7/1a33195a4f97afe5b5e2e6f42b3aadffac73b45b1600f7e3cce48aeb65c0/sibylline_skald-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-02 05:07:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sibyllinesoft",
"github_project": "skald#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "sibylline-skald"
}
