<div align="center">
<h1>🧪 MCP Testing</h1>
<p><strong>AI-powered testing framework for MCP servers</strong></p>
<p>Test your MCP servers with real AI agents conducting conversations and LLM judges evaluating results</p>
</div>
<div align="center">
<a href="https://docs.golf.dev/mcp-testing/getting-started/quickstart"><img src="https://img.shields.io/badge/docs-golf.dev-blue.svg" alt="Documentation"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"></a>
<a href="https://github.com/golf-mcp/golf-testing/pulls"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs Welcome"></a>
<a href="https://pypi.org/project/mcp-testing/"><img src="https://img.shields.io/pypi/v/mcp-testing" alt="PyPI"></a>
<a href="#"><img src="https://img.shields.io/badge/python-3.12+-blue.svg" alt="Python"></a>
</div>
## Why MCP Testing?
Traditional testing doesn't work for MCP servers. You can't write unit tests for natural language interactions. MCP Testing solves this with:
- **🤖 Real AI Agents** - Claude and ChatGPT actually use your MCP server
- **👤 User Simulation** - AI simulates realistic multi-turn user behavior
- **⚖️ LLM-as-a-Judge** - Intelligent evaluation instead of brittle assertions
- **🎭 Comprehensive Testing** - Security, compliance and performace all in one framework
- **🔌 Multiple Transports** - Supports HTTP and stdio servers
## Quick Start
Get testing in 3 steps:
1. **Install & Setup**
```bash
pip install mcp-testing
export ANTHROPIC_API_KEY="sk-ant-..." # For AI agents
export OPENAI_API_KEY="sk-..." # For LLM judge
```
2. **Interactive Onboarding**
```bash
mcp-t quickstart # Creates your first server & test suite
```
3. **Run Tests**
```bash
mcp-t run <suite-id> <server-id>
# Example: mcp-t run example_suite_001 hackernews_mcp_server
```
## Core Concepts
### Test Flow
```
Your Test Case → AI Agent (Claude/GPT-4) → Your MCP Server
↓ ↓ ↓
User Message Tool Calls Server Response
↓ ↓ ↓
User Simulator Conversation Loop More Tools
↓ ↓ ↓
LLM Judge Complete Transcript Pass/Fail + Reasoning
```
### Configuration Files
**Server Config - HTTP** (`examples/server.json`):
```json
{
"name": "linear_mcp_server",
"transport": "http",
"url": "https://mcp.linear.app/mcp"
}
```
**Server Config - stdio** (`examples/servers/time-server-stdio.json`):
```json
{
"name": "Time Server",
"transport": "stdio",
"command": "npx -y @modelcontextprotocol/server-time"
}
```
**Server Config - stdio with env** (`examples/servers/brave-search-stdio.json`):
```json
{
"name": "Brave Search",
"transport": "stdio",
"command": "npx -y @modelcontextprotocol/server-brave-search",
"env": {
"BRAVE_API_KEY": "your-api-key-here"
}
}
```
**Test Suite** (`examples/suite.json`):
```json
{
"suite_id": "example_suite_001",
"name": "Hacker News MCP Server Tests",
"test_cases": [
{
"test_id": "hackernews_greeting",
"user_message": "Hello! Can you help me browse Hacker News?",
"success_criteria": "Agent should respond politely and explain Hacker News capabilities",
"max_turns": 5
}
]
}
```
### Test Types
- **💬 Conversational** - Real user workflows
- **🔒 Security** - Authentication & vulnerabilities
- **✅ Compliance** - MCP protocol validation
## Commands
### Test Execution
```bash
mcp-t run <suite-id> <server-id> # Run specific suite
mcp-t run example_suite_001 hackernews_mcp_server -v # Verbose output
```
### Configuration Management
```bash
mcp-t quickstart # Complete onboarding
mcp-t create server # Interactive server setup
mcp-t create suite # Create test suite
mcp-t create test-case # Add test to suite
mcp-t list # Show all configs
mcp-t show suite example_suite_001 # View specific config
```
### Test Generation
Run wizard that analyzes your MCP server and automatically generates comprehensive test cases
```bash
mcp-t generate
```
## Test Results
### Understanding Evaluation
```json
{
"test_id": "hackernews_stories",
"verdict": "PASS",
"confidence_score": 0.89,
"judge_reasoning": "The agent successfully fetched and displayed Hacker News stories. Good use of available tools and clear presentation of results.",
"conversation_quality": 0.87,
"tool_calls": [
{ "tool": "get_top_stories", "args": {} },
{ "tool": "get_story_details", "args": { "story_id": 123 } }
]
}
```
## Support
- [Documentation](https://docs.golf.dev/mcp-testing/getting-started/quickstart)
- [Contributing Guide](CONTRIBUTING.md)
---
<div align="center">
<p>Built with ❤️ for the MCP ecosystem</p>
<p><sub>Made in San Francisco, CA</sub></p>
</div>
Raw data
{
"_id": null,
"home_page": null,
"name": "mcp-testing",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "mcp, testing, ai, claude, protocol, cli, local",
"author": null,
"author_email": "Antoni Gmitruk <antoni@golf.dev>",
"download_url": "https://files.pythonhosted.org/packages/8b/d6/c3a7419f0834b1405761de1c1c4106ba67851171df04c4f80da5f51ae6cb/mcp_testing-0.1.0b4.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <h1>\ud83e\uddea MCP Testing</h1>\n <p><strong>AI-powered testing framework for MCP servers</strong></p>\n <p>Test your MCP servers with real AI agents conducting conversations and LLM judges evaluating results</p>\n</div>\n\n<div align=\"center\">\n <a href=\"https://docs.golf.dev/mcp-testing/getting-started/quickstart\"><img src=\"https://img.shields.io/badge/docs-golf.dev-blue.svg\" alt=\"Documentation\"></a>\n <a href=\"LICENSE\"><img src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\" alt=\"License\"></a>\n <a href=\"https://github.com/golf-mcp/golf-testing/pulls\"><img src=\"https://img.shields.io/badge/PRs-welcome-brightgreen.svg\" alt=\"PRs Welcome\"></a>\n <a href=\"https://pypi.org/project/mcp-testing/\"><img src=\"https://img.shields.io/pypi/v/mcp-testing\" alt=\"PyPI\"></a>\n <a href=\"#\"><img src=\"https://img.shields.io/badge/python-3.12+-blue.svg\" alt=\"Python\"></a>\n</div>\n\n## Why MCP Testing?\n\nTraditional testing doesn't work for MCP servers. You can't write unit tests for natural language interactions. MCP Testing solves this with:\n\n- **\ud83e\udd16 Real AI Agents** - Claude and ChatGPT actually use your MCP server\n- **\ud83d\udc64 User Simulation** - AI simulates realistic multi-turn user behavior\n- **\u2696\ufe0f LLM-as-a-Judge** - Intelligent evaluation instead of brittle assertions\n- **\ud83c\udfad Comprehensive Testing** - Security, compliance and performace all in one framework\n- **\ud83d\udd0c Multiple Transports** - Supports HTTP and stdio servers\n\n## Quick Start\n\nGet testing in 3 steps:\n\n1. **Install & Setup**\n\n ```bash\n pip install mcp-testing\n export ANTHROPIC_API_KEY=\"sk-ant-...\" # For AI agents\n export OPENAI_API_KEY=\"sk-...\" # For LLM judge\n ```\n\n2. **Interactive Onboarding**\n\n ```bash\n mcp-t quickstart # Creates your first server & test suite\n ```\n\n3. **Run Tests**\n ```bash\n mcp-t run <suite-id> <server-id>\n # Example: mcp-t run example_suite_001 hackernews_mcp_server\n ```\n\n## Core Concepts\n\n### Test Flow\n\n```\nYour Test Case \u2192 AI Agent (Claude/GPT-4) \u2192 Your MCP Server\n \u2193 \u2193 \u2193\n User Message Tool Calls Server Response\n \u2193 \u2193 \u2193\nUser Simulator Conversation Loop More Tools\n \u2193 \u2193 \u2193\n LLM Judge Complete Transcript Pass/Fail + Reasoning\n```\n\n### Configuration Files\n\n**Server Config - HTTP** (`examples/server.json`):\n\n```json\n{\n \"name\": \"linear_mcp_server\",\n \"transport\": \"http\",\n \"url\": \"https://mcp.linear.app/mcp\"\n}\n```\n\n**Server Config - stdio** (`examples/servers/time-server-stdio.json`):\n\n```json\n{\n \"name\": \"Time Server\",\n \"transport\": \"stdio\",\n \"command\": \"npx -y @modelcontextprotocol/server-time\"\n}\n```\n\n**Server Config - stdio with env** (`examples/servers/brave-search-stdio.json`):\n\n```json\n{\n \"name\": \"Brave Search\",\n \"transport\": \"stdio\",\n \"command\": \"npx -y @modelcontextprotocol/server-brave-search\",\n \"env\": {\n \"BRAVE_API_KEY\": \"your-api-key-here\"\n }\n}\n```\n\n**Test Suite** (`examples/suite.json`):\n\n```json\n{\n \"suite_id\": \"example_suite_001\",\n \"name\": \"Hacker News MCP Server Tests\",\n \"test_cases\": [\n {\n \"test_id\": \"hackernews_greeting\",\n \"user_message\": \"Hello! Can you help me browse Hacker News?\",\n \"success_criteria\": \"Agent should respond politely and explain Hacker News capabilities\",\n \"max_turns\": 5\n }\n ]\n}\n```\n\n### Test Types\n\n- **\ud83d\udcac Conversational** - Real user workflows\n- **\ud83d\udd12 Security** - Authentication & vulnerabilities\n- **\u2705 Compliance** - MCP protocol validation\n\n## Commands\n\n### Test Execution\n\n```bash\nmcp-t run <suite-id> <server-id> # Run specific suite\nmcp-t run example_suite_001 hackernews_mcp_server -v # Verbose output\n```\n\n### Configuration Management\n\n```bash\nmcp-t quickstart # Complete onboarding\nmcp-t create server # Interactive server setup\nmcp-t create suite # Create test suite\nmcp-t create test-case # Add test to suite\nmcp-t list # Show all configs\nmcp-t show suite example_suite_001 # View specific config\n```\n\n### Test Generation\n\nRun wizard that analyzes your MCP server and automatically generates comprehensive test cases\n\n```bash\nmcp-t generate\n```\n\n## Test Results\n\n### Understanding Evaluation\n\n```json\n{\n \"test_id\": \"hackernews_stories\",\n \"verdict\": \"PASS\",\n \"confidence_score\": 0.89,\n \"judge_reasoning\": \"The agent successfully fetched and displayed Hacker News stories. Good use of available tools and clear presentation of results.\",\n \"conversation_quality\": 0.87,\n \"tool_calls\": [\n { \"tool\": \"get_top_stories\", \"args\": {} },\n { \"tool\": \"get_story_details\", \"args\": { \"story_id\": 123 } }\n ]\n}\n```\n\n## Support\n\n- [Documentation](https://docs.golf.dev/mcp-testing/getting-started/quickstart)\n- [Contributing Guide](CONTRIBUTING.md)\n\n---\n\n<div align=\"center\">\n <p>Built with \u2764\ufe0f for the MCP ecosystem</p>\n <p><sub>Made in San Francisco, CA</sub></p>\n</div>\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Local testing framework for MCP (Model Context Protocol) servers",
"version": "0.1.0b4",
"project_urls": {
"Bug Tracker": "https://github.com/golf-mcp/golf-testing/issues",
"Documentation": "https://github.com/golf-mcp/golf-testing#readme",
"Homepage": "https://github.com/golf-mcp/golf-testing",
"Repository": "https://github.com/golf-mcp/golf-testing"
},
"split_keywords": [
"mcp",
" testing",
" ai",
" claude",
" protocol",
" cli",
" local"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "6cfd9cf354672726d1c9df6835dfd67bdda76a7787442f932279a7f2b752abc6",
"md5": "cf8fe15a6c23d4d943a62b5ae94dfe0e",
"sha256": "69ee39861d706eb4817fc28a66fa0c927bff08fd570e237620d19e537e262e7e"
},
"downloads": -1,
"filename": "mcp_testing-0.1.0b4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cf8fe15a6c23d4d943a62b5ae94dfe0e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 173044,
"upload_time": "2025-10-13T16:33:24",
"upload_time_iso_8601": "2025-10-13T16:33:24.507712Z",
"url": "https://files.pythonhosted.org/packages/6c/fd/9cf354672726d1c9df6835dfd67bdda76a7787442f932279a7f2b752abc6/mcp_testing-0.1.0b4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8bd6c3a7419f0834b1405761de1c1c4106ba67851171df04c4f80da5f51ae6cb",
"md5": "2338d226856058c3a3783236792baa01",
"sha256": "8fa48b624917222c7d87175e4b71d50566f907df8a6b6a4a5a971c12260f2ecb"
},
"downloads": -1,
"filename": "mcp_testing-0.1.0b4.tar.gz",
"has_sig": false,
"md5_digest": "2338d226856058c3a3783236792baa01",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 148268,
"upload_time": "2025-10-13T16:33:26",
"upload_time_iso_8601": "2025-10-13T16:33:26.199084Z",
"url": "https://files.pythonhosted.org/packages/8b/d6/c3a7419f0834b1405761de1c1c4106ba67851171df04c4f80da5f51ae6cb/mcp_testing-0.1.0b4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-13 16:33:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "golf-mcp",
"github_project": "golf-testing",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "anthropic",
"specs": [
[
">=",
"0.65.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "httpx",
"specs": [
[
">=",
"0.25.0"
]
]
},
{
"name": "openai",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.0.0"
]
]
},
{
"name": "rich",
"specs": [
[
">=",
"13.0.0"
]
]
},
{
"name": "PyYAML",
"specs": [
[
">=",
"6.0.0"
]
]
},
{
"name": "mcp",
"specs": [
[
">=",
"1.15.0"
]
]
},
{
"name": "PyJWT",
"specs": [
[
">=",
"2.10.1"
]
]
},
{
"name": "packaging",
"specs": [
[
">=",
"21.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
">=",
"4.12.0"
]
]
}
],
"lcname": "mcp-testing"
}
JSON
