# ⚖️ MCP as a Judge
> **Prevent bad coding practices with AI-powered evaluation and user-driven decision making**
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
[](https://modelcontextprotocol.io/)
[](https://github.com/hepivax/mcp-as-a-judge/actions/workflows/ci.yml)
[](https://github.com/hepivax/mcp-as-a-judge/actions/workflows/release.yml)
[](https://pypi.org/project/mcp-as-a-judge/)
**MCP as a Judge** is a revolutionary Model Context Protocol (MCP) server that **transforms the developer-AI collaboration experience**. It acts as an intelligent gatekeeper for software development, preventing bad coding practices by using AI-powered evaluation and involving users in critical decisions when requirements are unclear or obstacles arise.
> **⚖️ Concept**: This project extends the **LLM-as-a-Judge** paradigm to software engineering workflows, where AI models evaluate and guide development decisions rather than just generating code.
## ⚖️ **Main Purpose: Improve Developer-AI Interface**
**The core mission is to enhance the interface between developers and AI coding assistants** by:
- 🛡️ **Preventing AI from making poor decisions** through intelligent evaluation
- 🤝 **Involving humans in critical choices** instead of AI making assumptions
- 🔍 **Enforcing research and best practices** before implementation
- ⚖️ **Creating a collaborative AI-human workflow** for better software quality
## **Vibe Coding doesn't have to be frustrating**
### **What It Enforces:**
- ✅ **Deep research** of existing solutions and best practices
- ✅ **Generic, reusable solutions** instead of quick fixes
- ✅ **User requirements alignment** in all implementations
- ✅ **Comprehensive planning** before coding begins
- ✅ **User involvement** in all critical decisions
- ✅ **Intelligent AI-human collaboration** with clear boundaries and responsibilities
## 🛠️ **Features**
### **🔍 Intelligent Code Evaluation**
- **LLM-powered analysis** using MCP [sampling](https://modelcontextprotocol.io/docs/learn/client-concepts#sampling) capability
- **Software engineering best practices** enforcement
- **Security vulnerability detection**
- **Performance and maintainability assessment**
### **📋 Comprehensive Planning Review**
- **Architecture validation** against industry standards
- **Research depth analysis** to prevent reinventing solutions
- **Requirements alignment** verification
- **Implementation approach evaluation**
### **🤝 User-Driven Decision Making**
- **Obstacle resolution** through user involvement via MCP [elicitation](https://modelcontextprotocol.io/docs/learn/client-concepts#elicitation)
- **Requirements clarification** when requests are unclear
- **No hidden fallbacks** - transparent decision making
- **Interactive problem solving** with real-time user input
### **🛠️ List of Tools**
| Tool Name | Description |
|-----------|-------------|
| **`get_workflow_guidance`** | Smart workflow analysis and tool recommendation |
| **`judge_coding_plan`** | Comprehensive plan evaluation with requirements alignment |
| **`judge_code_change`** | Code review with security and quality checks |
| **`raise_obstacle`** | User involvement when blockers arise |
| **`elicit_missing_requirements`** | Clarification of unclear requests |
## 🚀 **Quick Start**
### **Requirements & Recommendations**
#### **MCP Client Prerequisites**
MCP as a Judge is heavily dependent on **MCP Sampling** and **MCP Elicitation** features for its core functionality:
- **[MCP Sampling](https://modelcontextprotocol.io/docs/learn/client-concepts#sampling)** - Required for AI-powered code evaluation and judgment
- **[MCP Elicitation](https://modelcontextprotocol.io/docs/learn/client-concepts#elicitation)** - Required for interactive user decision prompts
#### **System Prerequisites**
- **Python 3.13+** - Required for running the MCP server
#### **Supported AI Assistants**
| AI Assistant | Platform | MCP Support | Status | Notes |
|---------------|----------|-------------|---------|-------|
| **GitHub Copilot** | Visual Studio Code | ✅ Full | **Recommended** | Complete MCP integration with tool calling |
**✅ Recommended Setup:** GitHub Copilot in Visual Studio Code for the best MCP as a Judge experience.
#### **💡 Recommendations**
- **Large Context Window Models**: 1M+ token size models are strongly preferred for optimal performance
- Models with larger context windows provide better code analysis and more comprehensive judgments
> **Note**: MCP servers communicate via stdio (standard input/output), not HTTP ports. No network configuration is needed.
## 🔧 **Visual Studio Code Configuration**
Configure MCP as a Judge in Visual Studio Code with GitHub Copilot:
### **Method 1: Using uv (Recommended)**
1. **Install the package:**
```bash
uv tool install mcp-as-a-judge
```
2. **Configure Visual Studio Code MCP settings:**
Add this to your Visual Studio Code MCP configuration file:
```json
{
"servers": {
"mcp-as-a-judge": {
"command": "uv",
"args": ["tool", "run", "mcp-as-a-judge"]
}
}
}
```
### **Method 2: Using Docker**
1. **Pull the Docker image:**
```bash
docker pull ghcr.io/hepivax/mcp-as-a-judge:latest
```
2. **Configure Visual Studio Code MCP settings:**
Add this to your Visual Studio Code MCP configuration file:
```json
{
"servers": {
"mcp-as-a-judge": {
"command": "docker",
"args": ["run", "--rm", "-i", "ghcr.io/hepivax/mcp-as-a-judge:latest"]
}
}
}
```
## 📖 **How It Works**
Once MCP as a Judge is configured in Visual Studio Code with GitHub Copilot, it automatically guides your AI assistant through a structured software engineering workflow. The system operates transparently in the background, ensuring every development task follows best practices.
### **🔄 Automatic Workflow Enforcement**
**1. Intelligent Workflow Guidance**
- When you make any development request, the AI assistant automatically calls `get_workflow_guidance`
- This tool uses AI analysis to determine which validation steps are required for your specific task
- Provides smart recommendations on which tools to use next and in what order
- No manual intervention needed - the workflow starts automatically with intelligent guidance
**2. Planning & Design Phase**
- For any implementation task, the AI assistant must first help you create:
- **Detailed coding plan** - Step-by-step implementation approach
- **System design** - Architecture, components, and technical decisions
- **Research findings** - Analysis of existing solutions and best practices
- Once complete, `judge_coding_plan` automatically evaluates the plan using MCP Sampling
- **AI-powered evaluation** checks for design quality, security, research thoroughness, and requirements alignment
**3. Code Implementation Review**
- After any code is written or modified, `judge_code_change` is automatically triggered
- **Mandatory code review** happens immediately after file creation/modification
- Uses MCP Sampling to evaluate code quality, security vulnerabilities, and best practices
- Ensures every code change meets professional standards
### **🤝 User Involvement When Needed**
**Obstacle Resolution**
- When the AI assistant encounters blockers or conflicting requirements, `raise_obstacle` automatically engages you
- Uses MCP Elicitation to present options and get your decision
- No hidden fallbacks - you're always involved in critical decisions
**Requirements Clarification**
- If your request lacks sufficient detail, `elicit_missing_requirements` automatically asks for clarification
- Uses MCP Elicitation to gather specific missing information
- Ensures implementation matches your actual needs
### **🎯 What to Expect**
- **Automatic guidance** - No need to explicitly ask the AI coding assistant to call tools
- **Comprehensive planning** - Every implementation starts with proper design and research
- **Quality enforcement** - All code changes are automatically reviewed against industry standards
- **User-driven decisions** - You're involved whenever your original request cannot be satisfied
- **Professional standards** - Consistent application of software engineering best practices
## 🔒 **Privacy & API Key Free**
### **🔑 No LLM API Key Required**
- All judgments are performed using **MCP Sampling** capability
- No need to configure or pay for external LLM API services
- Works directly with your MCP-compatible client's existing AI model
### **🛡️ Your Privacy Matters**
- The server runs **locally** on your machine
- **No data collection** - your code and conversations stay private
- **No external API calls** - everything happens within your local environment
- Complete control over your development workflow and sensitive information
## 🤝 **Contributing**
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
### **Development Setup**
```bash
# Clone the repository
git clone https://github.com/hepivax/mcp-as-a-judge.git
cd mcp-as-a-judge
# Install dependencies with uv
uv sync --all-extras --dev
# Install pre-commit hooks
uv run pre-commit install
# Run tests
uv run pytest
# Run all checks
uv run pytest && uv run ruff check && uv run ruff format --check && uv run mypy src
```
### **Release Process**
This project uses automated semantic versioning:
1. **Commit with conventional commits**: `feat:`, `fix:`, `docs:`, etc.
2. **Push to main**: Semantic release will automatically create tags and releases
3. **Manual releases**: Create a tag `v1.2.3` and push to trigger release workflow
```bash
# Example conventional commits
git commit -m "feat: add new validation rule for async functions"
git commit -m "fix: resolve memory leak in server startup"
git commit -m "docs: update installation instructions"
```
## 📄 **License**
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🙏 **Acknowledgments**
- [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic
- The amazing MCP community for inspiration and best practices
- All developers who will benefit from better coding practices
---
**🚨 Ready to revolutionize your development workflow? Install MCP as a Judge today!**
Raw data
{
"_id": null,
"home_page": null,
"name": "mcp-as-a-judge",
"maintainer": "Zvi Fried",
"docs_url": null,
"requires_python": ">=3.13",
"maintainer_email": null,
"keywords": "ai, automation, best-practices, code-review, judge, mcp, model-context-protocol, software-engineering",
"author": "Zvi Fried",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/9f/dc/1eb574b593b7bd39cb6c54d8d1b82a40c7470beb98b1f395b42bdbf73003/mcp_as_a_judge-0.1.9.tar.gz",
"platform": null,
"description": "# \u2696\ufe0f MCP as a Judge\n\n> **Prevent bad coding practices with AI-powered evaluation and user-driven decision making**\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n[](https://modelcontextprotocol.io/)\n\n[](https://github.com/hepivax/mcp-as-a-judge/actions/workflows/ci.yml)\n[](https://github.com/hepivax/mcp-as-a-judge/actions/workflows/release.yml)\n[](https://pypi.org/project/mcp-as-a-judge/)\n\n\n\n**MCP as a Judge** is a revolutionary Model Context Protocol (MCP) server that **transforms the developer-AI collaboration experience**. It acts as an intelligent gatekeeper for software development, preventing bad coding practices by using AI-powered evaluation and involving users in critical decisions when requirements are unclear or obstacles arise.\n\n> **\u2696\ufe0f Concept**: This project extends the **LLM-as-a-Judge** paradigm to software engineering workflows, where AI models evaluate and guide development decisions rather than just generating code.\n\n## \u2696\ufe0f **Main Purpose: Improve Developer-AI Interface**\n\n**The core mission is to enhance the interface between developers and AI coding assistants** by:\n\n- \ud83d\udee1\ufe0f **Preventing AI from making poor decisions** through intelligent evaluation\n- \ud83e\udd1d **Involving humans in critical choices** instead of AI making assumptions\n- \ud83d\udd0d **Enforcing research and best practices** before implementation\n- \u2696\ufe0f **Creating a collaborative AI-human workflow** for better software quality\n\n## **Vibe Coding doesn't have to be frustrating**\n\n### **What It Enforces:**\n\n- \u2705 **Deep research** of existing solutions and best practices\n- \u2705 **Generic, reusable solutions** instead of quick fixes\n- \u2705 **User requirements alignment** in all implementations\n- \u2705 **Comprehensive planning** before coding begins\n- \u2705 **User involvement** in all critical decisions\n- \u2705 **Intelligent AI-human collaboration** with clear boundaries and responsibilities\n\n## \ud83d\udee0\ufe0f **Features**\n\n### **\ud83d\udd0d Intelligent Code Evaluation**\n\n- **LLM-powered analysis** using MCP [sampling](https://modelcontextprotocol.io/docs/learn/client-concepts#sampling) capability\n- **Software engineering best practices** enforcement\n- **Security vulnerability detection**\n- **Performance and maintainability assessment**\n\n### **\ud83d\udccb Comprehensive Planning Review**\n\n- **Architecture validation** against industry standards\n- **Research depth analysis** to prevent reinventing solutions\n- **Requirements alignment** verification\n- **Implementation approach evaluation**\n\n### **\ud83e\udd1d User-Driven Decision Making**\n\n- **Obstacle resolution** through user involvement via MCP [elicitation](https://modelcontextprotocol.io/docs/learn/client-concepts#elicitation)\n- **Requirements clarification** when requests are unclear\n- **No hidden fallbacks** - transparent decision making\n- **Interactive problem solving** with real-time user input\n\n### **\ud83d\udee0\ufe0f List of Tools**\n\n| Tool Name | Description |\n|-----------|-------------|\n| **`get_workflow_guidance`** | Smart workflow analysis and tool recommendation |\n| **`judge_coding_plan`** | Comprehensive plan evaluation with requirements alignment |\n| **`judge_code_change`** | Code review with security and quality checks |\n| **`raise_obstacle`** | User involvement when blockers arise |\n| **`elicit_missing_requirements`** | Clarification of unclear requests |\n\n## \ud83d\ude80 **Quick Start**\n\n### **Requirements & Recommendations**\n\n#### **MCP Client Prerequisites**\n\nMCP as a Judge is heavily dependent on **MCP Sampling** and **MCP Elicitation** features for its core functionality:\n\n- **[MCP Sampling](https://modelcontextprotocol.io/docs/learn/client-concepts#sampling)** - Required for AI-powered code evaluation and judgment\n- **[MCP Elicitation](https://modelcontextprotocol.io/docs/learn/client-concepts#elicitation)** - Required for interactive user decision prompts\n\n#### **System Prerequisites**\n\n- **Python 3.13+** - Required for running the MCP server\n\n#### **Supported AI Assistants**\n\n| AI Assistant | Platform | MCP Support | Status | Notes |\n|---------------|----------|-------------|---------|-------|\n| **GitHub Copilot** | Visual Studio Code | \u2705 Full | **Recommended** | Complete MCP integration with tool calling |\n\n**\u2705 Recommended Setup:** GitHub Copilot in Visual Studio Code for the best MCP as a Judge experience.\n\n\n#### **\ud83d\udca1 Recommendations**\n\n- **Large Context Window Models**: 1M+ token size models are strongly preferred for optimal performance\n- Models with larger context windows provide better code analysis and more comprehensive judgments\n\n> **Note**: MCP servers communicate via stdio (standard input/output), not HTTP ports. No network configuration is needed.\n\n\n\n## \ud83d\udd27 **Visual Studio Code Configuration**\n\nConfigure MCP as a Judge in Visual Studio Code with GitHub Copilot:\n\n### **Method 1: Using uv (Recommended)**\n\n1. **Install the package:**\n\n ```bash\n uv tool install mcp-as-a-judge\n ```\n\n2. **Configure Visual Studio Code MCP settings:**\n\n Add this to your Visual Studio Code MCP configuration file:\n\n ```json\n {\n \"servers\": {\n \"mcp-as-a-judge\": {\n \"command\": \"uv\",\n \"args\": [\"tool\", \"run\", \"mcp-as-a-judge\"]\n }\n }\n }\n ```\n\n### **Method 2: Using Docker**\n\n1. **Pull the Docker image:**\n\n ```bash\n docker pull ghcr.io/hepivax/mcp-as-a-judge:latest\n ```\n\n2. **Configure Visual Studio Code MCP settings:**\n\n Add this to your Visual Studio Code MCP configuration file:\n\n ```json\n {\n \"servers\": {\n \"mcp-as-a-judge\": {\n \"command\": \"docker\",\n \"args\": [\"run\", \"--rm\", \"-i\", \"ghcr.io/hepivax/mcp-as-a-judge:latest\"]\n }\n }\n }\n ```\n\n\n## \ud83d\udcd6 **How It Works**\n\nOnce MCP as a Judge is configured in Visual Studio Code with GitHub Copilot, it automatically guides your AI assistant through a structured software engineering workflow. The system operates transparently in the background, ensuring every development task follows best practices.\n\n### **\ud83d\udd04 Automatic Workflow Enforcement**\n\n**1. Intelligent Workflow Guidance**\n\n- When you make any development request, the AI assistant automatically calls `get_workflow_guidance`\n- This tool uses AI analysis to determine which validation steps are required for your specific task\n- Provides smart recommendations on which tools to use next and in what order\n- No manual intervention needed - the workflow starts automatically with intelligent guidance\n\n**2. Planning & Design Phase**\n\n- For any implementation task, the AI assistant must first help you create:\n - **Detailed coding plan** - Step-by-step implementation approach\n - **System design** - Architecture, components, and technical decisions\n - **Research findings** - Analysis of existing solutions and best practices\n- Once complete, `judge_coding_plan` automatically evaluates the plan using MCP Sampling\n- **AI-powered evaluation** checks for design quality, security, research thoroughness, and requirements alignment\n\n**3. Code Implementation Review**\n\n- After any code is written or modified, `judge_code_change` is automatically triggered\n- **Mandatory code review** happens immediately after file creation/modification\n- Uses MCP Sampling to evaluate code quality, security vulnerabilities, and best practices\n- Ensures every code change meets professional standards\n\n### **\ud83e\udd1d User Involvement When Needed**\n\n**Obstacle Resolution**\n\n- When the AI assistant encounters blockers or conflicting requirements, `raise_obstacle` automatically engages you\n- Uses MCP Elicitation to present options and get your decision\n- No hidden fallbacks - you're always involved in critical decisions\n\n**Requirements Clarification**\n\n- If your request lacks sufficient detail, `elicit_missing_requirements` automatically asks for clarification\n- Uses MCP Elicitation to gather specific missing information\n- Ensures implementation matches your actual needs\n\n### **\ud83c\udfaf What to Expect**\n\n- **Automatic guidance** - No need to explicitly ask the AI coding assistant to call tools\n- **Comprehensive planning** - Every implementation starts with proper design and research\n- **Quality enforcement** - All code changes are automatically reviewed against industry standards\n- **User-driven decisions** - You're involved whenever your original request cannot be satisfied\n- **Professional standards** - Consistent application of software engineering best practices\n\n## \ud83d\udd12 **Privacy & API Key Free**\n\n### **\ud83d\udd11 No LLM API Key Required**\n\n- All judgments are performed using **MCP Sampling** capability\n- No need to configure or pay for external LLM API services\n- Works directly with your MCP-compatible client's existing AI model\n\n### **\ud83d\udee1\ufe0f Your Privacy Matters**\n\n- The server runs **locally** on your machine\n- **No data collection** - your code and conversations stay private\n- **No external API calls** - everything happens within your local environment\n- Complete control over your development workflow and sensitive information\n\n## \ud83e\udd1d **Contributing**\n\nWe welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### **Development Setup**\n\n```bash\n# Clone the repository\ngit clone https://github.com/hepivax/mcp-as-a-judge.git\ncd mcp-as-a-judge\n\n# Install dependencies with uv\nuv sync --all-extras --dev\n\n# Install pre-commit hooks\nuv run pre-commit install\n\n# Run tests\nuv run pytest\n\n# Run all checks\nuv run pytest && uv run ruff check && uv run ruff format --check && uv run mypy src\n```\n\n### **Release Process**\n\nThis project uses automated semantic versioning:\n\n1. **Commit with conventional commits**: `feat:`, `fix:`, `docs:`, etc.\n2. **Push to main**: Semantic release will automatically create tags and releases\n3. **Manual releases**: Create a tag `v1.2.3` and push to trigger release workflow\n\n```bash\n# Example conventional commits\ngit commit -m \"feat: add new validation rule for async functions\"\ngit commit -m \"fix: resolve memory leak in server startup\"\ngit commit -m \"docs: update installation instructions\"\n```\n\n## \ud83d\udcc4 **License**\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## \ud83d\ude4f **Acknowledgments**\n\n- [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic\n- The amazing MCP community for inspiration and best practices\n- All developers who will benefit from better coding practices\n\n---\n\n**\ud83d\udea8 Ready to revolutionize your development workflow? Install MCP as a Judge today!**\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "MCP as a Judge: Prevent bad coding practices with AI-powered evaluation and user-driven decision making",
"version": "0.1.9",
"project_urls": {
"Documentation": "https://github.com/hepivax/mcp-as-a-judge#readme",
"Homepage": "https://github.com/hepivax/mcp-as-a-judge",
"Issues": "https://github.com/hepivax/mcp-as-a-judge/issues",
"Repository": "https://github.com/hepivax/mcp-as-a-judge"
},
"split_keywords": [
"ai",
" automation",
" best-practices",
" code-review",
" judge",
" mcp",
" model-context-protocol",
" software-engineering"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "8afead25a03d81bab76d10203836a11af4e5d02d09f1af586cf826dd3662b43d",
"md5": "1634bcaa08e9a0804b0358e5dedab13b",
"sha256": "fb8168b1ec6d2a61d59f753ff5e9d6d5d9b9a96f2f0c90605b456beac5298699"
},
"downloads": -1,
"filename": "mcp_as_a_judge-0.1.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1634bcaa08e9a0804b0358e5dedab13b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.13",
"size": 26067,
"upload_time": "2025-08-30T18:41:58",
"upload_time_iso_8601": "2025-08-30T18:41:58.350277Z",
"url": "https://files.pythonhosted.org/packages/8a/fe/ad25a03d81bab76d10203836a11af4e5d02d09f1af586cf826dd3662b43d/mcp_as_a_judge-0.1.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "9fdc1eb574b593b7bd39cb6c54d8d1b82a40c7470beb98b1f395b42bdbf73003",
"md5": "74c16177c5cd350a28f8e36200cdc55d",
"sha256": "ed96911a25eb7eefb9dd7e36643112c899693475b7c35dd3634842669f22a602"
},
"downloads": -1,
"filename": "mcp_as_a_judge-0.1.9.tar.gz",
"has_sig": false,
"md5_digest": "74c16177c5cd350a28f8e36200cdc55d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.13",
"size": 77168,
"upload_time": "2025-08-30T18:41:59",
"upload_time_iso_8601": "2025-08-30T18:41:59.427146Z",
"url": "https://files.pythonhosted.org/packages/9f/dc/1eb574b593b7bd39cb6c54d8d1b82a40c7470beb98b1f395b42bdbf73003/mcp_as_a_judge-0.1.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-30 18:41:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hepivax",
"github_project": "mcp-as-a-judge#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mcp-as-a-judge"
}
JSON
