# Clyrdia CLI - Zero-Knowledge AI Benchmarking Platform
[](https://badge.fury.io/py/clyrdia-cli)
[](https://www.python.org/downloads/)
[](LICENSE)
🚀 **The most advanced local-first AI model benchmarking tool** - Test, compare, and evaluate AI models with zero data exposure.
## ✨ Features
- **🔒 Zero-Knowledge**: Your data never leaves your system
- **📊 Comprehensive Benchmarking**: Test multiple AI models simultaneously
- **🎯 Production-Ready**: Includes customizable production benchmark suite
- **⚡ High Performance**: Async processing with intelligent caching
- **📈 Quality Evaluation**: Advanced metrics and scoring systems
- **🔄 Continuous Testing**: Canary and ratchet systems for ongoing validation
- **💾 Local Storage**: All results stored locally with optional cloud sync
- **👥 Team Collaboration**: Multi-user support with role-based access (Business tier)
- **🚀 CI/CD Integration**: Automated testing and quality gates (Business tier)
## 🚀 Quick Start
### Installation
```bash
pip install clyrdia-cli
```
### Basic Usage
```bash
# Run a benchmark with the included production configuration
clyrdia-cli run --config production_benchmark.yaml
# Run with custom configuration
clyrdia-cli run --config your_config.yaml
# View help and available commands
clyrdia-cli --help
```
## 📋 Prerequisites
- Python 3.8 or higher
- API keys for the AI models you want to test:
- OpenAI API key for GPT models
- Anthropic API key for Claude models
## 🔧 Configuration
### Environment Setup
Create a `.env` file in your project directory:
```bash
# OpenAI API Configuration
OPENAI_API_KEY=your_openai_api_key_here
# Anthropic API Configuration
ANTHROPIC_API_KEY=your_anthropic_api_key_here
# Optional: Custom configuration
CLYRIDIA_CONFIG_PATH=./config.yaml
```
### Benchmark Configuration
The package includes a production-ready benchmark configuration (`production_benchmark.yaml`) that tests:
- **Financial Risk Assessment** - Complex business analysis
- **Legal Contract Review** - Deep legal understanding
- **Marketing Strategy Development** - Creative and analytical thinking
- **System Architecture Design** - Technical complexity
- **Customer Success Strategy** - Business operations
### Custom Benchmark Configuration
Create your own benchmark configuration:
```yaml
name: "Custom Benchmark Suite"
description: "Your custom AI model testing scenarios"
models:
- "claude-opus-4.1"
- "gpt-5o"
tests:
- name: "Your Test Case"
prompt: |
Your custom prompt here...
expected_output: "Expected output description"
max_tokens: 2000
temperature: 0.3
evaluation_criteria:
- "accuracy"
- "completeness"
- "relevance"
```
## 💰 Pricing Tiers
### **Developer Tier (Free)**
- **Target**: All developers, everywhere
- **Goal**: Drive massive top-of-funnel adoption and create fans
- **Offer**: 100 credits/month, single user, no CI/CD
### **Pro Tier ($25/month)**
- **Target**: The serious individual professional or freelancer
- **Goal**: Monetize power users and provide a stepping stone
- **Offer**: 1,000 credits/month, single user, still no CI/CD
### **Business Tier ($500/month)**
- **Target**: Professional teams of 2-10 developers
- **Goal**: Be your primary revenue engine
- **Offer**: 25,000 credits/month, up to 10 users, and the killer CI/CD Integration feature
## 🎯 Available Commands
### `run` - Execute Benchmark
```bash
clyrdia-cli run --config production_benchmark.yaml
```
Options:
- `--config, -c`: Path to benchmark configuration file
- `--output, -o`: Output directory for results
- `--cache, --no-cache`: Enable/disable caching
- `--verbose, -v`: Verbose output
### `auth` - Authentication Management
```bash
clyrdia-cli auth --setup
clyrdia-cli auth --status
```
### `dashboard` - Launch Web Dashboard
```bash
clyrdia-cli dashboard
```
### `cache` - Cache Management
```bash
clyrdia-cli cache --clear
clyrdia-cli cache --status
```
### `plans` - View Subscription Plans
```bash
clyrdia-cli plans
```
### `status` - Account Status
```bash
clyrdia-cli status
```
### `team` - Team Management (Business Tier)
```bash
clyrdia-cli team
```
### `cicd` - CI/CD Integration (Business Tier)
```bash
clyrdia-cli cicd
```
### `upgrade` - Plan Upgrade Information
```bash
clyrdia-cli upgrade
```
## 📊 Output and Results
Benchmark results are saved in JSON format with comprehensive metrics:
- **Performance Metrics**: Response time, token usage, cost analysis
- **Quality Scores**: Accuracy, completeness, business relevance
- **Comparative Analysis**: Model-to-model performance comparison
- **Detailed Logs**: Full conversation history and evaluation details
## 🔍 Advanced Features
### Caching System
- Smart caching for repeated requests
- Configurable TTL and storage limits
- Cost optimization for development and testing
### Quality Evaluation
- Multi-dimensional scoring system
- Business relevance assessment
- Execution feasibility analysis
### Continuous Testing
- Canary system for ongoing validation
- Ratchet system for quality gates
- Automated regression detection
## 🏗️ Architecture
```
clyrdia/
├── cli_modular.py # Main CLI application
├── benchmarking/ # Core benchmarking engine
├── models/ # Data models and configurations
├── caching/ # Intelligent caching system
├── auth/ # Authentication and licensing
├── database/ # Local data storage
├── core/ # Core utilities and decorators
└── utils/ # Helper functions
```
## 🧪 Testing
Run the test suite:
```bash
# Install development dependencies
pip install -e ".[dev]"
# Run tests
pytest tests/
# Run with coverage
pytest --cov=clyrdia tests/
```
## 📦 Package Development
### Building the Package
```bash
# Build source distribution
python -m build --sdist
# Build wheel
python -m build --wheel
# Build both
python -m build
```
### Publishing to Test PyPI
```bash
# Upload to Test PyPI
python -m twine upload --repository testpypi dist/*
# Install from Test PyPI
pip install --index-url https://test.pypi.org/simple/ clyrdia-cli
```
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request
## 📄 License
This project is proprietary software. See [LICENSE](LICENSE) for details.
## 🆘 Support
- **Documentation**: [https://docs.clyrdia.com](https://docs.clyrdia.com)
- **Issues**: [GitHub Issues](https://github.com/clyrdia/clyrdia-cli/issues)
- **Email**: team@clyrdia.com
## 🗺️ Roadmap
- [ ] Cloud synchronization
- [ ] Advanced analytics dashboard
- [ ] Model fine-tuning integration
- [ ] Enterprise SSO support
- [ ] Multi-language support
- [ ] API rate limiting optimization
---
**Built with ❤️ by the Clyrdia Team**
Raw data
{
"_id": null,
"home_page": "https://clyrdia.com",
"name": "clyrdia-cli",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Clyrdia Team <team@clyrdia.com>",
"keywords": "ai, benchmarking, machine learning, testing, evaluation, openai, anthropic",
"author": "Clyrdia Team",
"author_email": "Clyrdia Team <team@clyrdia.com>",
"download_url": "https://files.pythonhosted.org/packages/48/e9/b75cb50ee26f53bdc87654bf9d4961ffb0f8a901da84bd2064950d03bf17/clyrdia_cli-1.2.1.tar.gz",
"platform": null,
"description": "# Clyrdia CLI - Zero-Knowledge AI Benchmarking Platform\n\n[](https://badge.fury.io/py/clyrdia-cli)\n[](https://www.python.org/downloads/)\n[](LICENSE)\n\n\ud83d\ude80 **The most advanced local-first AI model benchmarking tool** - Test, compare, and evaluate AI models with zero data exposure.\n\n## \u2728 Features\n\n- **\ud83d\udd12 Zero-Knowledge**: Your data never leaves your system\n- **\ud83d\udcca Comprehensive Benchmarking**: Test multiple AI models simultaneously\n- **\ud83c\udfaf Production-Ready**: Includes customizable production benchmark suite\n- **\u26a1 High Performance**: Async processing with intelligent caching\n- **\ud83d\udcc8 Quality Evaluation**: Advanced metrics and scoring systems\n- **\ud83d\udd04 Continuous Testing**: Canary and ratchet systems for ongoing validation\n- **\ud83d\udcbe Local Storage**: All results stored locally with optional cloud sync\n- **\ud83d\udc65 Team Collaboration**: Multi-user support with role-based access (Business tier)\n- **\ud83d\ude80 CI/CD Integration**: Automated testing and quality gates (Business tier)\n\n## \ud83d\ude80 Quick Start\n\n### Installation\n\n```bash\npip install clyrdia-cli\n```\n\n### Basic Usage\n\n```bash\n# Run a benchmark with the included production configuration\nclyrdia-cli run --config production_benchmark.yaml\n\n# Run with custom configuration\nclyrdia-cli run --config your_config.yaml\n\n# View help and available commands\nclyrdia-cli --help\n```\n\n## \ud83d\udccb Prerequisites\n\n- Python 3.8 or higher\n- API keys for the AI models you want to test:\n - OpenAI API key for GPT models\n - Anthropic API key for Claude models\n\n## \ud83d\udd27 Configuration\n\n### Environment Setup\n\nCreate a `.env` file in your project directory:\n\n```bash\n# OpenAI API Configuration\nOPENAI_API_KEY=your_openai_api_key_here\n\n# Anthropic API Configuration \nANTHROPIC_API_KEY=your_anthropic_api_key_here\n\n# Optional: Custom configuration\nCLYRIDIA_CONFIG_PATH=./config.yaml\n```\n\n### Benchmark Configuration\n\nThe package includes a production-ready benchmark configuration (`production_benchmark.yaml`) that tests:\n\n- **Financial Risk Assessment** - Complex business analysis\n- **Legal Contract Review** - Deep legal understanding\n- **Marketing Strategy Development** - Creative and analytical thinking\n- **System Architecture Design** - Technical complexity\n- **Customer Success Strategy** - Business operations\n\n### Custom Benchmark Configuration\n\nCreate your own benchmark configuration:\n\n```yaml\nname: \"Custom Benchmark Suite\"\ndescription: \"Your custom AI model testing scenarios\"\n\nmodels:\n - \"claude-opus-4.1\"\n - \"gpt-5o\"\n\ntests:\n - name: \"Your Test Case\"\n prompt: |\n Your custom prompt here...\n expected_output: \"Expected output description\"\n max_tokens: 2000\n temperature: 0.3\n evaluation_criteria:\n - \"accuracy\"\n - \"completeness\"\n - \"relevance\"\n```\n\n## \ud83d\udcb0 Pricing Tiers\n\n### **Developer Tier (Free)**\n- **Target**: All developers, everywhere\n- **Goal**: Drive massive top-of-funnel adoption and create fans\n- **Offer**: 100 credits/month, single user, no CI/CD\n\n### **Pro Tier ($25/month)**\n- **Target**: The serious individual professional or freelancer\n- **Goal**: Monetize power users and provide a stepping stone\n- **Offer**: 1,000 credits/month, single user, still no CI/CD\n\n### **Business Tier ($500/month)**\n- **Target**: Professional teams of 2-10 developers\n- **Goal**: Be your primary revenue engine\n- **Offer**: 25,000 credits/month, up to 10 users, and the killer CI/CD Integration feature\n\n## \ud83c\udfaf Available Commands\n\n### `run` - Execute Benchmark\n```bash\nclyrdia-cli run --config production_benchmark.yaml\n```\n\nOptions:\n- `--config, -c`: Path to benchmark configuration file\n- `--output, -o`: Output directory for results\n- `--cache, --no-cache`: Enable/disable caching\n- `--verbose, -v`: Verbose output\n\n### `auth` - Authentication Management\n```bash\nclyrdia-cli auth --setup\nclyrdia-cli auth --status\n```\n\n### `dashboard` - Launch Web Dashboard\n```bash\nclyrdia-cli dashboard\n```\n\n### `cache` - Cache Management\n```bash\nclyrdia-cli cache --clear\nclyrdia-cli cache --status\n```\n\n### `plans` - View Subscription Plans\n```bash\nclyrdia-cli plans\n```\n\n### `status` - Account Status\n```bash\nclyrdia-cli status\n```\n\n### `team` - Team Management (Business Tier)\n```bash\nclyrdia-cli team\n```\n\n### `cicd` - CI/CD Integration (Business Tier)\n```bash\nclyrdia-cli cicd\n```\n\n### `upgrade` - Plan Upgrade Information\n```bash\nclyrdia-cli upgrade\n```\n\n## \ud83d\udcca Output and Results\n\nBenchmark results are saved in JSON format with comprehensive metrics:\n\n- **Performance Metrics**: Response time, token usage, cost analysis\n- **Quality Scores**: Accuracy, completeness, business relevance\n- **Comparative Analysis**: Model-to-model performance comparison\n- **Detailed Logs**: Full conversation history and evaluation details\n\n## \ud83d\udd0d Advanced Features\n\n### Caching System\n- Smart caching for repeated requests\n- Configurable TTL and storage limits\n- Cost optimization for development and testing\n\n### Quality Evaluation\n- Multi-dimensional scoring system\n- Business relevance assessment\n- Execution feasibility analysis\n\n### Continuous Testing\n- Canary system for ongoing validation\n- Ratchet system for quality gates\n- Automated regression detection\n\n## \ud83c\udfd7\ufe0f Architecture\n\n```\nclyrdia/\n\u251c\u2500\u2500 cli_modular.py # Main CLI application\n\u251c\u2500\u2500 benchmarking/ # Core benchmarking engine\n\u251c\u2500\u2500 models/ # Data models and configurations\n\u251c\u2500\u2500 caching/ # Intelligent caching system\n\u251c\u2500\u2500 auth/ # Authentication and licensing\n\u251c\u2500\u2500 database/ # Local data storage\n\u251c\u2500\u2500 core/ # Core utilities and decorators\n\u2514\u2500\u2500 utils/ # Helper functions\n```\n\n## \ud83e\uddea Testing\n\nRun the test suite:\n\n```bash\n# Install development dependencies\npip install -e \".[dev]\"\n\n# Run tests\npytest tests/\n\n# Run with coverage\npytest --cov=clyrdia tests/\n```\n\n## \ud83d\udce6 Package Development\n\n### Building the Package\n\n```bash\n# Build source distribution\npython -m build --sdist\n\n# Build wheel\npython -m build --wheel\n\n# Build both\npython -m build\n```\n\n### Publishing to Test PyPI\n\n```bash\n# Upload to Test PyPI\npython -m twine upload --repository testpypi dist/*\n\n# Install from Test PyPI\npip install --index-url https://test.pypi.org/simple/ clyrdia-cli\n```\n\n## \ud83e\udd1d Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests for new functionality\n5. Submit a pull request\n\n## \ud83d\udcc4 License\n\nThis project is proprietary software. See [LICENSE](LICENSE) for details.\n\n## \ud83c\udd98 Support\n\n- **Documentation**: [https://docs.clyrdia.com](https://docs.clyrdia.com)\n- **Issues**: [GitHub Issues](https://github.com/clyrdia/clyrdia-cli/issues)\n- **Email**: team@clyrdia.com\n\n## \ud83d\uddfa\ufe0f Roadmap\n\n- [ ] Cloud synchronization\n- [ ] Advanced analytics dashboard\n- [ ] Model fine-tuning integration\n- [ ] Enterprise SSO support\n- [ ] Multi-language support\n- [ ] API rate limiting optimization\n\n---\n\n**Built with \u2764\ufe0f by the Clyrdia Team**\n\n",
"bugtrack_url": null,
"license": "Proprietary",
"summary": "Zero-Knowledge AI Benchmarking Platform",
"version": "1.2.1",
"project_urls": {
"Homepage": "https://clyrdia.com"
},
"split_keywords": [
"ai",
" benchmarking",
" machine learning",
" testing",
" evaluation",
" openai",
" anthropic"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "5749291bba730e2611f18123f7966fe1e2ba0717cf89613b1ac14be1a853acc1",
"md5": "62406674de88be960993fdb66e652efa",
"sha256": "0ba548c924625bcd468b664558fe63bb964afdba8649ed6fa8aed71e86e29f79"
},
"downloads": -1,
"filename": "clyrdia_cli-1.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "62406674de88be960993fdb66e652efa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 122017,
"upload_time": "2025-08-28T01:02:30",
"upload_time_iso_8601": "2025-08-28T01:02:30.483734Z",
"url": "https://files.pythonhosted.org/packages/57/49/291bba730e2611f18123f7966fe1e2ba0717cf89613b1ac14be1a853acc1/clyrdia_cli-1.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "48e9b75cb50ee26f53bdc87654bf9d4961ffb0f8a901da84bd2064950d03bf17",
"md5": "d9f5b5e6ecf0cce6f1ba6219e4451759",
"sha256": "7ea6367826881cf7c0bd044c791ed80feff493dd5681de44acb6802f708f7eed"
},
"downloads": -1,
"filename": "clyrdia_cli-1.2.1.tar.gz",
"has_sig": false,
"md5_digest": "d9f5b5e6ecf0cce6f1ba6219e4451759",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 115453,
"upload_time": "2025-08-28T01:02:31",
"upload_time_iso_8601": "2025-08-28T01:02:31.553177Z",
"url": "https://files.pythonhosted.org/packages/48/e9/b75cb50ee26f53bdc87654bf9d4961ffb0f8a901da84bd2064950d03bf17/clyrdia_cli-1.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-28 01:02:31",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "clyrdia-cli"
}
JSON