# ContextCheck
A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.
**ContextCheck** is an open-source framework designed to evaluate, test, and validate large language models (LLMs), Retrieval-Augmented Generation (RAG) systems, and chatbots. It provides tools to automatically generate queries, request completions, detect regressions, perform penetration tests, and assess hallucinations, ensuring the robustness and reliability of these systems. ContextCheck is configurable via YAML and can be integrated into continuous integration (CI) pipelines for automated testing.
## Table of Contents
- [Features](https://github.com/Addepto/contextcheck#features)
- [Installation](https://github.com/Addepto/contextcheck#installation)
- [For Users](https://github.com/Addepto/contextcheck#for-users)
- [For Developers](https://github.com/Addepto/contextcheck#for-developers)
- [Tutorial](https://github.com/Addepto/contextcheck#tutorial)
- [CLI Features](https://github.com/Addepto/contextcheck#cli-features)
- [Output Test Results to Console](https://github.com/Addepto/contextcheck#output-test-results-to-console)
- [Running in CI/CD](https://github.com/Addepto/contextcheck#running-in-cicd)
- [Contributing](https://github.com/Addepto/contextcheck#contributing)
- [Running Tests](https://github.com/Addepto/contextcheck#running-tests)
- [Acknowledgments](https://github.com/Addepto/contextcheck#acknowledgments)
- [License](https://github.com/Addepto/contextcheck#license)
## Features
- **Simple test scenario definition** using human-readable `.yaml` files
- **Flexible endpoint configuration** for OpenAI, HTTP, and more
- **Customizable JSON request/response models**
- **Support for variables and Jinja2 templating** in YAML files
- **Response validation** options, including heuristics, LLM-based judgment, and human labeling
- **Enhanced output formatting** with the `rich` package for clear, readable displays
## Installation
### For Users
Install the package directly from PyPI using pip:
```sh
pip install ccheck
```
After installation, you can access the `ccheck` CLI command:
```sh
ccheck --help
```
This will display all available options and help you get started with using ContextCheck.
### For Developers
If you wish to contribute to the project or modify it for your own use, you can set up a development environment using Poetry.
1. Fork your own copy of Addepto/contextcheck on GitHub.
2. Clone the Repository:
```sh
git clone https://github.com/<your_username>/contextcheck.git
cd contextcheck
```
3. Ensure you have [Poetry](https://python-poetry.org/) installed.
4. Install Dependencies:
```sh
poetry install
```
5. Activate the Virtual Environment:
```sh
poetry shell
```
6. Activate the `ccheck` CLI command using:
```sh
poetry run ccheck --help
```
## Tutorial
Please refer to `examples/` folder for the tutorial.
## CLI Features
### Output Test Results to Console
- **Run a single scenario and output results to the console:**
```sh
ccheck --output-type console --filename path/to/file.yaml
```
- **Run multiple scenarios and output results to the console:**
```sh
ccheck --output-type console --filename path/to/file.yaml path/to/another_file.yaml
```
### Running in CI/CD
To automatically stop the CI/CD process if any tests fail, add the `--exit-on-failure` flag. Failed test will cause the script to exit with code 1:
```sh
ccheck --exit-on-failure --output-type console --folder my_tests
```
Use env variable `OPENAI_API_KEY` to be able to run:
- `tests/scenario_openai.yaml`
- `tests/scenario_defaults.yaml`
## Contributing
Contributions are welcomed!
### Running Tests
To run tests:
```
poetry run pytest tests/
```
To include tests which require calling LLM APIs (currently OpenAI and Ollama), run one of:
```
poetry run pytest --openai # includes tests that use OpenAI API
poetry run pytest --ollama # includes tests that use Ollama API
poetry run pytest --openai --ollama # includes tests that use both OpenAI and Ollama API
```
## Acknowledgments
Made with ❤️ by the Addepto Team
ContextCheck is an extension of the [ContextClue](https://context-clue.com/) product, created by the [Addepto](https://addepto.com/) team. This project is the result of our team’s dedication, combining innovation and expertise.
Addepto Team:
* Radoslaw Bodus
* Bartlomiej Grasza
* Volodymyr Kepsha
* Vadym Mariiechko
* Michal Tarkowski
Like what we’re building? ⭐ Give it a star to support its development!
## License
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details
Raw data
{
"_id": null,
"home_page": "https://github.com/Addepto/contextcheck",
"name": "ccheck",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "LLM, RAG, Chatbot, Testing, Validation",
"author": "Addepto",
"author_email": "hi@addepto.com",
"download_url": "https://files.pythonhosted.org/packages/da/61/7a57fa1db5654abda17b201c022d218e93e9bf28a1dbdfd475b5a405af7c/ccheck-0.1.1.tar.gz",
"platform": null,
"description": "\n\n# ContextCheck\n\nA human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.\n\n**ContextCheck** is an open-source framework designed to evaluate, test, and validate large language models (LLMs), Retrieval-Augmented Generation (RAG) systems, and chatbots. It provides tools to automatically generate queries, request completions, detect regressions, perform penetration tests, and assess hallucinations, ensuring the robustness and reliability of these systems. ContextCheck is configurable via YAML and can be integrated into continuous integration (CI) pipelines for automated testing.\n\n\n## Table of Contents\n- [Features](https://github.com/Addepto/contextcheck#features)\n- [Installation](https://github.com/Addepto/contextcheck#installation)\n - [For Users](https://github.com/Addepto/contextcheck#for-users)\n - [For Developers](https://github.com/Addepto/contextcheck#for-developers)\n- [Tutorial](https://github.com/Addepto/contextcheck#tutorial)\n- [CLI Features](https://github.com/Addepto/contextcheck#cli-features)\n - [Output Test Results to Console](https://github.com/Addepto/contextcheck#output-test-results-to-console)\n - [Running in CI/CD](https://github.com/Addepto/contextcheck#running-in-cicd)\n- [Contributing](https://github.com/Addepto/contextcheck#contributing)\n - [Running Tests](https://github.com/Addepto/contextcheck#running-tests)\n- [Acknowledgments](https://github.com/Addepto/contextcheck#acknowledgments)\n- [License](https://github.com/Addepto/contextcheck#license)\n\n\n## Features\n\n- **Simple test scenario definition** using human-readable `.yaml` files\n- **Flexible endpoint configuration** for OpenAI, HTTP, and more\n- **Customizable JSON request/response models**\n- **Support for variables and Jinja2 templating** in YAML files\n- **Response validation** options, including heuristics, LLM-based judgment, and human labeling\n- **Enhanced output formatting** with the `rich` package for clear, readable displays\n\n\n## Installation\n\n### For Users\n\nInstall the package directly from PyPI using pip:\n\n```sh\npip install ccheck\n```\n\nAfter installation, you can access the `ccheck` CLI command:\n\n```sh\nccheck --help\n```\n\nThis will display all available options and help you get started with using ContextCheck.\n\n### For Developers\n\nIf you wish to contribute to the project or modify it for your own use, you can set up a development environment using Poetry.\n\n1. Fork your own copy of Addepto/contextcheck on GitHub.\n2. Clone the Repository:\n ```sh\n git clone https://github.com/<your_username>/contextcheck.git\n cd contextcheck\n ```\n3. Ensure you have [Poetry](https://python-poetry.org/) installed.\n4. Install Dependencies:\n ```sh\n poetry install\n ```\n5. Activate the Virtual Environment:\n ```sh\n poetry shell\n ```\n6. Activate the `ccheck` CLI command using:\n ```sh\n poetry run ccheck --help\n ```\n\n\n## Tutorial\n\nPlease refer to `examples/` folder for the tutorial.\n\n\n## CLI Features\n\n### Output Test Results to Console\n\n- **Run a single scenario and output results to the console:**\n ```sh\n ccheck --output-type console --filename path/to/file.yaml\n ```\n- **Run multiple scenarios and output results to the console:**\n ```sh\n ccheck --output-type console --filename path/to/file.yaml path/to/another_file.yaml\n ```\n\n### Running in CI/CD\n\nTo automatically stop the CI/CD process if any tests fail, add the `--exit-on-failure` flag. Failed test will cause the script to exit with code 1:\n\n```sh\nccheck --exit-on-failure --output-type console --folder my_tests\n```\n\nUse env variable `OPENAI_API_KEY` to be able to run:\n- `tests/scenario_openai.yaml`\n- `tests/scenario_defaults.yaml`\n\n\n## Contributing\n\nContributions are welcomed!\n\n### Running Tests\n\nTo run tests:\n```\npoetry run pytest tests/\n```\n\nTo include tests which require calling LLM APIs (currently OpenAI and Ollama), run one of:\n```\npoetry run pytest --openai # includes tests that use OpenAI API\npoetry run pytest --ollama # includes tests that use Ollama API\npoetry run pytest --openai --ollama # includes tests that use both OpenAI and Ollama API\n```\n\n\n## Acknowledgments\n\nMade with \u2764\ufe0f by the Addepto Team\n\nContextCheck is an extension of the [ContextClue](https://context-clue.com/) product, created by the [Addepto](https://addepto.com/) team. This project is the result of our team\u2019s dedication, combining innovation and expertise.\n\nAddepto Team:\n\n* Radoslaw Bodus\n* Bartlomiej Grasza\n* Volodymyr Kepsha\n* Vadym Mariiechko\n* Michal Tarkowski\n\nLike what we\u2019re building? \u2b50 Give it a star to support its development!\n\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/Addepto/contextcheck",
"Repository": "https://github.com/Addepto/contextcheck"
},
"split_keywords": [
"llm",
" rag",
" chatbot",
" testing",
" validation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e9595cdd9a43358f5f8fb032d7d8f754956e557cc836fa40164cb3d2b37d4be8",
"md5": "1999c4adf781438a5c488a8453b479b3",
"sha256": "a16e045a040ccdfbe6f376e2ca977d3bb06b64ec1ee02c105c35b94126e6f607"
},
"downloads": -1,
"filename": "ccheck-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1999c4adf781438a5c488a8453b479b3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 37626,
"upload_time": "2024-11-15T15:42:24",
"upload_time_iso_8601": "2024-11-15T15:42:24.621913Z",
"url": "https://files.pythonhosted.org/packages/e9/59/5cdd9a43358f5f8fb032d7d8f754956e557cc836fa40164cb3d2b37d4be8/ccheck-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "da617a57fa1db5654abda17b201c022d218e93e9bf28a1dbdfd475b5a405af7c",
"md5": "b28a7c2df26a2650a80cc879f2302a0d",
"sha256": "44bf0c6026ad0649a35c516a39aea92e6b8495af4075c27433bc8e01bb54cacf"
},
"downloads": -1,
"filename": "ccheck-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "b28a7c2df26a2650a80cc879f2302a0d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 25358,
"upload_time": "2024-11-15T15:42:25",
"upload_time_iso_8601": "2024-11-15T15:42:25.753514Z",
"url": "https://files.pythonhosted.org/packages/da/61/7a57fa1db5654abda17b201c022d218e93e9bf28a1dbdfd475b5a405af7c/ccheck-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-15 15:42:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Addepto",
"github_project": "contextcheck",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "ccheck"
}
JSON
