# AILintTest
AILintTest is a Python testing library that uses Large Language Models to verify code properties through natural language assertions. It seamlessly integrates with popular testing frameworks like pytest and unittest, allowing developers to express complex code expectations in plain English while maintaining the structure of traditional test suites.
## Why AILintTest?
Traditional testing approaches often require extensive boilerplate code to verify complex properties. AILintTest bridges this gap by allowing developers to express sophisticated test cases in natural language, particularly useful for scenarios where writing conventional test code would be impractical or time-consuming.
### Key Features
1. **Natural Language Test Cases**: Write test assertions in plain English
2. **Framework Integration**: Works with pytest, unittest, and other testing frameworks
3. **Deterministic Results**: Uses voting mechanism and controlled sampling for consistent results
4. **Flexible Verification**: Test complex code properties that would be difficult to verify traditionally
## When to Use AILintTest
AILintTest is designed for scenarios where traditional test implementation would be impractical or require excessive code. For example:
```python
# Traditional approach would require:
# 1. Iterating through all methods
# 2. Parsing AST for each method
# 3. Checking exception handling patterns
# 4. Verifying logging calls
# 5. Maintaining complex test code
# With AILintTest:
def test_error_handling():
ailint.assert_code(
"All methods in {module} should use the custom ErrorHandler class for exception management, and log errors before re-raising them",
{"module": my_critical_module}
)
# Another example - checking documentation consistency
def test_docstring_completeness():
ailint.assert_code(
"All public methods in {module} should have docstrings that include Parameters, Returns, and Examples sections",
{"module": my_api_module}
)
```
## How It Works
### Deterministic Testing
AILintTest employs several mechanisms to ensure consistent and reliable results:
1. **Voting Mechanism**: Each assertion is evaluated multiple times (configurable through `quorum_size`), and the majority result is used
2. **Temperature Control**: Uses low temperature for LLM sampling to reduce randomness
3. **Structured Prompts**: Converts natural language assertions into structured prompts for consistent LLM interpretation
```python
# Configure determinism settings
options = AILintTestOptions(
quorum_size=5, # Number of evaluations per assertion
)
```
## Installation
```bash
pip install ailinttest
```
## Basic Usage
### With pytest
```python
from ailinttest import AILintTest
def test_code_properties():
ailint = AILintTest()
# Test code organization
ailint.assert_code(
"Classes in {module} should follow the Single Responsibility Principle",
{"module": my_module}
)
# Test security practices
ailint.assert_code(
"All database queries in {module} should be parameterized to prevent SQL injection",
{"module": db_module}
)
```
### With unittest
```python
import unittest
from ailinttest import AILintTest
class TestCodeQuality(unittest.TestCase):
def setUp(self):
self.ailint = AILintTest()
def test_error_handling(self):
self.ailint.assert_code(
"All API endpoints in {module} should have proper input validation",
{"module": api_module}
)
```
## Advanced Usage
### Custom Evaluation Options
```python
from ailinttest import AILintTest, AILintTestOptions
options = AILintTestOptions(
quorum_size=7, # Increase voting sample size
model="gpt-4o-2024-08-06", # Use a more capable model
)
ailint = AILintTest(options)
```
## Contributing
Contributions are welcome!
## License
[MIT License](LICENSE)
## Acknowledgements
This project uses [LiteLLM](https://github.com/BerriAI/litellm) for LLM integration.
---
AILintTest is designed to complement, not replace, traditional testing approaches. It's most effective when used for complex code properties that are difficult to verify through conventional means.
Raw data
{
"_id": null,
"home_page": "https://github.com/kdunee/ailinttest",
"name": "ailinttest",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "testing, pytest, unittest, ai-testing, llm, code-verification, natural-language, test-automation, code-quality, language-models",
"author": "Kosma Dunikowski",
"author_email": "kosmadunikowski@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/23/41/fab998f7fe49a61c85212f5ceb997f86a93829f9914e2bacf558812e7527/ailinttest-1.0.0.tar.gz",
"platform": null,
"description": "# AILintTest\n\nAILintTest is a Python testing library that uses Large Language Models to verify code properties through natural language assertions. It seamlessly integrates with popular testing frameworks like pytest and unittest, allowing developers to express complex code expectations in plain English while maintaining the structure of traditional test suites.\n\n## Why AILintTest?\n\nTraditional testing approaches often require extensive boilerplate code to verify complex properties. AILintTest bridges this gap by allowing developers to express sophisticated test cases in natural language, particularly useful for scenarios where writing conventional test code would be impractical or time-consuming.\n\n### Key Features\n\n1. **Natural Language Test Cases**: Write test assertions in plain English\n2. **Framework Integration**: Works with pytest, unittest, and other testing frameworks\n3. **Deterministic Results**: Uses voting mechanism and controlled sampling for consistent results\n4. **Flexible Verification**: Test complex code properties that would be difficult to verify traditionally\n\n## When to Use AILintTest\n\nAILintTest is designed for scenarios where traditional test implementation would be impractical or require excessive code. For example:\n\n```python\n# Traditional approach would require:\n# 1. Iterating through all methods\n# 2. Parsing AST for each method\n# 3. Checking exception handling patterns\n# 4. Verifying logging calls\n# 5. Maintaining complex test code\n\n# With AILintTest:\ndef test_error_handling():\n ailint.assert_code(\n \"All methods in {module} should use the custom ErrorHandler class for exception management, and log errors before re-raising them\",\n {\"module\": my_critical_module}\n )\n\n# Another example - checking documentation consistency\ndef test_docstring_completeness():\n ailint.assert_code(\n \"All public methods in {module} should have docstrings that include Parameters, Returns, and Examples sections\",\n {\"module\": my_api_module}\n )\n```\n\n## How It Works\n\n### Deterministic Testing\n\nAILintTest employs several mechanisms to ensure consistent and reliable results:\n\n1. **Voting Mechanism**: Each assertion is evaluated multiple times (configurable through `quorum_size`), and the majority result is used\n2. **Temperature Control**: Uses low temperature for LLM sampling to reduce randomness\n3. **Structured Prompts**: Converts natural language assertions into structured prompts for consistent LLM interpretation\n\n```python\n# Configure determinism settings\noptions = AILintTestOptions(\n quorum_size=5, # Number of evaluations per assertion\n)\n```\n\n## Installation\n\n```bash\npip install ailinttest\n```\n\n## Basic Usage\n\n### With pytest\n\n```python\nfrom ailinttest import AILintTest\n\ndef test_code_properties():\n ailint = AILintTest()\n \n # Test code organization\n ailint.assert_code(\n \"Classes in {module} should follow the Single Responsibility Principle\",\n {\"module\": my_module}\n )\n \n # Test security practices\n ailint.assert_code(\n \"All database queries in {module} should be parameterized to prevent SQL injection\",\n {\"module\": db_module}\n )\n```\n\n### With unittest\n\n```python\nimport unittest\nfrom ailinttest import AILintTest\n\nclass TestCodeQuality(unittest.TestCase):\n def setUp(self):\n self.ailint = AILintTest()\n \n def test_error_handling(self):\n self.ailint.assert_code(\n \"All API endpoints in {module} should have proper input validation\",\n {\"module\": api_module}\n )\n```\n\n## Advanced Usage\n\n### Custom Evaluation Options\n\n```python\nfrom ailinttest import AILintTest, AILintTestOptions\n\noptions = AILintTestOptions(\n quorum_size=7, # Increase voting sample size\n model=\"gpt-4o-2024-08-06\", # Use a more capable model\n)\n\nailint = AILintTest(options)\n```\n\n## Contributing\n\nContributions are welcome!\n\n## License\n\n[MIT License](LICENSE)\n\n## Acknowledgements\n\nThis project uses [LiteLLM](https://github.com/BerriAI/litellm) for LLM integration.\n\n---\n\nAILintTest is designed to complement, not replace, traditional testing approaches. It's most effective when used for complex code properties that are difficult to verify through conventional means.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Python testing library that uses Large Language Models to verify code properties through natural language assertions.",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/kdunee/ailinttest"
},
"split_keywords": [
"testing",
" pytest",
" unittest",
" ai-testing",
" llm",
" code-verification",
" natural-language",
" test-automation",
" code-quality",
" language-models"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f27aa167b7afba1f795d021a2b5ad1ee4e65322821b9a8cf2fc31316506c6e27",
"md5": "9d074e64dafe2fcc4f4c365778a9b379",
"sha256": "fbecce2b54114ff1ceb44831396ec844b6a0ee53f9059e51c5a425246abdc703"
},
"downloads": -1,
"filename": "ailinttest-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9d074e64dafe2fcc4f4c365778a9b379",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 7114,
"upload_time": "2024-10-25T19:44:05",
"upload_time_iso_8601": "2024-10-25T19:44:05.802155Z",
"url": "https://files.pythonhosted.org/packages/f2/7a/a167b7afba1f795d021a2b5ad1ee4e65322821b9a8cf2fc31316506c6e27/ailinttest-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2341fab998f7fe49a61c85212f5ceb997f86a93829f9914e2bacf558812e7527",
"md5": "d6d0804616db817154b76ba871b190bd",
"sha256": "38d4a27b4c9cb6db1d0438dd5ade1fb993fd8ae02023b5608f49c4a153ee6585"
},
"downloads": -1,
"filename": "ailinttest-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "d6d0804616db817154b76ba871b190bd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 5859,
"upload_time": "2024-10-25T19:44:06",
"upload_time_iso_8601": "2024-10-25T19:44:06.935728Z",
"url": "https://files.pythonhosted.org/packages/23/41/fab998f7fe49a61c85212f5ceb997f86a93829f9914e2bacf558812e7527/ailinttest-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-25 19:44:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kdunee",
"github_project": "ailinttest",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ailinttest"
}
JSON
