| Name | consistent JSON |
| Version |
0.0.10
JSON |
| download |
| home_page | |
| Summary | Make sure AI model outputs are consistent |
| upload_time | 2023-06-14 13:03:12 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2023 drorIvry Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
openai
llm
gpt
cohere
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<img align="center" src="./logo.png">
[](https://github.com/drorIvry/consisTent/actions/workflows/pre-commit.yml)
[](https://github.com/drorIvry/consisTent/actions/workflows/python-publish.yml)
consisTent is a full blows testing framework for prompts. The goal of consisTent is to create reproducible tests for LLM based applications regardless of the FM used.
## Installation
```sh
pip install consistent
```
## Concepts
consisTent comes with 2 types of validators (testers)
### syntactic validators
This type of validators is used to do static assertions of the LLM output. for example validating if the output is in a certain JSON format and assert the schema.
You can also use syntactic validators to assert something is a valid piece of code (JS or Python are supported at the moment)
**Syntactic validators are used to assert the FORM of the response**
### example
```python
consisTent.JsValidator().validate('console.log("Im a JS program!")')
consisTent.PyValidator().validate('print("Im a python program!")')
consisTent.JsonValidator().validate('{"question": "is this a valid JSON?"}')
```
### syntactic validators
This type of validators is used to assert the quality of the response with more "soft" parameters for example, check if something is factually correct, check for hallucinations, check for labels like "funny"/"interesting" etc... another type of semantic validator is the semantic consistency validator where you provide a seed of validated input and a threshold, and the test will assert the semantic distance of the new output from the seed cluster.
**Semantic validators are used to assert the CONTENT of the response**
### example
```python
import consisTent
seed = [
"the cat sat on the mat",
"the feline layed on the carpet",
]
consisTent.ConsistencyValidator(
seed_size=2,
consistency_threshold=0.5,
).validate(
seed=seed,
model_output="the dog sat on the mat",
)
```
### Label Test
```python
OPENAI_KEY = "XXXXXXXXXXXXXXX"
consisTent.LabelsValidator(openai_key=OPENAI_KEY).validate(
labels=[
"funny",
"short",
"about rabbits",
],
model_output="What do you call a rabbit that tells jokes? A funny bunny!",
)
```
### facts validation
```python
OPENAI_KEY = "XXXXXXXXXXXXXXX"
consisTent.FactsValidator(openai_key=OPENAI_KEY).validate(
facts=["this car weighs 1000KG"],
model_output="I can lift this car",
)
```
Raw data
{
"_id": null,
"home_page": "",
"name": "consistent",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "OpenAI,LLM,GPT,Cohere",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/d5/63/d00eb436be523b5a88f96316032e6f545f2802814c373079651dd85791f7/consistent-0.0.10.tar.gz",
"platform": null,
"description": "<img align=\"center\" src=\"./logo.png\">\n\n[](https://github.com/drorIvry/consisTent/actions/workflows/pre-commit.yml)\n\n[](https://github.com/drorIvry/consisTent/actions/workflows/python-publish.yml)\n\nconsisTent is a full blows testing framework for prompts. The goal of consisTent is to create reproducible tests for LLM based applications regardless of the FM used.\n\n\n## Installation\n\n```sh\npip install consistent\n```\n\n## Concepts\n\nconsisTent comes with 2 types of validators (testers)\n\n### syntactic validators\n\nThis type of validators is used to do static assertions of the LLM output. for example validating if the output is in a certain JSON format and assert the schema.\nYou can also use syntactic validators to assert something is a valid piece of code (JS or Python are supported at the moment)\n\n\n**Syntactic validators are used to assert the FORM of the response**\n\n### example\n\n```python\nconsisTent.JsValidator().validate('console.log(\"Im a JS program!\")')\n\nconsisTent.PyValidator().validate('print(\"Im a python program!\")')\n\nconsisTent.JsonValidator().validate('{\"question\": \"is this a valid JSON?\"}')\n```\n\n### syntactic validators\n\nThis type of validators is used to assert the quality of the response with more \"soft\" parameters for example, check if something is factually correct, check for hallucinations, check for labels like \"funny\"/\"interesting\" etc... another type of semantic validator is the semantic consistency validator where you provide a seed of validated input and a threshold, and the test will assert the semantic distance of the new output from the seed cluster.\n\n**Semantic validators are used to assert the CONTENT of the response**\n\n### example\n\n```python\nimport consisTent\n\nseed = [\n \"the cat sat on the mat\",\n \"the feline layed on the carpet\",\n]\n\n\nconsisTent.ConsistencyValidator(\n seed_size=2,\n consistency_threshold=0.5,\n).validate(\n seed=seed,\n model_output=\"the dog sat on the mat\",\n)\n```\n\n### Label Test\n\n```python\nOPENAI_KEY = \"XXXXXXXXXXXXXXX\"\n\nconsisTent.LabelsValidator(openai_key=OPENAI_KEY).validate(\n labels=[\n \"funny\",\n \"short\",\n \"about rabbits\",\n ],\n model_output=\"What do you call a rabbit that tells jokes? A funny bunny!\",\n)\n```\n\n### facts validation\n\n```python\nOPENAI_KEY = \"XXXXXXXXXXXXXXX\"\n\nconsisTent.FactsValidator(openai_key=OPENAI_KEY).validate(\n facts=[\"this car weighs 1000KG\"],\n model_output=\"I can lift this car\",\n)\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2023 drorIvry Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Make sure AI model outputs are consistent",
"version": "0.0.10",
"project_urls": {
"Homepage": "https://github.com/drorivry/consistent"
},
"split_keywords": [
"openai",
"llm",
"gpt",
"cohere"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bf70aa4691b4bdc59de3289e796bf0563f6a7a8907814446c8be36eaf06053ef",
"md5": "28d0a2b534bb24b040df50ee0e3a10b7",
"sha256": "97312c05b6eaa77a8fa3843929801b1f5c0ed56d34778d3aafeec6043ee2cdaa"
},
"downloads": -1,
"filename": "consistent-0.0.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "28d0a2b534bb24b040df50ee0e3a10b7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10538,
"upload_time": "2023-06-14T13:03:11",
"upload_time_iso_8601": "2023-06-14T13:03:11.151784Z",
"url": "https://files.pythonhosted.org/packages/bf/70/aa4691b4bdc59de3289e796bf0563f6a7a8907814446c8be36eaf06053ef/consistent-0.0.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d563d00eb436be523b5a88f96316032e6f545f2802814c373079651dd85791f7",
"md5": "23d0ff729eba2364c2e9516e91401735",
"sha256": "ee674a4721eb23e8aea3c65d3ae17abb9725e7c1cc127cc4747d7333e0059e78"
},
"downloads": -1,
"filename": "consistent-0.0.10.tar.gz",
"has_sig": false,
"md5_digest": "23d0ff729eba2364c2e9516e91401735",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 8487,
"upload_time": "2023-06-14T13:03:12",
"upload_time_iso_8601": "2023-06-14T13:03:12.231454Z",
"url": "https://files.pythonhosted.org/packages/d5/63/d00eb436be523b5a88f96316032e6f545f2802814c373079651dd85791f7/consistent-0.0.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-14 13:03:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "drorivry",
"github_project": "consistent",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "consistent"
}
