quotientai


Namequotientai JSON
Version 0.1.9 PyPI version JSON
download
home_pageNone
SummaryCLI for evaluating large language models with Quotient
upload_time2025-02-25 18:40:21
maintainerNone
docs_urlNone
authorFreddie Vargus
requires_python<4.0,>=3.9
licenseApache-2.0
keywords quotient evaluation llms machine learning ai
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # quotientai
[![PyPI version](https://img.shields.io/pypi/v/quotientai)](https://pypi.org/project/quotientai)

## Overview

`quotientai` is an SDK and CLI built to manage artifacts (prompts, datasets), and run evaluations on [Quotient](https://quotientai.co).

## Installation

```console
pip install quotientai
```

## Usage

Create an API key on Quotient and set it as an environment variable called `QUOTIENT_API_KEY`. Then follow the examples below or see our [docs](https://docs.quotientai.co) for a more comprehensive walkthrough.

### Examples

**Create a prompt:**

```python
from quotientai import QuotientAI

quotient = QuotientAI()

new_prompt = quotient.prompts.create(
    name="customer-support-inquiry"
    system_prompt="You are a helpful assistant.",
    user_prompt="How can I assist you today?"
)

print(new_prompt)
```

**Create a dataset:**

```python
from quotientai import QuotientAI

quotient = QuotientAI()

new_dataset = quotient.datasets.create(
    name="my-sample-dataset"
    description="My first dataset",
    rows=[
        {"input": "Sample input", "expected": "Sample output"},
        {"input": "Another input", "expected": "Another output"}
    ]
)

print(new_dataset)
```

**Create a log with hallucination detection:**
Log an event with hallucination detection. This will create a log event in Quotient and perform hallucination detection on the model output, input, and documents. This is a fire and forget operation, so it will not block the execution of your code.

Additional examples can be found in the [examples](examples) directory.

```python
from quotientai import QuotientAI

quotient = QuotientAI()
quotient_logger = quotient.logger.init(
    # Required
    app_name="my-app",
    environment="dev",
    # dynamic labels for slicing/dicing analytics e.g. by customer, feature, etc
    tags={"model": "gpt-4o", "feature": "customer-support"},
    hallucination_detection=True,
    inconsistency_detection=True,
)

# Mock retrieved documents
retrieved_documents = [{"page_content": "Sample document"}]

response = quotient_logger.log(
    user_query="Sample input",
    model_output="Sample output",
    # Page content from Documents from your retriever used to generate the model output
    documents=[doc["page_content"] for doc in retrieved_documents],
    # Message history from your chat history
    message_history=[
        {"role": "system", "content": "You are an expert on geography."},
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "The capital of France is Paris"},
    ],
    # Instructions for the model to follow
    instructions=[
        "You are a helpful assistant that answers questions about the world.",
        "Answer the question in a concise manner. If you are not sure, say 'I don't know'.",
    ],
    # Tags can be overridden at log time
    tags={"model": "gpt-4o-mini", "feature": "customer-support"},
)

print(response)
```

You can also use the async client if you need to create logs asynchronously.

```python
from quotientai import AsyncQuotientAI
import asyncio

quotient = AsyncQuotientAI()

quotient_logger = quotient.logger.init(
    # Required
    app_name="my-app",
    environment="dev",
    # dynamic labels for slicing/dicing analytics e.g. by customer, feature, etc
    tags={"model": "gpt-4o", "feature": "customer-support"},
    hallucination_detection=True,
    inconsistency_detection=True,
)


async def main():
    # Mock retrieved documents
    retrieved_documents = [{"page_content": "Sample document"}]

    response = await quotient_logger.log(
        user_query="Sample input",
        model_output="Sample output",
        # Page content from Documents from your retriever used to generate the model output
        documents=[doc["page_content"] for doc in retrieved_documents],
        # Message history from your chat history
        message_history=[
            {"role": "system", "content": "You are an expert on geography."},
            {"role": "user", "content": "What is the capital of France?"},
            {"role": "assistant", "content": "The capital of France is Paris"},
        ],
        # Instructions for the model to follow
        instructions=[
            "You are a helpful assistant that answers questions about the world.",
            "Answer the question in a concise manner. If you are not sure, say 'I don't know'.",
        ],
        # Tags can be overridden at log time
        tags={"model": "gpt-4o-mini", "feature": "customer-support"},
    )

    print(response)


# Run the async function
asyncio.run(main())
```


            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "quotientai",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.9",
    "maintainer_email": null,
    "keywords": "quotient, evaluation, llms, machine learning, ai",
    "author": "Freddie Vargus",
    "author_email": "freddie@quotientai.co",
    "download_url": "https://files.pythonhosted.org/packages/12/b0/2d9d8bf7aa9d74867b260141997e46acfe68091de00858e3c4f061a7a222/quotientai-0.1.9.tar.gz",
    "platform": null,
    "description": "# quotientai\n[![PyPI version](https://img.shields.io/pypi/v/quotientai)](https://pypi.org/project/quotientai)\n\n## Overview\n\n`quotientai` is an SDK and CLI built to manage artifacts (prompts, datasets), and run evaluations on [Quotient](https://quotientai.co).\n\n## Installation\n\n```console\npip install quotientai\n```\n\n## Usage\n\nCreate an API key on Quotient and set it as an environment variable called `QUOTIENT_API_KEY`. Then follow the examples below or see our [docs](https://docs.quotientai.co) for a more comprehensive walkthrough.\n\n### Examples\n\n**Create a prompt:**\n\n```python\nfrom quotientai import QuotientAI\n\nquotient = QuotientAI()\n\nnew_prompt = quotient.prompts.create(\n    name=\"customer-support-inquiry\"\n    system_prompt=\"You are a helpful assistant.\",\n    user_prompt=\"How can I assist you today?\"\n)\n\nprint(new_prompt)\n```\n\n**Create a dataset:**\n\n```python\nfrom quotientai import QuotientAI\n\nquotient = QuotientAI()\n\nnew_dataset = quotient.datasets.create(\n    name=\"my-sample-dataset\"\n    description=\"My first dataset\",\n    rows=[\n        {\"input\": \"Sample input\", \"expected\": \"Sample output\"},\n        {\"input\": \"Another input\", \"expected\": \"Another output\"}\n    ]\n)\n\nprint(new_dataset)\n```\n\n**Create a log with hallucination detection:**\nLog an event with hallucination detection. This will create a log event in Quotient and perform hallucination detection on the model output, input, and documents. This is a fire and forget operation, so it will not block the execution of your code.\n\nAdditional examples can be found in the [examples](examples) directory.\n\n```python\nfrom quotientai import QuotientAI\n\nquotient = QuotientAI()\nquotient_logger = quotient.logger.init(\n    # Required\n    app_name=\"my-app\",\n    environment=\"dev\",\n    # dynamic labels for slicing/dicing analytics e.g. by customer, feature, etc\n    tags={\"model\": \"gpt-4o\", \"feature\": \"customer-support\"},\n    hallucination_detection=True,\n    inconsistency_detection=True,\n)\n\n# Mock retrieved documents\nretrieved_documents = [{\"page_content\": \"Sample document\"}]\n\nresponse = quotient_logger.log(\n    user_query=\"Sample input\",\n    model_output=\"Sample output\",\n    # Page content from Documents from your retriever used to generate the model output\n    documents=[doc[\"page_content\"] for doc in retrieved_documents],\n    # Message history from your chat history\n    message_history=[\n        {\"role\": \"system\", \"content\": \"You are an expert on geography.\"},\n        {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n        {\"role\": \"assistant\", \"content\": \"The capital of France is Paris\"},\n    ],\n    # Instructions for the model to follow\n    instructions=[\n        \"You are a helpful assistant that answers questions about the world.\",\n        \"Answer the question in a concise manner. If you are not sure, say 'I don't know'.\",\n    ],\n    # Tags can be overridden at log time\n    tags={\"model\": \"gpt-4o-mini\", \"feature\": \"customer-support\"},\n)\n\nprint(response)\n```\n\nYou can also use the async client if you need to create logs asynchronously.\n\n```python\nfrom quotientai import AsyncQuotientAI\nimport asyncio\n\nquotient = AsyncQuotientAI()\n\nquotient_logger = quotient.logger.init(\n    # Required\n    app_name=\"my-app\",\n    environment=\"dev\",\n    # dynamic labels for slicing/dicing analytics e.g. by customer, feature, etc\n    tags={\"model\": \"gpt-4o\", \"feature\": \"customer-support\"},\n    hallucination_detection=True,\n    inconsistency_detection=True,\n)\n\n\nasync def main():\n    # Mock retrieved documents\n    retrieved_documents = [{\"page_content\": \"Sample document\"}]\n\n    response = await quotient_logger.log(\n        user_query=\"Sample input\",\n        model_output=\"Sample output\",\n        # Page content from Documents from your retriever used to generate the model output\n        documents=[doc[\"page_content\"] for doc in retrieved_documents],\n        # Message history from your chat history\n        message_history=[\n            {\"role\": \"system\", \"content\": \"You are an expert on geography.\"},\n            {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n            {\"role\": \"assistant\", \"content\": \"The capital of France is Paris\"},\n        ],\n        # Instructions for the model to follow\n        instructions=[\n            \"You are a helpful assistant that answers questions about the world.\",\n            \"Answer the question in a concise manner. If you are not sure, say 'I don't know'.\",\n        ],\n        # Tags can be overridden at log time\n        tags={\"model\": \"gpt-4o-mini\", \"feature\": \"customer-support\"},\n    )\n\n    print(response)\n\n\n# Run the async function\nasyncio.run(main())\n```\n\n",
    "bugtrack_url": null,
    "license": "Apache-2.0",
    "summary": "CLI for evaluating large language models with Quotient",
    "version": "0.1.9",
    "project_urls": null,
    "split_keywords": [
        "quotient",
        " evaluation",
        " llms",
        " machine learning",
        " ai"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "9b5404f5e052886718a6e5301c86700d717fceaf36bb5485ee10fc161c880b7a",
                "md5": "fbbe029622dcd89523e8a4bd189a40fb",
                "sha256": "5bebdef8eaa18738b2f5eb583ec9b71fb0df93321b0e1fa3eb6f5fd423dcc13f"
            },
            "downloads": -1,
            "filename": "quotientai-0.1.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "fbbe029622dcd89523e8a4bd189a40fb",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.9",
            "size": 27024,
            "upload_time": "2025-02-25T18:40:19",
            "upload_time_iso_8601": "2025-02-25T18:40:19.867473Z",
            "url": "https://files.pythonhosted.org/packages/9b/54/04f5e052886718a6e5301c86700d717fceaf36bb5485ee10fc161c880b7a/quotientai-0.1.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "12b02d9d8bf7aa9d74867b260141997e46acfe68091de00858e3c4f061a7a222",
                "md5": "2457109678869dd2d02a736be6f8f4e5",
                "sha256": "29fe723cbf4145e0d0eb62ab6f71979e78dfb53ee2e44ccf4fce65d561a311de"
            },
            "downloads": -1,
            "filename": "quotientai-0.1.9.tar.gz",
            "has_sig": false,
            "md5_digest": "2457109678869dd2d02a736be6f8f4e5",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.9",
            "size": 21058,
            "upload_time": "2025-02-25T18:40:21",
            "upload_time_iso_8601": "2025-02-25T18:40:21.063674Z",
            "url": "https://files.pythonhosted.org/packages/12/b0/2d9d8bf7aa9d74867b260141997e46acfe68091de00858e3c4f061a7a222/quotientai-0.1.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-02-25 18:40:21",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "quotientai"
}
        
Elapsed time: 1.20165s