# arize-phoenix-evals
Phoenix provides tooling to evaluate LLM applications, including tools to determine the relevance or irrelevance of documents retrieved by retrieval-augmented generation (RAG) application, whether or not the response is toxic, and much more.
Phoenix's approach to LLM evals is notable for the following reasons:
- Includes pre-tested templates and convenience functions for a set of common Eval “tasks”
- Data science rigor applied to the testing of model and template combinations
- Designed to run as fast as possible on batches of data
- Includes benchmark datasets and tests for each eval function
## Installation
Install the arize-phoenix sub-package via `pip`
```shell
pip install arize-phoenix-evals
```
Note you will also have to install the LLM vendor SDK you would like to use with LLM Evals. For example, to use OpenAI's GPT-4, you will need to install the OpenAI Python SDK:
```shell
pip install 'openai>=1.0.0'
```
## Usage
Here is an example of running the RAG relevance eval on a dataset of Wikipedia questions and answers:
```python
import os
from phoenix.evals import (
RAG_RELEVANCY_PROMPT_TEMPLATE,
RAG_RELEVANCY_PROMPT_RAILS_MAP,
OpenAIModel,
download_benchmark_dataset,
llm_classify,
)
from sklearn.metrics import precision_recall_fscore_support, confusion_matrix, ConfusionMatrixDisplay
os.environ["OPENAI_API_KEY"] = "<your-openai-key>"
# Download the benchmark golden dataset
df = download_benchmark_dataset(
task="binary-relevance-classification", dataset_name="wiki_qa-train"
)
# Sample and re-name the columns to match the template
df = df.sample(100)
df = df.rename(
columns={
"query_text": "input",
"document_text": "reference",
},
)
model = OpenAIModel(
model="gpt-4",
temperature=0.0,
)
rails =list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values())
df[["eval_relevance"]] = llm_classify(df, model, RAG_RELEVANCY_PROMPT_TEMPLATE, rails)
#Golden dataset has True/False map to -> "irrelevant" / "relevant"
#we can then scikit compare to output of template - same format
y_true = df["relevant"].map({True: "relevant", False: "irrelevant"})
y_pred = df["eval_relevance"]
# Compute Per-Class Precision, Recall, F1 Score, Support
precision, recall, f1, support = precision_recall_fscore_support(y_true, y_pred)
```
To learn more about LLM Evals, see the [LLM Evals documentation](https://docs.arize.com/phoenix/concepts/llm-evals/).
Raw data
{
"_id": null,
"home_page": null,
"name": "arize-phoenix-evals",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.14,>=3.8",
"maintainer_email": null,
"keywords": "Explainability, Monitoring, Observability",
"author": null,
"author_email": "Arize AI <phoenix-devs@arize.com>",
"download_url": "https://files.pythonhosted.org/packages/33/e3/8693a1fc84c1520c3fc5e56a75cc7642d7e04e4cc5716285618433d1d77c/arize_phoenix_evals-0.17.4.tar.gz",
"platform": null,
"description": "# arize-phoenix-evals\n\nPhoenix provides tooling to evaluate LLM applications, including tools to determine the relevance or irrelevance of documents retrieved by retrieval-augmented generation (RAG) application, whether or not the response is toxic, and much more.\n\nPhoenix's approach to LLM evals is notable for the following reasons:\n\n- Includes pre-tested templates and convenience functions for a set of common Eval \u201ctasks\u201d\n- Data science rigor applied to the testing of model and template combinations\n- Designed to run as fast as possible on batches of data\n- Includes benchmark datasets and tests for each eval function\n\n## Installation\n\nInstall the arize-phoenix sub-package via `pip`\n\n```shell\npip install arize-phoenix-evals\n```\n\nNote you will also have to install the LLM vendor SDK you would like to use with LLM Evals. For example, to use OpenAI's GPT-4, you will need to install the OpenAI Python SDK:\n\n```shell\npip install 'openai>=1.0.0'\n```\n\n## Usage\n\nHere is an example of running the RAG relevance eval on a dataset of Wikipedia questions and answers:\n\n```python\nimport os\nfrom phoenix.evals import (\n RAG_RELEVANCY_PROMPT_TEMPLATE,\n RAG_RELEVANCY_PROMPT_RAILS_MAP,\n OpenAIModel,\n download_benchmark_dataset,\n llm_classify,\n)\nfrom sklearn.metrics import precision_recall_fscore_support, confusion_matrix, ConfusionMatrixDisplay\n\nos.environ[\"OPENAI_API_KEY\"] = \"<your-openai-key>\"\n\n# Download the benchmark golden dataset\ndf = download_benchmark_dataset(\n task=\"binary-relevance-classification\", dataset_name=\"wiki_qa-train\"\n)\n# Sample and re-name the columns to match the template\ndf = df.sample(100)\ndf = df.rename(\n columns={\n \"query_text\": \"input\",\n \"document_text\": \"reference\",\n },\n)\nmodel = OpenAIModel(\n model=\"gpt-4\",\n temperature=0.0,\n)\n\n\nrails =list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values())\ndf[[\"eval_relevance\"]] = llm_classify(df, model, RAG_RELEVANCY_PROMPT_TEMPLATE, rails)\n#Golden dataset has True/False map to -> \"irrelevant\" / \"relevant\"\n#we can then scikit compare to output of template - same format\ny_true = df[\"relevant\"].map({True: \"relevant\", False: \"irrelevant\"})\ny_pred = df[\"eval_relevance\"]\n\n# Compute Per-Class Precision, Recall, F1 Score, Support\nprecision, recall, f1, support = precision_recall_fscore_support(y_true, y_pred)\n```\n\nTo learn more about LLM Evals, see the [LLM Evals documentation](https://docs.arize.com/phoenix/concepts/llm-evals/).\n",
"bugtrack_url": null,
"license": "Elastic-2.0",
"summary": "LLM Evaluations",
"version": "0.17.4",
"project_urls": {
"Documentation": "https://docs.arize.com/phoenix/",
"Issues": "https://github.com/Arize-ai/phoenix/issues",
"Source": "https://github.com/Arize-ai/phoenix"
},
"split_keywords": [
"explainability",
" monitoring",
" observability"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b388b0879d492ca2e13b02b1e535190cfe3c03674297e9151937830d5578d92c",
"md5": "10804bac9421c481a99f371e58e394cd",
"sha256": "c24ca7478aa68077a16d5350c222ae947cd9aabafd0d69844a6b8e42b7259d6d"
},
"downloads": -1,
"filename": "arize_phoenix_evals-0.17.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "10804bac9421c481a99f371e58e394cd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.14,>=3.8",
"size": 57055,
"upload_time": "2024-11-12T20:09:57",
"upload_time_iso_8601": "2024-11-12T20:09:57.578827Z",
"url": "https://files.pythonhosted.org/packages/b3/88/b0879d492ca2e13b02b1e535190cfe3c03674297e9151937830d5578d92c/arize_phoenix_evals-0.17.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "33e38693a1fc84c1520c3fc5e56a75cc7642d7e04e4cc5716285618433d1d77c",
"md5": "1e332034390d81ad2efdb636f534c5e3",
"sha256": "c04fc759c2669110f5a5152dd63af50a1e43280a64e444f308d178e354b08c0f"
},
"downloads": -1,
"filename": "arize_phoenix_evals-0.17.4.tar.gz",
"has_sig": false,
"md5_digest": "1e332034390d81ad2efdb636f534c5e3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.14,>=3.8",
"size": 43510,
"upload_time": "2024-11-12T20:09:59",
"upload_time_iso_8601": "2024-11-12T20:09:59.317740Z",
"url": "https://files.pythonhosted.org/packages/33/e3/8693a1fc84c1520c3fc5e56a75cc7642d7e04e4cc5716285618433d1d77c/arize_phoenix_evals-0.17.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-12 20:09:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Arize-ai",
"github_project": "phoenix",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "arize-phoenix-evals"
}
JSON
