# Azure AI Evaluation client library for Python
We are excited to introduce the public preview of the Azure AI Evaluation SDK.
[Source code][source_code]
| [Package (PyPI)][evaluation_pypi]
| [API reference documentation][evaluation_ref_docs]
| [Product documentation][product_documentation]
| [Samples][evaluation_samples]
This package has been tested with Python 3.8, 3.9, 3.10, 3.11, and 3.12.
For a more complete set of Azure libraries, see https://aka.ms/azsdk/python/all
## Getting started
### Prerequisites
- Python 3.8 or later is required to use this package.
### Install the package
Install the Azure AI Evaluation library for Python with [pip][pip_link]::
```bash
pip install azure-ai-evaluation
```
## Key concepts
Evaluators are custom or prebuilt classes or functions that are designed to measure the quality of the outputs from language models.
## Examples
### Evaluators
Users can create evaluator runs on the local machine as shown in the example below:
```python
import os
from pprint import pprint
from azure.ai.evaluation import evaluate, RelevanceEvaluator, ViolenceEvaluator
def response_length(response, **kwargs):
return {"value": len(response)}
if __name__ == "__main__":
# Built-in evaluators
# Initialize Azure OpenAI Model Configuration
model_config = {
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
"api_key": os.environ.get("AZURE_OPENAI_KEY"),
"azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
}
# Initialzing Relevance Evaluator
relevance_eval = RelevanceEvaluator(model_config)
# Running Relevance Evaluator on single input row
relevance_score = relevance_eval(
response="The Alpine Explorer Tent is the most waterproof.",
query="Which tent is the most waterproof?",
)
pprint(relevance_score)
# {'gpt_relevance': 5.0}
# Content Safety Evaluator
# Initialize Project Scope
azure_ai_project = {
"subscription_id": <subscription_id>,
"resource_group_name": <resource_group_name>,
"project_name": <project_name>
}
violence_eval = ViolenceEvaluator(azure_ai_project)
violence_score = violence_eval(query="What is the capital of France?", response="Paris.")
pprint(violence_score)
# {'violence': 'Very low',
# 'violence_reason': "The system's response is a straightforward factual response "
# 'to a geography question. There is no violent content or '
# 'language present.',
# 'violence_score': 0}
# Code based evaluator
response_length("The Alpine Explorer Tent is the most waterproof.")
# {'value': 48}
# Using multiple evaluators together using `Evaluate` API
result = evaluate(
data="evaluate_test_data.jsonl",
evaluators={
"response_length": response_length,
"violence": violence_eval,
},
)
pprint(result)
```
### Simulator
Simulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes
their AI application.
#### Simulating with a Prompty
```yaml
---
name: ApplicationPrompty
description: Simulates an application
model:
api: chat
parameters:
temperature: 0.0
top_p: 1.0
presence_penalty: 0
frequency_penalty: 0
response_format:
type: text
inputs:
conversation_history:
type: dict
---
system:
You are a helpful assistant and you're helping with the user's query. Keep the conversation engaging and interesting.
Output with a string that continues the conversation, responding to the latest message from the user, given the conversation history:
{{ conversation_history }}
```
Query Response generaing prompty for gpt-4o with `json_schema` support
Use this file as an override.
```yaml
---
name: TaskSimulatorQueryResponseGPT4o
description: Gets queries and responses from a blob of text
model:
api: chat
parameters:
temperature: 0.0
top_p: 1.0
presence_penalty: 0
frequency_penalty: 0
response_format:
type: json_schema
json_schema:
name: QRJsonSchema
schema:
type: object
properties:
items:
type: array
items:
type: object
properties:
q:
type: string
r:
type: string
required:
- q
- r
inputs:
text:
type: string
num_queries:
type: integer
---
system:
You're an AI that helps in preparing a Question/Answer quiz from Text for "Who wants to be a millionaire" tv show
Both Questions and Answers MUST BE extracted from given Text
Frame Question in a way so that Answer is RELEVANT SHORT BITE-SIZED info from Text
RELEVANT info could be: NUMBER, DATE, STATISTIC, MONEY, NAME
A sentence should contribute multiple QnAs if it has more info in it
Answer must not be more than 5 words
Answer must be picked from Text as is
Question should be as descriptive as possible and must include as much context as possible from Text
Output must always have the provided number of QnAs
Output must be in JSON format.
Output must have {{num_queries}} objects in the format specified below. Any other count is unacceptable.
Text:
<|text_start|>
On January 24, 1984, former Apple CEO Steve Jobs introduced the first Macintosh. In late 2003, Apple had 2.06 percent of the desktop share in the United States.
Some years later, research firms IDC and Gartner reported that Apple's market share in the U.S. had increased to about 6%.
<|text_end|>
Output with 5 QnAs:
{
"qna": [{
"q": "When did the former Apple CEO Steve Jobs introduced the first Macintosh?",
"r": "January 24, 1984"
},
{
"q": "Who was the former Apple CEO that introduced the first Macintosh on January 24, 1984?",
"r": "Steve Jobs"
},
{
"q": "What percent of the desktop share did Apple have in the United States in late 2003?",
"r": "2.06 percent"
},
{
"q": "What were the research firms that reported on Apple's market share in the U.S.?",
"r": "IDC and Gartner"
},
{
"q": "What was the percentage increase of Apple's market share in the U.S., as reported by research firms IDC and Gartner?",
"r": "6%"
}]
}
Text:
<|text_start|>
{{ text }}
<|text_end|>
Output with {{ num_queries }} QnAs:
```
Application code:
```python
import json
import asyncio
from typing import Any, Dict, List, Optional
from azure.ai.evaluation.simulator import Simulator
from promptflow.client import load_flow
import os
import wikipedia
# Set up the model configuration without api_key, using DefaultAzureCredential
model_config = {
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
"azure_deployment": os.environ.get("AZURE_DEPLOYMENT"),
# not providing key would make the SDK pick up `DefaultAzureCredential`
# use "api_key": "<your API key>"
"api_version": "2024-08-01-preview" # keep this for gpt-4o
}
# Use Wikipedia to get some text for the simulation
wiki_search_term = "Leonardo da Vinci"
wiki_title = wikipedia.search(wiki_search_term)[0]
wiki_page = wikipedia.page(wiki_title)
text = wiki_page.summary[:1000]
def method_to_invoke_application_prompty(query: str, messages_list: List[Dict], context: Optional[Dict]):
try:
current_dir = os.path.dirname(__file__)
prompty_path = os.path.join(current_dir, "application.prompty")
_flow = load_flow(
source=prompty_path,
model=model_config,
credential=DefaultAzureCredential()
)
response = _flow(
query=query,
context=context,
conversation_history=messages_list
)
return response
except Exception as e:
print(f"Something went wrong invoking the prompty: {e}")
return "something went wrong"
async def callback(
messages: Dict[str, List[Dict]],
stream: bool = False,
session_state: Any = None, # noqa: ANN401
context: Optional[Dict[str, Any]] = None,
) -> dict:
messages_list = messages["messages"]
# Get the last message from the user
latest_message = messages_list[-1]
query = latest_message["content"]
# Call your endpoint or AI application here
response = method_to_invoke_application_prompty(query, messages_list, context)
# Format the response to follow the OpenAI chat protocol format
formatted_response = {
"content": response,
"role": "assistant",
"context": "",
}
messages["messages"].append(formatted_response)
return {"messages": messages["messages"], "stream": stream, "session_state": session_state, "context": context}
async def main():
simulator = Simulator(model_config=model_config)
current_dir = os.path.dirname(__file__)
query_response_override_for_latest_gpt_4o = os.path.join(current_dir, "TaskSimulatorQueryResponseGPT4o.prompty")
outputs = await simulator(
target=callback,
text=text,
query_response_generating_prompty=query_response_override_for_latest_gpt_4o, # use this only with latest gpt-4o
num_queries=2,
max_conversation_turns=1,
user_persona=[
f"I am a student and I want to learn more about {wiki_search_term}",
f"I am a teacher and I want to teach my students about {wiki_search_term}"
],
)
print(json.dumps(outputs, indent=2))
if __name__ == "__main__":
# Ensure that the following environment variables are set in your environment:
# AZURE_OPENAI_ENDPOINT and AZURE_DEPLOYMENT
# Example:
# os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-endpoint.openai.azure.com/"
# os.environ["AZURE_DEPLOYMENT"] = "your-deployment-name"
asyncio.run(main())
print("done!")
```
#### Adversarial Simulator
```python
from azure.ai.evaluation.simulator import AdversarialSimulator, AdversarialScenario
from azure.identity import DefaultAzureCredential
from typing import Any, Dict, List, Optional
import asyncio
azure_ai_project = {
"subscription_id": <subscription_id>,
"resource_group_name": <resource_group_name>,
"project_name": <project_name>
}
async def callback(
messages: List[Dict],
stream: bool = False,
session_state: Any = None,
context: Dict[str, Any] = None
) -> dict:
messages_list = messages["messages"]
# get last message
latest_message = messages_list[-1]
query = latest_message["content"]
context = None
if 'file_content' in messages["template_parameters"]:
query += messages["template_parameters"]['file_content']
# the next few lines explains how to use the AsyncAzureOpenAI's chat.completions
# to respond to the simulator. You should replace it with a call to your model/endpoint/application
# make sure you pass the `query` and format the response as we have shown below
from openai import AsyncAzureOpenAI
oai_client = AsyncAzureOpenAI(
api_key=<api_key>,
azure_endpoint=<endpoint>,
api_version="2023-12-01-preview",
)
try:
response_from_oai_chat_completions = await oai_client.chat.completions.create(messages=[{"content": query, "role": "user"}], model="gpt-4", max_tokens=300)
except Exception as e:
print(f"Error: {e}")
# to continue the conversation, return the messages, else you can fail the adversarial with an exception
message = {
"content": "Something went wrong. Check the exception e for more details.",
"role": "assistant",
"context": None,
}
messages["messages"].append(message)
return {
"messages": messages["messages"],
"stream": stream,
"session_state": session_state
}
response_result = response_from_oai_chat_completions.choices[0].message.content
formatted_response = {
"content": response_result,
"role": "assistant",
"context": {},
}
messages["messages"].append(formatted_response)
return {
"messages": messages["messages"],
"stream": stream,
"session_state": session_state,
"context": context
}
```
#### Adversarial QA
```python
scenario = AdversarialScenario.ADVERSARIAL_QA
simulator = AdversarialSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())
outputs = asyncio.run(
simulator(
scenario=scenario,
max_conversation_turns=1,
max_simulation_results=3,
target=callback
)
)
print(outputs.to_eval_qa_json_lines())
```
#### Direct Attack Simulator
```python
scenario = AdversarialScenario.ADVERSARIAL_QA
simulator = DirectAttackSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())
outputs = asyncio.run(
simulator(
scenario=scenario,
max_conversation_turns=1,
max_simulation_results=2,
target=callback
)
)
print(outputs)
```
## Troubleshooting
### General
Azure ML clients raise exceptions defined in [Azure Core][azure_core_readme].
### Logging
This library uses the standard
[logging][python_logging] library for logging.
Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
level.
Detailed DEBUG level logging, including request/response bodies and unredacted
headers, can be enabled on a client with the `logging_enable` argument.
See full SDK logging documentation with examples [here][sdk_logging_docs].
## Next steps
- View our [samples][evaluation_samples].
- View our [documentation][product_documentation]
## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.
<!-- LINKS -->
[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/evaluation/azure-ai-evaluation
[evaluation_pypi]: https://pypi.org/project/azure-ai-evaluation/
[evaluation_ref_docs]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview
[evaluation_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios
[product_documentation]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk
[python_logging]: https://docs.python.org/3/library/logging.html
[sdk_logging_docs]: https://docs.microsoft.com/azure/developer/python/azure-sdk-logging
[azure_core_readme]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
[pip_link]: https://pypi.org/project/pip/
[azure_core_ref_docs]: https://aka.ms/azsdk-python-core-policies
[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
[azure_identity]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity
[cla]: https://cla.microsoft.com
[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:opencode@microsoft.com
# Release History
## 1.0.0b5 (2024-10-28)
### Features Added
- Added `GroundednessProEvaluator`, which is a service-based evaluator for determining response groundedness.
- Groundedness detection in Non Adversarial Simulator via query/context pairs
```python
import importlib.resources as pkg_resources
package = "azure.ai.evaluation.simulator._data_sources"
resource_name = "grounding.json"
custom_simulator = Simulator(model_config=model_config)
conversation_turns = []
with pkg_resources.path(package, resource_name) as grounding_file:
with open(grounding_file, "r") as file:
data = json.load(file)
for item in data:
conversation_turns.append([item])
outputs = asyncio.run(custom_simulator(
target=callback,
conversation_turns=conversation_turns,
max_conversation_turns=1,
))
```
- Adding evaluator for multimodal use cases
### Breaking Changes
- Renamed environment variable `PF_EVALS_BATCH_USE_ASYNC` to `AI_EVALS_BATCH_USE_ASYNC`.
- `RetrievalEvaluator` now requires a `context` input in addition to `query` in single-turn evaluation.
- `RelevanceEvaluator` no longer takes `context` as an input. It now only takes `query` and `response` in single-turn evaluation.
- `FluencyEvaluator` no longer takes `query` as an input. It now only takes `response` in single-turn evaluation.
- AdversarialScenario enum does not include `ADVERSARIAL_INDIRECT_JAILBREAK`, invoking IndirectJailbreak or XPIA should be done with `IndirectAttackSimulator`
- Outputs of `Simulator` and `AdversarialSimulator` previously had `to_eval_qa_json_lines` and now has `to_eval_qr_json_lines`. Where `to_eval_qa_json_lines` had:
```json
{"question": <user_message>, "answer": <assistant_message>}
```
`to_eval_qr_json_lines` now has:
```json
{"query": <user_message>, "response": assistant_message}
```
### Bugs Fixed
- Non adversarial simulator works with `gpt-4o` models using the `json_schema` response format
- Fixed an issue where the `evaluate` API would fail with "[WinError 32] The process cannot access the file because it is being used by another process" when venv folder and target function file are in the same directory.
- Fix evaluate API failure when `trace.destination` is set to `none`
- Non adversarial simulator now accepts context from the callback
### Other Changes
- Improved error messages for the `evaluate` API by enhancing the validation of input parameters. This update provides more detailed and actionable error descriptions.
- `GroundednessEvaluator` now supports `query` as an optional input in single-turn evaluation. If `query` is provided, a different prompt template will be used for the evaluation.
- To align with our support of a diverse set of models, the following evaluators will now have a new key in their result output without the `gpt_` prefix. To maintain backwards compatibility, the old key with the `gpt_` prefix will still be present in the output; however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.
- `CoherenceEvaluator`
- `RelevanceEvaluator`
- `FluencyEvaluator`
- `GroundednessEvaluator`
- `SimilarityEvaluator`
- `RetrievalEvaluator`
- The following evaluators will now have a new key in their result output including LLM reasoning behind the score. The new key will follow the pattern "<metric_name>_reason". The reasoning is the result of a more detailed prompt template being used to generate the LLM response. Note that this requires the maximum number of tokens used to run these evaluators to be increased.
| Evaluator | New Token Limit |
| --- | --- |
| `CoherenceEvaluator` | 800 |
| `RelevanceEvaluator` | 800 |
| `FluencyEvaluator` | 800 |
| `GroundednessEvaluator` | 800 |
| `RetrievalEvaluator` | 1600 |
- Improved the error message for storage access permission issues to provide clearer guidance for users.
## 1.0.0b4 (2024-10-16)
### Breaking Changes
- Removed `numpy` dependency. All NaN values returned by the SDK have been changed to from `numpy.nan` to `math.nan`.
- `credential` is now required to be passed in for all content safety evaluators and `ProtectedMaterialsEvaluator`. `DefaultAzureCredential` will no longer be chosen if a credential is not passed.
- Changed package extra name from "pf-azure" to "remote".
### Bugs Fixed
- Adversarial Conversation simulations would fail with `Forbidden`. Added logic to re-fetch token in the exponential retry logic to retrive RAI Service response.
- Fixed an issue where the Evaluate API did not fail due to missing inputs when the target did not return columns required by the evaluators.
### Other Changes
- Enhance the error message to provide clearer instruction when required packages for the remote tracking feature are missing.
- Print the per-evaluator run summary at the end of the Evaluate API call to make troubleshooting row-level failures easier.
## 1.0.0b3 (2024-10-01)
### Features Added
- Added `type` field to `AzureOpenAIModelConfiguration` and `OpenAIModelConfiguration`
- The following evaluators now support `conversation` as an alternative input to their usual single-turn inputs:
- `ViolenceEvaluator`
- `SexualEvaluator`
- `SelfHarmEvaluator`
- `HateUnfairnessEvaluator`
- `ProtectedMaterialEvaluator`
- `IndirectAttackEvaluator`
- `CoherenceEvaluator`
- `RelevanceEvaluator`
- `FluencyEvaluator`
- `GroundednessEvaluator`
- Surfaced `RetrievalScoreEvaluator`, formally an internal part of `ChatEvaluator` as a standalone conversation-only evaluator.
### Breaking Changes
- Removed `ContentSafetyChatEvaluator` and `ChatEvaluator`
- The `evaluator_config` parameter of `evaluate` now maps in evaluator name to a dictionary `EvaluatorConfig`, which is a `TypedDict`. The
`column_mapping` between `data` or `target` and evaluator field names should now be specified inside this new dictionary:
Before:
```python
evaluate(
...,
evaluator_config={
"hate_unfairness": {
"query": "${data.question}",
"response": "${data.answer}",
}
},
...
)
```
After
```python
evaluate(
...,
evaluator_config={
"hate_unfairness": {
"column_mapping": {
"query": "${data.question}",
"response": "${data.answer}",
}
}
},
...
)
```
- Simulator now requires a model configuration to call the prompty instead of an Azure AI project scope. This enables the usage of simulator with Entra ID based auth.
Before:
```python
azure_ai_project = {
"subscription_id": os.environ.get("AZURE_SUBSCRIPTION_ID"),
"resource_group_name": os.environ.get("RESOURCE_GROUP"),
"project_name": os.environ.get("PROJECT_NAME"),
}
sim = Simulator(azure_ai_project=azure_ai_project, credentails=DefaultAzureCredentials())
```
After:
```python
model_config = {
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
"azure_deployment": os.environ.get("AZURE_DEPLOYMENT"),
}
sim = Simulator(model_config=model_config)
```
If `api_key` is not included in the `model_config`, the prompty runtime in `promptflow-core` will pick up `DefaultAzureCredential`.
### Bugs Fixed
- Fixed issue where Entra ID authentication was not working with `AzureOpenAIModelConfiguration`
## 1.0.0b2 (2024-09-24)
### Breaking Changes
- `data` and `evaluators` are now required keywords in `evaluate`.
## 1.0.0b1 (2024-09-20)
### Breaking Changes
- The `synthetic` namespace has been renamed to `simulator`, and sub-namespaces under this module have been removed
- The `evaluate` and `evaluators` namespaces have been removed, and everything previously exposed in those modules has been added to the root namespace `azure.ai.evaluation`
- The parameter name `project_scope` in content safety evaluators have been renamed to `azure_ai_project` for consistency with evaluate API and simulators.
- Model configurations classes are now of type `TypedDict` and are exposed in the `azure.ai.evaluation` module instead of coming from `promptflow.core`.
- Updated the parameter names for `question` and `answer` in built-in evaluators to more generic terms: `query` and `response`.
### Features Added
- First preview
- This package is port of `promptflow-evals`. New features will be added only to this package moving forward.
- Added a `TypedDict` for `AzureAIProject` that allows for better intellisense and type checking when passing in project information
Raw data
{
"_id": null,
"home_page": "https://github.com/Azure/azure-sdk-for-python",
"name": "azure-ai-evaluation",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "azure, azure sdk",
"author": "Microsoft Corporation",
"author_email": "azuresdkengsysadmins@microsoft.com",
"download_url": "https://files.pythonhosted.org/packages/4a/0c/28e0eecc52e771b0ba1f635fe907d1b03cbc6f36feff948e3b60af37056b/azure_ai_evaluation-1.0.0b5.tar.gz",
"platform": null,
"description": "# Azure AI Evaluation client library for Python\n\nWe are excited to introduce the public preview of the Azure AI Evaluation SDK. \n\n[Source code][source_code]\n| [Package (PyPI)][evaluation_pypi]\n| [API reference documentation][evaluation_ref_docs]\n| [Product documentation][product_documentation]\n| [Samples][evaluation_samples]\n\nThis package has been tested with Python 3.8, 3.9, 3.10, 3.11, and 3.12.\n\nFor a more complete set of Azure libraries, see https://aka.ms/azsdk/python/all\n\n## Getting started\n\n### Prerequisites\n\n- Python 3.8 or later is required to use this package.\n\n### Install the package\n\nInstall the Azure AI Evaluation library for Python with [pip][pip_link]::\n\n```bash\npip install azure-ai-evaluation\n```\n\n## Key concepts\n\nEvaluators are custom or prebuilt classes or functions that are designed to measure the quality of the outputs from language models.\n\n## Examples\n\n### Evaluators\n\nUsers can create evaluator runs on the local machine as shown in the example below:\n\n```python\nimport os\nfrom pprint import pprint\n\nfrom azure.ai.evaluation import evaluate, RelevanceEvaluator, ViolenceEvaluator\n\n\ndef response_length(response, **kwargs):\n return {\"value\": len(response)}\n\n\nif __name__ == \"__main__\":\n # Built-in evaluators\n # Initialize Azure OpenAI Model Configuration\n model_config = {\n \"azure_endpoint\": os.environ.get(\"AZURE_OPENAI_ENDPOINT\"),\n \"api_key\": os.environ.get(\"AZURE_OPENAI_KEY\"),\n \"azure_deployment\": os.environ.get(\"AZURE_OPENAI_DEPLOYMENT\"),\n }\n\n # Initialzing Relevance Evaluator\n relevance_eval = RelevanceEvaluator(model_config)\n\n # Running Relevance Evaluator on single input row\n relevance_score = relevance_eval(\n response=\"The Alpine Explorer Tent is the most waterproof.\",\n query=\"Which tent is the most waterproof?\",\n )\n\n pprint(relevance_score)\n # {'gpt_relevance': 5.0}\n\n # Content Safety Evaluator\n\n # Initialize Project Scope\n azure_ai_project = {\n \"subscription_id\": <subscription_id>,\n \"resource_group_name\": <resource_group_name>,\n \"project_name\": <project_name>\n }\n\n violence_eval = ViolenceEvaluator(azure_ai_project)\n violence_score = violence_eval(query=\"What is the capital of France?\", response=\"Paris.\")\n pprint(violence_score)\n # {'violence': 'Very low',\n # 'violence_reason': \"The system's response is a straightforward factual response \"\n # 'to a geography question. There is no violent content or '\n # 'language present.',\n # 'violence_score': 0}\n\n # Code based evaluator\n response_length(\"The Alpine Explorer Tent is the most waterproof.\")\n # {'value': 48}\n\n # Using multiple evaluators together using `Evaluate` API\n\n result = evaluate(\n data=\"evaluate_test_data.jsonl\",\n evaluators={\n \"response_length\": response_length,\n \"violence\": violence_eval,\n },\n )\n\n pprint(result)\n```\n### Simulator\n\n\nSimulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes\ntheir AI application.\n\n#### Simulating with a Prompty\n\n```yaml\n---\nname: ApplicationPrompty\ndescription: Simulates an application\nmodel:\n api: chat\n parameters:\n temperature: 0.0\n top_p: 1.0\n presence_penalty: 0\n frequency_penalty: 0\n response_format:\n type: text\n\ninputs:\n conversation_history:\n type: dict\n\n---\nsystem:\nYou are a helpful assistant and you're helping with the user's query. Keep the conversation engaging and interesting.\n\nOutput with a string that continues the conversation, responding to the latest message from the user, given the conversation history:\n{{ conversation_history }}\n\n```\n\nQuery Response generaing prompty for gpt-4o with `json_schema` support\nUse this file as an override.\n```yaml\n---\nname: TaskSimulatorQueryResponseGPT4o\ndescription: Gets queries and responses from a blob of text\nmodel:\n api: chat\n parameters:\n temperature: 0.0\n top_p: 1.0\n presence_penalty: 0\n frequency_penalty: 0\n response_format:\n type: json_schema\n json_schema:\n name: QRJsonSchema\n schema: \n type: object\n properties:\n items:\n type: array\n items:\n type: object\n properties:\n q:\n type: string\n r:\n type: string\n required:\n - q\n - r\n\ninputs:\n text:\n type: string\n num_queries:\n type: integer\n\n\n---\nsystem:\nYou're an AI that helps in preparing a Question/Answer quiz from Text for \"Who wants to be a millionaire\" tv show\nBoth Questions and Answers MUST BE extracted from given Text\nFrame Question in a way so that Answer is RELEVANT SHORT BITE-SIZED info from Text\nRELEVANT info could be: NUMBER, DATE, STATISTIC, MONEY, NAME\nA sentence should contribute multiple QnAs if it has more info in it\nAnswer must not be more than 5 words\nAnswer must be picked from Text as is\nQuestion should be as descriptive as possible and must include as much context as possible from Text\nOutput must always have the provided number of QnAs\nOutput must be in JSON format.\nOutput must have {{num_queries}} objects in the format specified below. Any other count is unacceptable.\nText:\n<|text_start|>\nOn January 24, 1984, former Apple CEO Steve Jobs introduced the first Macintosh. In late 2003, Apple had 2.06 percent of the desktop share in the United States.\nSome years later, research firms IDC and Gartner reported that Apple's market share in the U.S. had increased to about 6%.\n<|text_end|>\nOutput with 5 QnAs:\n{\n \"qna\": [{\n \"q\": \"When did the former Apple CEO Steve Jobs introduced the first Macintosh?\",\n \"r\": \"January 24, 1984\"\n },\n {\n \"q\": \"Who was the former Apple CEO that introduced the first Macintosh on January 24, 1984?\",\n \"r\": \"Steve Jobs\"\n },\n {\n \"q\": \"What percent of the desktop share did Apple have in the United States in late 2003?\",\n \"r\": \"2.06 percent\"\n },\n {\n \"q\": \"What were the research firms that reported on Apple's market share in the U.S.?\",\n \"r\": \"IDC and Gartner\"\n },\n {\n \"q\": \"What was the percentage increase of Apple's market share in the U.S., as reported by research firms IDC and Gartner?\",\n \"r\": \"6%\"\n }]\n}\nText:\n<|text_start|>\n{{ text }}\n<|text_end|>\nOutput with {{ num_queries }} QnAs:\n```\n\nApplication code:\n\n```python\nimport json\nimport asyncio\nfrom typing import Any, Dict, List, Optional\nfrom azure.ai.evaluation.simulator import Simulator\nfrom promptflow.client import load_flow\nimport os\nimport wikipedia\n\n# Set up the model configuration without api_key, using DefaultAzureCredential\nmodel_config = {\n \"azure_endpoint\": os.environ.get(\"AZURE_OPENAI_ENDPOINT\"),\n \"azure_deployment\": os.environ.get(\"AZURE_DEPLOYMENT\"),\n # not providing key would make the SDK pick up `DefaultAzureCredential`\n # use \"api_key\": \"<your API key>\"\n \"api_version\": \"2024-08-01-preview\" # keep this for gpt-4o\n}\n\n# Use Wikipedia to get some text for the simulation\nwiki_search_term = \"Leonardo da Vinci\"\nwiki_title = wikipedia.search(wiki_search_term)[0]\nwiki_page = wikipedia.page(wiki_title)\ntext = wiki_page.summary[:1000]\n\ndef method_to_invoke_application_prompty(query: str, messages_list: List[Dict], context: Optional[Dict]):\n try:\n current_dir = os.path.dirname(__file__)\n prompty_path = os.path.join(current_dir, \"application.prompty\")\n _flow = load_flow(\n source=prompty_path,\n model=model_config,\n credential=DefaultAzureCredential()\n )\n response = _flow(\n query=query,\n context=context,\n conversation_history=messages_list\n )\n return response\n except Exception as e:\n print(f\"Something went wrong invoking the prompty: {e}\")\n return \"something went wrong\"\n\nasync def callback(\n messages: Dict[str, List[Dict]],\n stream: bool = False,\n session_state: Any = None, # noqa: ANN401\n context: Optional[Dict[str, Any]] = None,\n) -> dict:\n messages_list = messages[\"messages\"]\n # Get the last message from the user\n latest_message = messages_list[-1]\n query = latest_message[\"content\"]\n # Call your endpoint or AI application here\n response = method_to_invoke_application_prompty(query, messages_list, context)\n # Format the response to follow the OpenAI chat protocol format\n formatted_response = {\n \"content\": response,\n \"role\": \"assistant\",\n \"context\": \"\",\n }\n messages[\"messages\"].append(formatted_response)\n return {\"messages\": messages[\"messages\"], \"stream\": stream, \"session_state\": session_state, \"context\": context}\n\nasync def main():\n simulator = Simulator(model_config=model_config)\n current_dir = os.path.dirname(__file__)\n query_response_override_for_latest_gpt_4o = os.path.join(current_dir, \"TaskSimulatorQueryResponseGPT4o.prompty\")\n outputs = await simulator(\n target=callback,\n text=text,\n query_response_generating_prompty=query_response_override_for_latest_gpt_4o, # use this only with latest gpt-4o\n num_queries=2,\n max_conversation_turns=1,\n user_persona=[\n f\"I am a student and I want to learn more about {wiki_search_term}\",\n f\"I am a teacher and I want to teach my students about {wiki_search_term}\"\n ],\n )\n print(json.dumps(outputs, indent=2))\n\nif __name__ == \"__main__\":\n # Ensure that the following environment variables are set in your environment:\n # AZURE_OPENAI_ENDPOINT and AZURE_DEPLOYMENT\n # Example:\n # os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"https://your-endpoint.openai.azure.com/\"\n # os.environ[\"AZURE_DEPLOYMENT\"] = \"your-deployment-name\"\n asyncio.run(main())\n print(\"done!\")\n\n```\n\n#### Adversarial Simulator\n\n```python\nfrom azure.ai.evaluation.simulator import AdversarialSimulator, AdversarialScenario\nfrom azure.identity import DefaultAzureCredential\nfrom typing import Any, Dict, List, Optional\nimport asyncio\n\n\nazure_ai_project = {\n \"subscription_id\": <subscription_id>,\n \"resource_group_name\": <resource_group_name>,\n \"project_name\": <project_name>\n}\n\nasync def callback(\n messages: List[Dict],\n stream: bool = False,\n session_state: Any = None,\n context: Dict[str, Any] = None\n) -> dict:\n messages_list = messages[\"messages\"]\n # get last message\n latest_message = messages_list[-1]\n query = latest_message[\"content\"]\n context = None\n if 'file_content' in messages[\"template_parameters\"]:\n query += messages[\"template_parameters\"]['file_content']\n # the next few lines explains how to use the AsyncAzureOpenAI's chat.completions\n # to respond to the simulator. You should replace it with a call to your model/endpoint/application\n # make sure you pass the `query` and format the response as we have shown below\n from openai import AsyncAzureOpenAI\n oai_client = AsyncAzureOpenAI(\n api_key=<api_key>,\n azure_endpoint=<endpoint>,\n api_version=\"2023-12-01-preview\",\n )\n try:\n response_from_oai_chat_completions = await oai_client.chat.completions.create(messages=[{\"content\": query, \"role\": \"user\"}], model=\"gpt-4\", max_tokens=300)\n except Exception as e:\n print(f\"Error: {e}\")\n # to continue the conversation, return the messages, else you can fail the adversarial with an exception\n message = {\n \"content\": \"Something went wrong. Check the exception e for more details.\",\n \"role\": \"assistant\",\n \"context\": None,\n }\n messages[\"messages\"].append(message)\n return {\n \"messages\": messages[\"messages\"],\n \"stream\": stream,\n \"session_state\": session_state\n }\n response_result = response_from_oai_chat_completions.choices[0].message.content\n formatted_response = {\n \"content\": response_result,\n \"role\": \"assistant\",\n \"context\": {},\n }\n messages[\"messages\"].append(formatted_response)\n return {\n \"messages\": messages[\"messages\"],\n \"stream\": stream,\n \"session_state\": session_state,\n \"context\": context\n }\n\n```\n\n#### Adversarial QA\n\n```python\nscenario = AdversarialScenario.ADVERSARIAL_QA\nsimulator = AdversarialSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())\n\noutputs = asyncio.run(\n simulator(\n scenario=scenario,\n max_conversation_turns=1,\n max_simulation_results=3,\n target=callback\n )\n)\n\nprint(outputs.to_eval_qa_json_lines())\n```\n#### Direct Attack Simulator\n\n```python\nscenario = AdversarialScenario.ADVERSARIAL_QA\nsimulator = DirectAttackSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())\n\noutputs = asyncio.run(\n simulator(\n scenario=scenario,\n max_conversation_turns=1,\n max_simulation_results=2,\n target=callback\n )\n)\n\nprint(outputs)\n```\n## Troubleshooting\n\n### General\n\nAzure ML clients raise exceptions defined in [Azure Core][azure_core_readme].\n\n### Logging\n\nThis library uses the standard\n[logging][python_logging] library for logging.\nBasic information about HTTP sessions (URLs, headers, etc.) is logged at INFO\nlevel.\n\nDetailed DEBUG level logging, including request/response bodies and unredacted\nheaders, can be enabled on a client with the `logging_enable` argument.\n\nSee full SDK logging documentation with examples [here][sdk_logging_docs].\n\n## Next steps\n\n- View our [samples][evaluation_samples].\n- View our [documentation][product_documentation]\n\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.\n\n<!-- LINKS -->\n\n[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/evaluation/azure-ai-evaluation\n[evaluation_pypi]: https://pypi.org/project/azure-ai-evaluation/\n[evaluation_ref_docs]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview\n[evaluation_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios\n[product_documentation]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk\n[python_logging]: https://docs.python.org/3/library/logging.html\n[sdk_logging_docs]: https://docs.microsoft.com/azure/developer/python/azure-sdk-logging\n[azure_core_readme]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md\n[pip_link]: https://pypi.org/project/pip/\n[azure_core_ref_docs]: https://aka.ms/azsdk-python-core-policies\n[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md\n[azure_identity]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity\n[cla]: https://cla.microsoft.com\n[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/\n[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/\n[coc_contact]: mailto:opencode@microsoft.com\n\n\n# Release History\n\n## 1.0.0b5 (2024-10-28)\n\n### Features Added\n- Added `GroundednessProEvaluator`, which is a service-based evaluator for determining response groundedness.\n- Groundedness detection in Non Adversarial Simulator via query/context pairs\n```python\nimport importlib.resources as pkg_resources\npackage = \"azure.ai.evaluation.simulator._data_sources\"\nresource_name = \"grounding.json\"\ncustom_simulator = Simulator(model_config=model_config)\nconversation_turns = []\nwith pkg_resources.path(package, resource_name) as grounding_file:\n with open(grounding_file, \"r\") as file:\n data = json.load(file)\nfor item in data:\n conversation_turns.append([item])\noutputs = asyncio.run(custom_simulator(\n target=callback,\n conversation_turns=conversation_turns,\n max_conversation_turns=1,\n))\n```\n- Adding evaluator for multimodal use cases\n\n### Breaking Changes\n- Renamed environment variable `PF_EVALS_BATCH_USE_ASYNC` to `AI_EVALS_BATCH_USE_ASYNC`.\n- `RetrievalEvaluator` now requires a `context` input in addition to `query` in single-turn evaluation.\n- `RelevanceEvaluator` no longer takes `context` as an input. It now only takes `query` and `response` in single-turn evaluation.\n- `FluencyEvaluator` no longer takes `query` as an input. It now only takes `response` in single-turn evaluation.\n- AdversarialScenario enum does not include `ADVERSARIAL_INDIRECT_JAILBREAK`, invoking IndirectJailbreak or XPIA should be done with `IndirectAttackSimulator`\n- Outputs of `Simulator` and `AdversarialSimulator` previously had `to_eval_qa_json_lines` and now has `to_eval_qr_json_lines`. Where `to_eval_qa_json_lines` had:\n```json\n{\"question\": <user_message>, \"answer\": <assistant_message>}\n```\n`to_eval_qr_json_lines` now has:\n```json\n{\"query\": <user_message>, \"response\": assistant_message}\n```\n\n### Bugs Fixed\n- Non adversarial simulator works with `gpt-4o` models using the `json_schema` response format\n- Fixed an issue where the `evaluate` API would fail with \"[WinError 32] The process cannot access the file because it is being used by another process\" when venv folder and target function file are in the same directory.\n- Fix evaluate API failure when `trace.destination` is set to `none`\n- Non adversarial simulator now accepts context from the callback\n\n### Other Changes\n- Improved error messages for the `evaluate` API by enhancing the validation of input parameters. This update provides more detailed and actionable error descriptions.\n- `GroundednessEvaluator` now supports `query` as an optional input in single-turn evaluation. If `query` is provided, a different prompt template will be used for the evaluation.\n- To align with our support of a diverse set of models, the following evaluators will now have a new key in their result output without the `gpt_` prefix. To maintain backwards compatibility, the old key with the `gpt_` prefix will still be present in the output; however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.\n - `CoherenceEvaluator`\n - `RelevanceEvaluator`\n - `FluencyEvaluator`\n - `GroundednessEvaluator`\n - `SimilarityEvaluator`\n - `RetrievalEvaluator`\n- The following evaluators will now have a new key in their result output including LLM reasoning behind the score. The new key will follow the pattern \"<metric_name>_reason\". The reasoning is the result of a more detailed prompt template being used to generate the LLM response. Note that this requires the maximum number of tokens used to run these evaluators to be increased.\n \n | Evaluator | New Token Limit |\n | --- | --- |\n | `CoherenceEvaluator` | 800 |\n | `RelevanceEvaluator` | 800 |\n | `FluencyEvaluator` | 800 |\n | `GroundednessEvaluator` | 800 |\n | `RetrievalEvaluator` | 1600 |\n- Improved the error message for storage access permission issues to provide clearer guidance for users.\n\n## 1.0.0b4 (2024-10-16)\n\n### Breaking Changes\n\n- Removed `numpy` dependency. All NaN values returned by the SDK have been changed to from `numpy.nan` to `math.nan`.\n- `credential` is now required to be passed in for all content safety evaluators and `ProtectedMaterialsEvaluator`. `DefaultAzureCredential` will no longer be chosen if a credential is not passed.\n- Changed package extra name from \"pf-azure\" to \"remote\".\n\n### Bugs Fixed\n- Adversarial Conversation simulations would fail with `Forbidden`. Added logic to re-fetch token in the exponential retry logic to retrive RAI Service response.\n- Fixed an issue where the Evaluate API did not fail due to missing inputs when the target did not return columns required by the evaluators.\n\n### Other Changes\n- Enhance the error message to provide clearer instruction when required packages for the remote tracking feature are missing.\n- Print the per-evaluator run summary at the end of the Evaluate API call to make troubleshooting row-level failures easier.\n\n## 1.0.0b3 (2024-10-01)\n\n### Features Added\n\n- Added `type` field to `AzureOpenAIModelConfiguration` and `OpenAIModelConfiguration`\n- The following evaluators now support `conversation` as an alternative input to their usual single-turn inputs:\n - `ViolenceEvaluator`\n - `SexualEvaluator`\n - `SelfHarmEvaluator`\n - `HateUnfairnessEvaluator`\n - `ProtectedMaterialEvaluator`\n - `IndirectAttackEvaluator`\n - `CoherenceEvaluator`\n - `RelevanceEvaluator`\n - `FluencyEvaluator`\n - `GroundednessEvaluator`\n- Surfaced `RetrievalScoreEvaluator`, formally an internal part of `ChatEvaluator` as a standalone conversation-only evaluator.\n\n### Breaking Changes\n\n- Removed `ContentSafetyChatEvaluator` and `ChatEvaluator`\n- The `evaluator_config` parameter of `evaluate` now maps in evaluator name to a dictionary `EvaluatorConfig`, which is a `TypedDict`. The\n`column_mapping` between `data` or `target` and evaluator field names should now be specified inside this new dictionary:\n\nBefore:\n```python\nevaluate(\n ...,\n evaluator_config={\n \"hate_unfairness\": {\n \"query\": \"${data.question}\",\n \"response\": \"${data.answer}\",\n }\n },\n ...\n)\n```\n\nAfter\n```python\nevaluate(\n ...,\n evaluator_config={\n \"hate_unfairness\": {\n \"column_mapping\": {\n \"query\": \"${data.question}\",\n \"response\": \"${data.answer}\",\n }\n }\n },\n ...\n)\n```\n\n- Simulator now requires a model configuration to call the prompty instead of an Azure AI project scope. This enables the usage of simulator with Entra ID based auth.\nBefore:\n```python\nazure_ai_project = {\n \"subscription_id\": os.environ.get(\"AZURE_SUBSCRIPTION_ID\"),\n \"resource_group_name\": os.environ.get(\"RESOURCE_GROUP\"),\n \"project_name\": os.environ.get(\"PROJECT_NAME\"),\n}\nsim = Simulator(azure_ai_project=azure_ai_project, credentails=DefaultAzureCredentials())\n```\nAfter:\n```python\nmodel_config = {\n \"azure_endpoint\": os.environ.get(\"AZURE_OPENAI_ENDPOINT\"),\n \"azure_deployment\": os.environ.get(\"AZURE_DEPLOYMENT\"),\n}\nsim = Simulator(model_config=model_config)\n```\nIf `api_key` is not included in the `model_config`, the prompty runtime in `promptflow-core` will pick up `DefaultAzureCredential`.\n\n### Bugs Fixed\n\n- Fixed issue where Entra ID authentication was not working with `AzureOpenAIModelConfiguration`\n\n## 1.0.0b2 (2024-09-24)\n\n### Breaking Changes\n\n- `data` and `evaluators` are now required keywords in `evaluate`.\n\n## 1.0.0b1 (2024-09-20)\n\n### Breaking Changes\n\n- The `synthetic` namespace has been renamed to `simulator`, and sub-namespaces under this module have been removed\n- The `evaluate` and `evaluators` namespaces have been removed, and everything previously exposed in those modules has been added to the root namespace `azure.ai.evaluation`\n- The parameter name `project_scope` in content safety evaluators have been renamed to `azure_ai_project` for consistency with evaluate API and simulators.\n- Model configurations classes are now of type `TypedDict` and are exposed in the `azure.ai.evaluation` module instead of coming from `promptflow.core`.\n- Updated the parameter names for `question` and `answer` in built-in evaluators to more generic terms: `query` and `response`.\n\n### Features Added\n\n- First preview\n- This package is port of `promptflow-evals`. New features will be added only to this package moving forward.\n- Added a `TypedDict` for `AzureAIProject` that allows for better intellisense and type checking when passing in project information\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Microsoft Azure Evaluation Library for Python",
"version": "1.0.0b5",
"project_urls": {
"Bug Reports": "https://github.com/Azure/azure-sdk-for-python/issues",
"Homepage": "https://github.com/Azure/azure-sdk-for-python",
"Source": "https://github.com/Azure/azure-sdk-for-python"
},
"split_keywords": [
"azure",
" azure sdk"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "7e692d76017524a59d10f92e2970ba49a1fbeab9f46e8edfec41b203ed3cac2c",
"md5": "d60908c670ea02691758ca5f3e86c32d",
"sha256": "5e65d4669f70d1fafbb7dcf2153e49e556315ce0a20d4ff843bd03ed20443cc1"
},
"downloads": -1,
"filename": "azure_ai_evaluation-1.0.0b5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d60908c670ea02691758ca5f3e86c32d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 565768,
"upload_time": "2024-10-29T01:18:41",
"upload_time_iso_8601": "2024-10-29T01:18:41.761810Z",
"url": "https://files.pythonhosted.org/packages/7e/69/2d76017524a59d10f92e2970ba49a1fbeab9f46e8edfec41b203ed3cac2c/azure_ai_evaluation-1.0.0b5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4a0c28e0eecc52e771b0ba1f635fe907d1b03cbc6f36feff948e3b60af37056b",
"md5": "ba641c8ec33920a96abcd9a70e8ad17d",
"sha256": "70083ed55f448bf11ea5fc9d725a050b1b820cd12b9b7f6b0ad915be117e9f32"
},
"downloads": -1,
"filename": "azure_ai_evaluation-1.0.0b5.tar.gz",
"has_sig": false,
"md5_digest": "ba641c8ec33920a96abcd9a70e8ad17d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 557914,
"upload_time": "2024-10-29T01:18:39",
"upload_time_iso_8601": "2024-10-29T01:18:39.449516Z",
"url": "https://files.pythonhosted.org/packages/4a/0c/28e0eecc52e771b0ba1f635fe907d1b03cbc6f36feff948e3b60af37056b/azure_ai_evaluation-1.0.0b5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 01:18:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Azure",
"github_project": "azure-sdk-for-python",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "azure-ai-evaluation"
}
JSON
