| Name | comparisonframe JSON |
| Version |
0.0.5
JSON |
| download |
| home_page | None |
| Summary | A simple tool to compare textual data against validation sets. |
| upload_time | 2024-09-28 00:40:56 |
| maintainer | None |
| docs_url | None |
| author | Kyrylo Mordan |
| requires_python | None |
| license | mit |
| keywords |
['aa-paa-tool']
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Comparisonframe
Comparison Frame is designed to automate and streamline the
process of comparing textual data, particularly focusing on various
metrics such as character and word count, punctuation usage, and
semantic similarity.
It's particularly useful for scenarios where consistent text analysis is required,
such as evaluating the performance of natural language processing models,
monitoring content quality, or tracking changes in textual data over
time using manual evaluation.
```python
from comparisonframe import ComparisonFrame
```
### 1. Creating validation set
#### 1.1 Initialize comparison class
```python
comparer = ComparisonFrame(
# optionally
## mocker default parameters
mocker_params = {
'file_path' : "./comparisonframe_storage",
'persist' : True},
## scores to calculate
compare_scores = ['word_count_diff','semantic_similarity'],
aggr_scores = ['median']
)
```
/home/kyriosskia/miniforge3/envs/testenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
#### 1.2 Recording queries and expected responses (validation set)
```python
comparer.record_queries(
queries = ["Black metal",
"Tribulation"],
expected_texts = ["Black metal is an extreme subgenre of heavy metal music.",
"Tribulation are a Swedish heavy metal band from Arvika that formed in 2005."],
metadata = {'name' : 'metal_bands'})
```
### 2. Comparing newly generated data with expected results
#### 2.1 Initialize new comparison class
```python
comparer = ComparisonFrame(
# optionally
## mocker default parameters
mocker_params = {
'file_path' : "./comparisonframe_storage",
'persist' : True},
## scores to calculate
compare_scores = ['word_count_diff','semantic_similarity'],
aggr_scores = ['median']
)
```
### 2.2 Show validation set
```python
untested_queries = comparer.get_all_queries(
## optional
metadata_filters={'name' : 'metal_bands'})
print(untested_queries)
```
['Black metal', 'Tribulation']
```python
comparer.get_all_records()
```
[{'expected_text': 'Black metal is an extreme subgenre of heavy metal music.',
'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',
'query': 'Black metal'},
{'expected_text': 'Tribulation are a Swedish heavy metal band from Arvika that formed in 2005.',
'record_id': 'eecd9c2a5b25ee6053891b894157fa30372ed694763385e1ada1dc9ad8e41625',
'query': 'Tribulation'}]
```python
comparer.get_all_records_df()
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>expected_text</th>
<th>record_id</th>
<th>query</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>Black metal is an extreme subgenre of heavy me...</td>
<td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>
<td>Black metal</td>
</tr>
<tr>
<th>1</th>
<td>Tribulation are a Swedish heavy metal band fro...</td>
<td>eecd9c2a5b25ee6053891b894157fa30372ed694763385...</td>
<td>Tribulation</td>
</tr>
</tbody>
</table>
</div>
#### 2.3 Insert newly generated with records
```python
valid_answer_query_1 = "Black metal is an extreme subgenre of heavy metal music."
very_similar_answer_query_1 = "Black metal is a subgenre of heavy metal music."
unexpected_answer_query_1 = "Black metals are beautiful and are often used in jewelry design."
```
```python
comparer.record_runs(queries = ["Black metal"],
provided_texts = [valid_answer_query_1,
very_similar_answer_query_1,
unexpected_answer_query_1],
metadata={'desc' : 'definitions'})
```
```python
comparer.get_all_runs()
```
[{'query': 'Black metal',
'provided_text': 'Black metal is an extreme subgenre of heavy metal music.',
'run_id': 'faf5aab28ee8d460cbb69c6f434bee622aff8cdfb8796282bdc547fff2c1abf8',
'timestamp': '2024-09-26 01:36:13'},
{'query': 'Black metal',
'provided_text': 'Black metal is a subgenre of heavy metal music.',
'run_id': '9fbd80050d382972c012ffcb4641f48d6220afb2210a20a11da5c7a48664f033',
'timestamp': '2024-09-26 01:36:13'},
{'query': 'Black metal',
'provided_text': 'Black metals are beautiful and are often used in jewelry design.',
'run_id': 'e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d05cfc8ee4ea0bd7129',
'timestamp': '2024-09-26 01:36:13'}]
```python
df = comparer.get_all_runs_df()
df
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>query</th>
<th>provided_text</th>
<th>run_id</th>
<th>timestamp</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>Black metal</td>
<td>Black metal is an extreme subgenre of heavy me...</td>
<td>faf5aab28ee8d460cbb69c6f434bee622aff8cdfb87962...</td>
<td>2024-09-26 01:36:13</td>
</tr>
<tr>
<th>1</th>
<td>Black metal</td>
<td>Black metal is a subgenre of heavy metal music.</td>
<td>9fbd80050d382972c012ffcb4641f48d6220afb2210a20...</td>
<td>2024-09-26 01:36:13</td>
</tr>
<tr>
<th>2</th>
<td>Black metal</td>
<td>Black metals are beautiful and are often used ...</td>
<td>e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d...</td>
<td>2024-09-26 01:36:13</td>
</tr>
</tbody>
</table>
</div>
#### 2.4 Comparing runs with records
```python
comparer.compare_runs_with_records()
```
WARNING:ComparisonFrame:No data was found with applied filters!
```python
comparer.get_all_run_scores()
```
[{'query': 'Black metal',
'provided_text': 'Black metal is an extreme subgenre of heavy metal music.',
'run_id': 'faf5aab28ee8d460cbb69c6f434bee622aff8cdfb8796282bdc547fff2c1abf8',
'timestamp': '2024-09-26 01:36:13',
'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',
'word_count_diff': 0,
'semantic_similarity': 0.9999999403953552,
'comparison_id': 'cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d6ef9154daacd9b732d'},
{'query': 'Black metal',
'provided_text': 'Black metal is a subgenre of heavy metal music.',
'run_id': '9fbd80050d382972c012ffcb4641f48d6220afb2210a20a11da5c7a48664f033',
'timestamp': '2024-09-26 01:36:13',
'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',
'word_count_diff': 1,
'semantic_similarity': 0.9859851002693176,
'comparison_id': '16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b7480a51dc8c6796c41'},
{'query': 'Black metal',
'provided_text': 'Black metals are beautiful and are often used in jewelry design.',
'run_id': 'e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d05cfc8ee4ea0bd7129',
'timestamp': '2024-09-26 01:36:13',
'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',
'word_count_diff': 1,
'semantic_similarity': 0.4940534234046936,
'comparison_id': '966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be11529d2cf7a222deb9a'}]
```python
comparer.get_all_run_scores_df()
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>query</th>
<th>provided_text</th>
<th>run_id</th>
<th>timestamp</th>
<th>record_id</th>
<th>word_count_diff</th>
<th>semantic_similarity</th>
<th>comparison_id</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>Black metal</td>
<td>Black metal is an extreme subgenre of heavy me...</td>
<td>faf5aab28ee8d460cbb69c6f434bee622aff8cdfb87962...</td>
<td>2024-09-26 01:36:13</td>
<td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>
<td>0</td>
<td>1.000000</td>
<td>cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d...</td>
</tr>
<tr>
<th>1</th>
<td>Black metal</td>
<td>Black metal is a subgenre of heavy metal music.</td>
<td>9fbd80050d382972c012ffcb4641f48d6220afb2210a20...</td>
<td>2024-09-26 01:36:13</td>
<td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>
<td>1</td>
<td>0.985985</td>
<td>16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b...</td>
</tr>
<tr>
<th>2</th>
<td>Black metal</td>
<td>Black metals are beautiful and are often used ...</td>
<td>e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d...</td>
<td>2024-09-26 01:36:13</td>
<td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>
<td>1</td>
<td>0.494053</td>
<td>966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be1...</td>
</tr>
</tbody>
</table>
</div>
### 3 Calculating aggregate comparison scores
```python
comparer.calculate_aggr_scores(group_by = ['desc'])
```
WARNING:ComparisonFrame:No data was found with applied filters!
```python
comparer.get_all_aggr_scores()
```
[{'timestamp': '2024-09-26 01:36:13',
'comparison_id': ['cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d6ef9154daacd9b732d',
'16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b7480a51dc8c6796c41',
'966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be11529d2cf7a222deb9a'],
'query': ['Black metal'],
'grouped_by': ['query'],
'group': {'query': 'Black metal'},
'median_word_count_diff': 1.0,
'median_semantic_similarity': 0.9859851002693176,
'record_status_id': 'dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d430dea99fdd4b8c628'}]
```python
comparer.get_all_aggr_scores_df(grouped_by = ['desc'])
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>timestamp</th>
<th>comparison_id</th>
<th>query</th>
<th>grouped_by</th>
<th>group</th>
<th>median_word_count_diff</th>
<th>median_semantic_similarity</th>
<th>record_status_id</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>2024-09-26 01:36:13</td>
<td>[cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01...</td>
<td>[Black metal]</td>
<td>[desc]</td>
<td>{'desc': 'definitions'}</td>
<td>1.0</td>
<td>0.985985</td>
<td>c9d97729c5b03641fbf8fd35d257f2f1024a812f097ffb...</td>
</tr>
</tbody>
</table>
</div>
### 4. Recording test statuses
```python
comparer.calculate_test_statuses(test_query = "median_semantic_similarity > 0.9")
```
```python
comparer.get_test_statuses()
```
[{'timestamp': '2024-09-26 01:36:13',
'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',
'record_status_id': 'dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d430dea99fdd4b8c628',
'query': 'Black metal',
'test': 'median_semantic_similarity > 0.9',
'valid': True}]
```python
comparer.get_test_statuses_df()
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>timestamp</th>
<th>record_id</th>
<th>record_status_id</th>
<th>query</th>
<th>test</th>
<th>valid</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>2024-09-26 01:36:13</td>
<td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>
<td>dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d...</td>
<td>Black metal</td>
<td>median_semantic_similarity > 0.9</td>
<td>True</td>
</tr>
</tbody>
</table>
</div>
### 5. Reseting statuses, flushing records and comparison results
```python
comparer.flush_records()
```
```python
comparer.flush_runs()
```
```python
comparer.flush_comparison_scores()
```
```python
comparer.flush_aggregate_scores()
```
```python
comparer.flush_test_statuses()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "comparisonframe",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "['aa-paa-tool']",
"author": "Kyrylo Mordan",
"author_email": "parachute.repo@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f0/83/7e84cefb528c3a8031e00c2d94eb9275e96be9621ae2dd586724ac7ee163/comparisonframe-0.0.5.tar.gz",
"platform": null,
"description": "# Comparisonframe\n\nComparison Frame is designed to automate and streamline the \nprocess of comparing textual data, particularly focusing on various \nmetrics such as character and word count, punctuation usage, and \nsemantic similarity.\nIt's particularly useful for scenarios where consistent text analysis is required,\nsuch as evaluating the performance of natural language processing models, \nmonitoring content quality, or tracking changes in textual data over \ntime using manual evaluation.\n\n```python\nfrom comparisonframe import ComparisonFrame\n```\n\n### 1. Creating validation set\n\n#### 1.1 Initialize comparison class\n\n\n```python\ncomparer = ComparisonFrame(\n # optionally\n ## mocker default parameters\n mocker_params = {\n 'file_path' : \"./comparisonframe_storage\",\n 'persist' : True},\n\n ## scores to calculate\n compare_scores = ['word_count_diff','semantic_similarity'],\n aggr_scores = ['median']\n)\n```\n\n /home/kyriosskia/miniforge3/envs/testenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n warnings.warn(\n\n\n#### 1.2 Recording queries and expected responses (validation set)\n\n\n```python\ncomparer.record_queries(\n queries = [\"Black metal\", \n \"Tribulation\"],\n expected_texts = [\"Black metal is an extreme subgenre of heavy metal music.\",\n \"Tribulation are a Swedish heavy metal band from Arvika that formed in 2005.\"],\n metadata = {'name' : 'metal_bands'})\n```\n\n### 2. Comparing newly generated data with expected results \n\n#### 2.1 Initialize new comparison class\n\n\n```python\ncomparer = ComparisonFrame(\n # optionally\n ## mocker default parameters\n mocker_params = {\n 'file_path' : \"./comparisonframe_storage\",\n 'persist' : True},\n\n ## scores to calculate\n compare_scores = ['word_count_diff','semantic_similarity'],\n aggr_scores = ['median']\n)\n```\n\n### 2.2 Show validation set\n\n\n```python\nuntested_queries = comparer.get_all_queries(\n ## optional\n metadata_filters={'name' : 'metal_bands'})\nprint(untested_queries)\n```\n\n ['Black metal', 'Tribulation']\n\n\n\n```python\ncomparer.get_all_records()\n```\n\n\n\n\n [{'expected_text': 'Black metal is an extreme subgenre of heavy metal music.',\n 'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',\n 'query': 'Black metal'},\n {'expected_text': 'Tribulation are a Swedish heavy metal band from Arvika that formed in 2005.',\n 'record_id': 'eecd9c2a5b25ee6053891b894157fa30372ed694763385e1ada1dc9ad8e41625',\n 'query': 'Tribulation'}]\n\n\n\n\n```python\ncomparer.get_all_records_df()\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>expected_text</th>\n <th>record_id</th>\n <th>query</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>Black metal is an extreme subgenre of heavy me...</td>\n <td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>\n <td>Black metal</td>\n </tr>\n <tr>\n <th>1</th>\n <td>Tribulation are a Swedish heavy metal band fro...</td>\n <td>eecd9c2a5b25ee6053891b894157fa30372ed694763385...</td>\n <td>Tribulation</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n#### 2.3 Insert newly generated with records\n\n\n```python\nvalid_answer_query_1 = \"Black metal is an extreme subgenre of heavy metal music.\"\nvery_similar_answer_query_1 = \"Black metal is a subgenre of heavy metal music.\"\nunexpected_answer_query_1 = \"Black metals are beautiful and are often used in jewelry design.\"\n```\n\n\n```python\ncomparer.record_runs(queries = [\"Black metal\"],\n provided_texts = [valid_answer_query_1,\n very_similar_answer_query_1,\n unexpected_answer_query_1],\n metadata={'desc' : 'definitions'})\n```\n\n\n```python\ncomparer.get_all_runs()\n```\n\n\n\n\n [{'query': 'Black metal',\n 'provided_text': 'Black metal is an extreme subgenre of heavy metal music.',\n 'run_id': 'faf5aab28ee8d460cbb69c6f434bee622aff8cdfb8796282bdc547fff2c1abf8',\n 'timestamp': '2024-09-26 01:36:13'},\n {'query': 'Black metal',\n 'provided_text': 'Black metal is a subgenre of heavy metal music.',\n 'run_id': '9fbd80050d382972c012ffcb4641f48d6220afb2210a20a11da5c7a48664f033',\n 'timestamp': '2024-09-26 01:36:13'},\n {'query': 'Black metal',\n 'provided_text': 'Black metals are beautiful and are often used in jewelry design.',\n 'run_id': 'e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d05cfc8ee4ea0bd7129',\n 'timestamp': '2024-09-26 01:36:13'}]\n\n\n\n\n```python\ndf = comparer.get_all_runs_df()\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>query</th>\n <th>provided_text</th>\n <th>run_id</th>\n <th>timestamp</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>Black metal</td>\n <td>Black metal is an extreme subgenre of heavy me...</td>\n <td>faf5aab28ee8d460cbb69c6f434bee622aff8cdfb87962...</td>\n <td>2024-09-26 01:36:13</td>\n </tr>\n <tr>\n <th>1</th>\n <td>Black metal</td>\n <td>Black metal is a subgenre of heavy metal music.</td>\n <td>9fbd80050d382972c012ffcb4641f48d6220afb2210a20...</td>\n <td>2024-09-26 01:36:13</td>\n </tr>\n <tr>\n <th>2</th>\n <td>Black metal</td>\n <td>Black metals are beautiful and are often used ...</td>\n <td>e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d...</td>\n <td>2024-09-26 01:36:13</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n#### 2.4 Comparing runs with records\n\n\n```python\ncomparer.compare_runs_with_records()\n```\n\n WARNING:ComparisonFrame:No data was found with applied filters!\n\n\n\n```python\ncomparer.get_all_run_scores()\n```\n\n\n\n\n [{'query': 'Black metal',\n 'provided_text': 'Black metal is an extreme subgenre of heavy metal music.',\n 'run_id': 'faf5aab28ee8d460cbb69c6f434bee622aff8cdfb8796282bdc547fff2c1abf8',\n 'timestamp': '2024-09-26 01:36:13',\n 'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',\n 'word_count_diff': 0,\n 'semantic_similarity': 0.9999999403953552,\n 'comparison_id': 'cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d6ef9154daacd9b732d'},\n {'query': 'Black metal',\n 'provided_text': 'Black metal is a subgenre of heavy metal music.',\n 'run_id': '9fbd80050d382972c012ffcb4641f48d6220afb2210a20a11da5c7a48664f033',\n 'timestamp': '2024-09-26 01:36:13',\n 'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',\n 'word_count_diff': 1,\n 'semantic_similarity': 0.9859851002693176,\n 'comparison_id': '16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b7480a51dc8c6796c41'},\n {'query': 'Black metal',\n 'provided_text': 'Black metals are beautiful and are often used in jewelry design.',\n 'run_id': 'e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d05cfc8ee4ea0bd7129',\n 'timestamp': '2024-09-26 01:36:13',\n 'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',\n 'word_count_diff': 1,\n 'semantic_similarity': 0.4940534234046936,\n 'comparison_id': '966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be11529d2cf7a222deb9a'}]\n\n\n\n\n```python\ncomparer.get_all_run_scores_df()\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>query</th>\n <th>provided_text</th>\n <th>run_id</th>\n <th>timestamp</th>\n <th>record_id</th>\n <th>word_count_diff</th>\n <th>semantic_similarity</th>\n <th>comparison_id</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>Black metal</td>\n <td>Black metal is an extreme subgenre of heavy me...</td>\n <td>faf5aab28ee8d460cbb69c6f434bee622aff8cdfb87962...</td>\n <td>2024-09-26 01:36:13</td>\n <td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>\n <td>0</td>\n <td>1.000000</td>\n <td>cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d...</td>\n </tr>\n <tr>\n <th>1</th>\n <td>Black metal</td>\n <td>Black metal is a subgenre of heavy metal music.</td>\n <td>9fbd80050d382972c012ffcb4641f48d6220afb2210a20...</td>\n <td>2024-09-26 01:36:13</td>\n <td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>\n <td>1</td>\n <td>0.985985</td>\n <td>16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b...</td>\n </tr>\n <tr>\n <th>2</th>\n <td>Black metal</td>\n <td>Black metals are beautiful and are often used ...</td>\n <td>e4fc3f56c95d4266b6543a306c4305e0d8b960a1e0196d...</td>\n <td>2024-09-26 01:36:13</td>\n <td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>\n <td>1</td>\n <td>0.494053</td>\n <td>966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be1...</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n### 3 Calculating aggregate comparison scores\n\n\n```python\ncomparer.calculate_aggr_scores(group_by = ['desc'])\n```\n\n WARNING:ComparisonFrame:No data was found with applied filters!\n\n\n\n```python\ncomparer.get_all_aggr_scores()\n```\n\n\n\n\n [{'timestamp': '2024-09-26 01:36:13',\n 'comparison_id': ['cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01d6ef9154daacd9b732d',\n '16472e44ac7d2d74e18ea583490c2f6b8661cc8b48cc9b7480a51dc8c6796c41',\n '966c1da5e641480e8ccd33a7d0f544d9ec6c4e2e799be11529d2cf7a222deb9a'],\n 'query': ['Black metal'],\n 'grouped_by': ['query'],\n 'group': {'query': 'Black metal'},\n 'median_word_count_diff': 1.0,\n 'median_semantic_similarity': 0.9859851002693176,\n 'record_status_id': 'dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d430dea99fdd4b8c628'}]\n\n\n\n\n```python\ncomparer.get_all_aggr_scores_df(grouped_by = ['desc'])\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>timestamp</th>\n <th>comparison_id</th>\n <th>query</th>\n <th>grouped_by</th>\n <th>group</th>\n <th>median_word_count_diff</th>\n <th>median_semantic_similarity</th>\n <th>record_status_id</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>2024-09-26 01:36:13</td>\n <td>[cdb16a8d16a95e85d879c29aaf9762c9e2776843f2a01...</td>\n <td>[Black metal]</td>\n <td>[desc]</td>\n <td>{'desc': 'definitions'}</td>\n <td>1.0</td>\n <td>0.985985</td>\n <td>c9d97729c5b03641fbf8fd35d257f2f1024a812f097ffb...</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n### 4. Recording test statuses\n\n\n```python\ncomparer.calculate_test_statuses(test_query = \"median_semantic_similarity > 0.9\")\n\n```\n\n\n```python\ncomparer.get_test_statuses()\n```\n\n\n\n\n [{'timestamp': '2024-09-26 01:36:13',\n 'record_id': '0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8bd7b663c92f2f16e87',\n 'record_status_id': 'dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d430dea99fdd4b8c628',\n 'query': 'Black metal',\n 'test': 'median_semantic_similarity > 0.9',\n 'valid': True}]\n\n\n\n\n```python\ncomparer.get_test_statuses_df()\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>timestamp</th>\n <th>record_id</th>\n <th>record_status_id</th>\n <th>query</th>\n <th>test</th>\n <th>valid</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>2024-09-26 01:36:13</td>\n <td>0cc157453395b440f36d1a1aee24aa76a03f5f9ab0a7a8...</td>\n <td>dc1126e128d42f74bb98bad9ce4101fe1a4ea5a46df57d...</td>\n <td>Black metal</td>\n <td>median_semantic_similarity > 0.9</td>\n <td>True</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n### 5. Reseting statuses, flushing records and comparison results\n\n\n```python\ncomparer.flush_records()\n```\n\n\n```python\ncomparer.flush_runs()\n```\n\n\n```python\ncomparer.flush_comparison_scores()\n```\n\n\n```python\ncomparer.flush_aggregate_scores()\n```\n\n\n```python\ncomparer.flush_test_statuses()\n```\n",
"bugtrack_url": null,
"license": "mit",
"summary": "A simple tool to compare textual data against validation sets.",
"version": "0.0.5",
"project_urls": null,
"split_keywords": [
"['aa-paa-tool']"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "99b341732838d1b26aeba99f61076d1d2305c51aa4961a1b3147007afc064828",
"md5": "c780c131ead8c1e55caabdc1db4b87a4",
"sha256": "22a489a894a0d0e5e025a22542b143e698f6f8471c456716774008b2b4161bce"
},
"downloads": -1,
"filename": "comparisonframe-0.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c780c131ead8c1e55caabdc1db4b87a4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 882173,
"upload_time": "2024-09-28T00:40:54",
"upload_time_iso_8601": "2024-09-28T00:40:54.432170Z",
"url": "https://files.pythonhosted.org/packages/99/b3/41732838d1b26aeba99f61076d1d2305c51aa4961a1b3147007afc064828/comparisonframe-0.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f0837e84cefb528c3a8031e00c2d94eb9275e96be9621ae2dd586724ac7ee163",
"md5": "b865dadd150247af938de2e095c7108b",
"sha256": "233d63e50d892bc385a7c95dc9f465293e82c45ff614a52f61f93f40c06e247f"
},
"downloads": -1,
"filename": "comparisonframe-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "b865dadd150247af938de2e095c7108b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 846978,
"upload_time": "2024-09-28T00:40:56",
"upload_time_iso_8601": "2024-09-28T00:40:56.178130Z",
"url": "https://files.pythonhosted.org/packages/f0/83/7e84cefb528c3a8031e00c2d94eb9275e96be9621ae2dd586724ac7ee163/comparisonframe-0.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-28 00:40:56",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "comparisonframe"
}
