| Name | dqc-toolkit JSON |
| Version |
0.2.0
JSON |
| download |
| home_page | None |
| Summary | Data Quality Check for Machine Learning |
| upload_time | 2024-10-02 12:01:16 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) 2024 sumanthprabhu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
nlp
data curation
machine learning
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<a href="https://github.com/sumanthprabhu/DQC-Toolkit/actions" alt="Build Status"><img src='https://img.shields.io/github/actions/workflow/status/sumanthprabhu/DQC-Toolkit/test.yml' alt="Build status"/></a>
<a href="https://sumanthprabhu.github.io/DQC-Toolkit/latest/" alt="Docs Status"><img src='https://img.shields.io/website?url=https%3A%2F%2Fsumanthprabhu.github.io%2FDQC-Toolkit%2F&label=docs' alt="Docs status"/></a>
<a href='https://pypi.org/project/dqc-toolkit/'><img src='https://img.shields.io/pypi/pyversions/DQC-Toolkit' alt="Python version"/></a>
<a href='https://pypi.org/project/dqc-toolkit/'><img src='https://img.shields.io/pypi/v/DQC-Toolkit' alt='Pypi version' /></a>
<a href='https://github.com/sumanthprabhu/DQC-Toolkit/blob/main/LICENSE'><img src='https://img.shields.io/pypi/l/DQC-toolkit' alt='Licence' /></a>
DQC Toolkit is a Python library and framework designed with the goal to facilitate improvement of Machine Learning models by identifying and mitigating label errors in training dataset. Currently, DQC toolkit offers `CrossValCurate` and `LLMCurate`. `CrossValCurate` can be used for label error detection / correction in text classification (binary / multi-class) based on cross validation. `LLMCurate` extends [PEDAL: Enhancing Greedy Decoding with Large Language Models using Diverse Exemplars](https://arxiv.org/abs/2408.08869) to compute LLM-based confidence scores for free-text labels.
## Installation
Installation of DQC-toolkit can be done as shown below
```python
pip install dqc-toolkit
```
## Quick Start
### CrossValCurate
Assuming your text classification data is stored as a pandas dataframe `data`, with each sample represented by the `text` column and its corresponding noisy label represented by the `label` column, here is how you use `CrossValCurate` -
```python linenums="1"
from dqc import CrossValCurate
cvc = CrossValCurate()
data_curated = cvc.fit_transform(data[['text', 'label']])
```
The result stored in `data_curated` is a pandas dataframe similar to `data` with the following columns -
```python
>>> data_curated.columns
['text', 'label', 'label_correctness_score', 'is_label_correct', 'predicted_label', 'prediction_probability']
```
* `'label_correctness_score'` represents a normalized score quantifying the correctness of `'label'`.
* `'is_label_correct'` is a boolean flag indicating whether the given `'label'` is correct (`True`) or incorrect (`False`).
* `'predicted_label'` and `'prediction_probability'` represent the curation model's prediction and the corresponding probability score.
### LLMCurate
Assuming `data` is a pandas dataframe containing samples with our target text for curation under column `column_to_curate`, here is how you use `LLMCurate` -
```python
llmc = LLMCurate(model, tokenizer)
ds = llmc.run(
data,
column_to_curate,
ds_column_mapping,
prompt_variants,
llm_response_cleaned_column_list,
answer_start_token,
answer_end_token,
batch_size,
max_new_tokens
)
```
where
* `model` and `tokenizer` are the instantiated LLM model and tokenizer objects respectively
* `ds_column_mapping` is the dictionary mapping of entities used in the LLM prompt to the corresponding columns in `data`. For example, `ds_column_mapping={'INPUT' : 'input_column'}` would imply that text under `input_column` in `data` would be passed to the LLM in the format `"[INPUT]row['input_column'][/INPUT]"` for each `row` in `data`
* `prompt_variants` is the list of LLM prompts to be used to curate `column_to_curate` and `llm_response_cleaned_column_list` is the corresponding list of column names to store the reference responses generated using each prompt
* `answer_start_token` and `answer_end_token` are optional text phrases representing the start and end of the answer respectively.
`ds` is a dataset object with the following additional features -
1. Feature for each column name in `llm_response_cleaned_column_list`
2. LLM Confidence score for each text in `column_to_curate`
For more details regarding different hyperparameters available in `CrossValCurate` and `LLMCurate`, please refer to the [API documentation](https://sumanthprabhu.github.io/DQC-Toolkit/).
Raw data
{
"_id": null,
"home_page": null,
"name": "dqc-toolkit",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "nlp, data curation, machine learning",
"author": null,
"author_email": "Sumanth S Prabhu <sumanthprabhu.104@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/c0/0c/ab4d956b4b2b537be563df964c9ae2fab0beff9fccd9e7aab913ef79dc53/dqc_toolkit-0.2.0.tar.gz",
"platform": null,
"description": "<a href=\"https://github.com/sumanthprabhu/DQC-Toolkit/actions\" alt=\"Build Status\"><img src='https://img.shields.io/github/actions/workflow/status/sumanthprabhu/DQC-Toolkit/test.yml' alt=\"Build status\"/></a> \n<a href=\"https://sumanthprabhu.github.io/DQC-Toolkit/latest/\" alt=\"Docs Status\"><img src='https://img.shields.io/website?url=https%3A%2F%2Fsumanthprabhu.github.io%2FDQC-Toolkit%2F&label=docs' alt=\"Docs status\"/></a>\n<a href='https://pypi.org/project/dqc-toolkit/'><img src='https://img.shields.io/pypi/pyversions/DQC-Toolkit' alt=\"Python version\"/></a>\n<a href='https://pypi.org/project/dqc-toolkit/'><img src='https://img.shields.io/pypi/v/DQC-Toolkit' alt='Pypi version' /></a> \n<a href='https://github.com/sumanthprabhu/DQC-Toolkit/blob/main/LICENSE'><img src='https://img.shields.io/pypi/l/DQC-toolkit' alt='Licence' /></a>\n\n\n\n\nDQC Toolkit is a Python library and framework designed with the goal to facilitate improvement of Machine Learning models by identifying and mitigating label errors in training dataset. Currently, DQC toolkit offers `CrossValCurate` and `LLMCurate`. `CrossValCurate` can be used for label error detection / correction in text classification (binary / multi-class) based on cross validation. `LLMCurate` extends [PEDAL: Enhancing Greedy Decoding with Large Language Models using Diverse Exemplars](https://arxiv.org/abs/2408.08869) to compute LLM-based confidence scores for free-text labels.\n\n## Installation\n\nInstallation of DQC-toolkit can be done as shown below\n```python\npip install dqc-toolkit\n```\n\n## Quick Start\n### CrossValCurate\nAssuming your text classification data is stored as a pandas dataframe `data`, with each sample represented by the `text` column and its corresponding noisy label represented by the `label` column, here is how you use `CrossValCurate` - \n\n\n```python linenums=\"1\"\n\nfrom dqc import CrossValCurate\n\ncvc = CrossValCurate()\ndata_curated = cvc.fit_transform(data[['text', 'label']])\n```\nThe result stored in `data_curated` is a pandas dataframe similar to `data` with the following columns -\n```python\n>>> data_curated.columns\n['text', 'label', 'label_correctness_score', 'is_label_correct', 'predicted_label', 'prediction_probability']\n```\n\n* `'label_correctness_score'` represents a normalized score quantifying the correctness of `'label'`. \n* `'is_label_correct'` is a boolean flag indicating whether the given `'label'` is correct (`True`) or incorrect (`False`). \n* `'predicted_label'` and `'prediction_probability'` represent the curation model's prediction and the corresponding probability score. \n \n### LLMCurate\nAssuming `data` is a pandas dataframe containing samples with our target text for curation under column `column_to_curate`, here is how you use `LLMCurate` -\n```python\n \n llmc = LLMCurate(model, tokenizer)\n ds = llmc.run(\n data,\n column_to_curate,\n ds_column_mapping,\n prompt_variants,\n llm_response_cleaned_column_list,\n answer_start_token,\n answer_end_token,\n batch_size,\n max_new_tokens\n )\n```\nwhere\n* `model` and `tokenizer` are the instantiated LLM model and tokenizer objects respectively\n* `ds_column_mapping` is the dictionary mapping of entities used in the LLM prompt to the corresponding columns in `data`. For example, `ds_column_mapping={'INPUT' : 'input_column'}` would imply that text under `input_column` in `data` would be passed to the LLM in the format `\"[INPUT]row['input_column'][/INPUT]\"` for each `row` in `data` \n* `prompt_variants` is the list of LLM prompts to be used to curate `column_to_curate` and `llm_response_cleaned_column_list` is the corresponding list of column names to store the reference responses generated using each prompt\n* `answer_start_token` and `answer_end_token` are optional text phrases representing the start and end of the answer respectively.\n\n`ds` is a dataset object with the following additional features -\n\n1. Feature for each column name in `llm_response_cleaned_column_list`\n2. LLM Confidence score for each text in `column_to_curate`\n \nFor more details regarding different hyperparameters available in `CrossValCurate` and `LLMCurate`, please refer to the [API documentation](https://sumanthprabhu.github.io/DQC-Toolkit/).\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 sumanthprabhu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Data Quality Check for Machine Learning",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/sumanthprabhu/DQC-Toolkit",
"Source": "https://github.com/sumanthprabhu/DQC-Toolkit",
"Tracker": "https://github.com/sumanthprabhu/DQC-Toolkit/issues"
},
"split_keywords": [
"nlp",
" data curation",
" machine learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1060257e76802bfe85aa9cf84450a028e3c2f50785c206db63446f9b389d4917",
"md5": "580778503cdca6a9df4fb18ce2ada24f",
"sha256": "7feb9b24c1b7f8b319d8d9017a440fb4d92c6aecdad5b3701cd9202c23003182"
},
"downloads": -1,
"filename": "dqc_toolkit-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "580778503cdca6a9df4fb18ce2ada24f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 31975,
"upload_time": "2024-10-02T12:01:14",
"upload_time_iso_8601": "2024-10-02T12:01:14.150459Z",
"url": "https://files.pythonhosted.org/packages/10/60/257e76802bfe85aa9cf84450a028e3c2f50785c206db63446f9b389d4917/dqc_toolkit-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c00cab4d956b4b2b537be563df964c9ae2fab0beff9fccd9e7aab913ef79dc53",
"md5": "4f35851015f6688fe887c26a502012b4",
"sha256": "0093885e01c558fd5906b2b95a82a2bedae8a906165ba1ec5c2d182e10c4b07e"
},
"downloads": -1,
"filename": "dqc_toolkit-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "4f35851015f6688fe887c26a502012b4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 27146,
"upload_time": "2024-10-02T12:01:16",
"upload_time_iso_8601": "2024-10-02T12:01:16.403049Z",
"url": "https://files.pythonhosted.org/packages/c0/0c/ab4d956b4b2b537be563df964c9ae2fab0beff9fccd9e7aab913ef79dc53/dqc_toolkit-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-02 12:01:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sumanthprabhu",
"github_project": "DQC-Toolkit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dqc-toolkit"
}
