| Name | dtlpymetrics JSON |
| Version |
1.0.106
JSON |
| download |
| home_page | None |
| Summary | Scoring and metrics app |
| upload_time | 2024-05-01 08:35:30 |
| maintainer | None |
| docs_url | None |
| author | Dataloop Team |
| requires_python | None |
| license | None |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Scoring and metrics app
## Description
This app improves the efficiency of annotating data by improving annotation quality while reducing the time required to
produce them.
The components of this app are:
1. Functions to calculate scores for quality tasks and model predictions.
2. Custom nodes that can be added to pipelines to calculate scores when a task's quality items are completed.
Also check [this notebook](docs/metrics.ipynb) for more information
## Quality Flows
To understand more about each of these tasks, refer to the main Dataloop documentation linked:
1. [Qualification tasks](https://dataloop.ai/docs/qualification-honeypot)
2. [Honeypot tasks](https://dataloop.ai/docs/qualification-honeypot#honeypot)
3. [Consensus tasks](https://dataloop.ai/docs/consensus)
In general, an annotator will receive an assignment to complete their annotation task. For a given item in a consensus task,
each assignment will be cross-compared with every other assignment. In the case of qualification and honeypot tasks,
each item will only have one assignment associated with it.
### What's Supported?
Supported file types:
- image
- video
Scoring is currently supported for quality tasks with the following annotation types (with geometry score method in parentheses, where applicable):
- classification
- bounding box (IOU)
- polygon (IOU)
- segmentation (IOU)
- point (distance)
## Score Types
During scoring, the following scores will be created for each annotation:
- `raw_annotation_scores` - for each annotation comparison we have `geometry`, `label` and `attribute` matching scores
- `annotation_overall` - the mean of each annotation’s raw scores
- `user_confusion_score` - the mean of every annotation overall score, relative to ref or another assignee
- `item_confusion_score` - the count of the number of label pairs associated with the assignee’s label, relative to the reference’s label
- `item_overall_score` - the mean value of *each* annotation overall score associated with an item
**1) Raw annotation scores:**
There are three types of scores for annotations: `annotation_iou`, `annotation_label` and `annotation_attribute`.
These scores can be determined by the user, and the default is to include all three scores, and the default value is 1 (which can be modified).
**2) Annotation overall**
For `annotation_overall` score we calculate the mean value for all raw annotation scores per annotation.
**3) User confusion score**
The `user_confusion` score represents the mean annotation score a given assignee has, relative to raw scores when comparing it to another set of annotations (either the reference or another assignee).
**4) Label confusion score**
The `label_confusion` score represents the count for a label annotated by a given assignee, relative to label each label class in the other set of annotations (either reference or another assignee).
**5) Item overall score**
The `item_overall` score is the mean value of all annotations associated with an item, averaging the mean overall annotation score.
Any calculated and uploaded scores will replace any previous scores for all items of a given task.
_Note about videos_: Video scores will differ slightly from image scores. Video sores are calculated frame by frame, and then specific annotation scores will be the average of these scores across all relevant frames for that specific annotation. Confusion scores are not calculated due to the multi-frame nature of videos. Item overall scores remain an average of all annotations of the video item.
## Confusion Example
There are generally two kinds of scores: regular scores, and “confusion” scores.
Regular scores show the level of agreement or overlap between two sets of annotations. They use the ID of the entities being compared for the `entityID` and `relative` fields. This can be for comparing annotations or items. `value` will typically be a number between 0 and 1.
There are two types of confusion scores: item label confusion, and user confusion. **Item label confusion** shows the number of instances in which an assignee’s label corresponds with the ground truth labels.
_Ground truth annotations_:
`item = dl.items.dl(item_id='64c0fc0730b03f27ca3a58db')`
_Assignee annotations_:
`item = dl.items.dl(item_id='64c0f2e1ec9103d52eaedbe2')`
In this example item, the ground truth has 3 for each cat and dog class. The assignee however, labels 1 as cat and 5 as dog. This would result in the following item label confusion scores:
```python
{
"type": "label_confusion",
"value": 1,
"entityId": "cat",
"context": {
"relative": "cat",
"taskId": "<TASK_ID>",
"itemId": "<ITEM_ID">,
"datasetId": "<DATASET_ID>"
}
},
{
"type": "label_confusion",
"value": 3,
"entityId": "dog",
"context": {
"relative": "dog",
"taskId": "<TASK_ID>",
"itemId": "<ITEM_ID">,
"datasetId": "<DATASET_ID>"
}
},
{
"type": "label_confusion",
"value": 2,
"entityId": "dog",
"context": {
"relative": "cat",
"taskId": "<TASK_ID>",
"itemId": "<ITEM_ID">,
"datasetId": "<DATASET_ID>"
}
}
```
## Python installation
```shell
pip install dtlpymetrics
```
## Functions
See [this page](docs/dtlpymetrics_fxns.md) for details on additional functions.
## Contributions, Bugs and Issues - How to Contribute
We welcome anyone to help us improve this app.
[Here](CONTRIBUTING.md) are detailed instructions to help you open a bug or ask for a feature request
Raw data
{
"_id": null,
"home_page": null,
"name": "dtlpymetrics",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Dataloop Team",
"author_email": "yaya.t@dataloop.ai",
"download_url": null,
"platform": null,
"description": "# Scoring and metrics app\n\n\n## Description\n\nThis app improves the efficiency of annotating data by improving annotation quality while reducing the time required to\nproduce them.\n\nThe components of this app are:\n\n1. Functions to calculate scores for quality tasks and model predictions.\n2. Custom nodes that can be added to pipelines to calculate scores when a task's quality items are completed.\n\nAlso check [this notebook](docs/metrics.ipynb) for more information\n\n## Quality Flows\nTo understand more about each of these tasks, refer to the main Dataloop documentation linked:\n\n1. [Qualification tasks](https://dataloop.ai/docs/qualification-honeypot)\n2. [Honeypot tasks](https://dataloop.ai/docs/qualification-honeypot#honeypot)\n3. [Consensus tasks](https://dataloop.ai/docs/consensus)\n\n\nIn general, an annotator will receive an assignment to complete their annotation task. For a given item in a consensus task, \neach assignment will be cross-compared with every other assignment. In the case of qualification and honeypot tasks, \neach item will only have one assignment associated with it. \n\n### What's Supported?\n\nSupported file types:\n- image\n- video\n\nScoring is currently supported for quality tasks with the following annotation types (with geometry score method in parentheses, where applicable):\n- classification\n- bounding box (IOU)\n- polygon (IOU)\n- segmentation (IOU)\n- point (distance)\n\n\n## Score Types\n\nDuring scoring, the following scores will be created for each annotation:\n\n- `raw_annotation_scores` - for each annotation comparison we have `geometry`, `label` and `attribute` matching scores\n- `annotation_overall` - the mean of each annotation\u2019s raw scores\n- `user_confusion_score` - the mean of every annotation overall score, relative to ref or another assignee\n- `item_confusion_score` - the count of the number of label pairs associated with the assignee\u2019s label, relative to the reference\u2019s label\n- `item_overall_score` - the mean value of *each* annotation overall score associated with an item\n\n**1) Raw annotation scores:** \n\nThere are three types of scores for annotations: `annotation_iou`, `annotation_label` and `annotation_attribute`. \nThese scores can be determined by the user, and the default is to include all three scores, and the default value is 1 (which can be modified).\n\n**2) Annotation overall**\n\nFor `annotation_overall` score we calculate the mean value for all raw annotation scores per annotation. \n\n**3) User confusion score**\n\nThe `user_confusion` score represents the mean annotation score a given assignee has, relative to raw scores when comparing it to another set of annotations (either the reference or another assignee). \n\n**4) Label confusion score**\n\nThe `label_confusion` score represents the count for a label annotated by a given assignee, relative to label each label class in the other set of annotations (either reference or another assignee).\n\n**5) Item overall score**\n\nThe `item_overall` score is the mean value of all annotations associated with an item, averaging the mean overall annotation score.\n\nAny calculated and uploaded scores will replace any previous scores for all items of a given task.\n\n_Note about videos_: Video scores will differ slightly from image scores. Video sores are calculated frame by frame, and then specific annotation scores will be the average of these scores across all relevant frames for that specific annotation. Confusion scores are not calculated due to the multi-frame nature of videos. Item overall scores remain an average of all annotations of the video item.\n\n## Confusion Example\n\nThere are generally two kinds of scores: regular scores, and \u201cconfusion\u201d scores. \n\nRegular scores show the level of agreement or overlap between two sets of annotations. They use the ID of the entities being compared for the `entityID` and `relative` fields. This can be for comparing annotations or items. `value` will typically be a number between 0 and 1. \n\n\nThere are two types of confusion scores: item label confusion, and user confusion. **Item label confusion** shows the number of instances in which an assignee\u2019s label corresponds with the ground truth labels. \n\n_Ground truth annotations_:\n\n\n\n`item = dl.items.dl(item_id='64c0fc0730b03f27ca3a58db')`\n\n_Assignee annotations_:\n\n\n\n`item = dl.items.dl(item_id='64c0f2e1ec9103d52eaedbe2')`\n\n\nIn this example item, the ground truth has 3 for each cat and dog class. The assignee however, labels 1 as cat and 5 as dog. This would result in the following item label confusion scores:\n\n```python\n{\n \"type\": \"label_confusion\",\n \"value\": 1,\n \"entityId\": \"cat\",\n \"context\": {\n \"relative\": \"cat\",\n \"taskId\": \"<TASK_ID>\",\n \"itemId\": \"<ITEM_ID\">,\n \"datasetId\": \"<DATASET_ID>\"\n }\n},\n{\n \"type\": \"label_confusion\",\n \"value\": 3,\n \"entityId\": \"dog\",\n \"context\": {\n \"relative\": \"dog\",\n \"taskId\": \"<TASK_ID>\",\n \"itemId\": \"<ITEM_ID\">,\n \"datasetId\": \"<DATASET_ID>\"\n }\n},\n{\n \"type\": \"label_confusion\",\n \"value\": 2,\n \"entityId\": \"dog\",\n \"context\": {\n \"relative\": \"cat\",\n \"taskId\": \"<TASK_ID>\",\n \"itemId\": \"<ITEM_ID\">,\n \"datasetId\": \"<DATASET_ID>\"\n }\n}\n```\n\n## Python installation\n\n```shell\npip install dtlpymetrics\n```\n\n## Functions\n\nSee [this page](docs/dtlpymetrics_fxns.md) for details on additional functions.\n\n## Contributions, Bugs and Issues - How to Contribute\n\nWe welcome anyone to help us improve this app. \n[Here](CONTRIBUTING.md) are detailed instructions to help you open a bug or ask for a feature request\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Scoring and metrics app",
"version": "1.0.106",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f12cef0da002dd3c6f7fba99d17e9d524931479f15e35102477e22c044d45751",
"md5": "b8c683b4c5bb1a9c1287c67e9ae5d402",
"sha256": "c04f4c62e2b909ca2a2dca87ab77a1198dfead42a23018f7f641c5b228d006d9"
},
"downloads": -1,
"filename": "dtlpymetrics-1.0.106-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b8c683b4c5bb1a9c1287c67e9ae5d402",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 36124,
"upload_time": "2024-05-01T08:35:30",
"upload_time_iso_8601": "2024-05-01T08:35:30.248014Z",
"url": "https://files.pythonhosted.org/packages/f1/2c/ef0da002dd3c6f7fba99d17e9d524931479f15e35102477e22c044d45751/dtlpymetrics-1.0.106-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-01 08:35:30",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "dtlpymetrics"
}
