| Name | digital-eval JSON |
| Version |
1.6.0
JSON |
| download |
| home_page | None |
| Summary | Evaluate Digitalization Data |
| upload_time | 2024-05-24 17:43:01 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# digital eval
[](https://pypi.org/project/digital-eval)   
Python3 Tool to report evaluation outcomes from mass digitalization workflows.
## Features
* [OCR-D compliant](https://ocr-d.de/en/spec/ocrd_eval#character-error-rate-cer) normalized similarity for edit-distance based metrics based on characters, letters and words
* choose from textual metrics based on characters or words plus common Information Retrieval
* choose from different UTF-8 Python norms
* match groundtruth (i.e. reference data) and candidates by filename start
* use geometric information to evaluate only specific frame (i.e. specific column or region from large page) of
candidates (requires ALTO or PAGE format)
* aggregate evaluation outcomes on domain range (with multiple subdomains) according to folder layout
* formats: ALTO, PAGE or plain text for both groundtruth and candidates
* speedup with parallel execution
* additional OCR util:
* filter custom areas of single OCR files of ALTO files
## Installation
```bash
pip install digital-eval
```
## Usage
### Metrics
#### Edit-Distance based Strin Similarity
Calculate similarity for each single reference/groundtruth and test/candidate item.
Complete haracter-based text string (`Cs`, `Characters`) or Letter-based (`Ls`, `Letters`) minus whitespaces,
punctuation and common digits (arabic, persian).
Word/Token-based edit-distance of single tokens identified by Word or String elements or whitespaces, depending on data.
#### Set based
Calculate union of sets of tokens/words (`BoW`, `BagOfWords`).
Operate on sets of tokens/words with respect to language specific stopwords using [nltk](https://www.nltk.org/)
-framework for:
* Precision (`IRPre`, `Pre`, `Precision`): How many tokens from candidate are in groundtruth reference?
* Recall (`IRRec`, `Rec`, `Recall`): How many tokens from groundtruth reference should candidate include?
* F-Measure (`IRFMeasure`, `FM`): weighted ratio Precision / Recall
### UTF-8 Normalisations
Use standard Python Implementation of UTF-8 normalizations; default: `NFKD`.
### Statistics
Statistics calculated via [numpy](https://numpy.org/) include arithmetic mean, median and outlier detection with
interquartile range and are based on the specific groundtruth/reference (ref) for each metric, i.e. char, letters or
tokens.
### Evaluate treelike structures
To evaluate OCR-candidate-data batch-like versus existing Groundtruth, please make sure that your structures fit this
way:
```bash
groundtruth root/
├── <domain>/
│ └── <subdomain>/
│ └── <page-01>.gt.xml
candidate root/
├── <domain>/
│ └── <subdomain>/
│ └── <page-01>.xml
```
Now call via:
```bash
digital-eval <path-candidate-root>/domain/ -ref <path-groundtruth>/domain/
```
for an aggregated overview on stdout. Feel free to increase verbosity via `-v` (or even `-vv`) to get detailed
information about each single data set which was evaluated.
Structured OCR is considered to contain valid geometrical and textual data on word level, even though for recent PAGE
also line level is possible.
### Data problems
Inconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONTENT or PAGE without TextEquiv) or
invalid geometrical coordinates (less than 3 points or even empty) will lead to evaluation errors if geometry must be
respected.
## Additional OCR Utils
### Filter Area
You can filter a custom area of a page of an OCR file by providing the points of an arbitrary shape.
The format of the `-p, --points` argument is `<pt_1_x>,<pt_1_y> <pt_2_x>,<pt_2_y> <pt_3_x>,<pt_3_y> ... <pt_n_x>,<pt_n_y>` . For simple rectangular areas this can be expressed also with two points, with first point as top left and second point as bottom right: `<pt_top_left_x>,<pt_top_left_y> <pt_bottom_right_x>,<pt_bottom_right_y>`.
The following example filters a rectangular area of 600x400 pixels of a page, which is described by an input ALTO file and saves the result to an output ALTO file
```bash
ocr-util frame -i page_1.alto.xml -p "0,0 600,0 600,400 0,400" -o page_1_area.alto.xml
```
Short version with top left and bottom right:
```bash
ocr-util frame -i page_1.alto.xml -p "0,0 600,400" -o page_1_area.alto.xml
```
## Development
Plattform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 20.04 LTS, Python 3.8.
```bash
# clone local
git clone <repository-url> <local-dir>
cd <local-dir>
# enable virtual python 3 environment (linux)
# and update pip itself
python3 -m venv venv
. venv/bin/activate
python -m pip install -U pip
# install
python -m pip install -e .
# install additional development dependencies
python -m pip install -r tests/test_requirements.txt
# run tests
python -m pytest -v
```
## Contribute
Contributions, suggestions and proposals welcome!
## License
Under terms of the [MIT license](https://opensource.org/licenses/MIT).
**NOTE**: This software depends on packages that _might_ be licensed under different terms.
Raw data
{
"_id": null,
"home_page": null,
"name": "digital-eval",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Universit\u00e4ts- und Landesbibliothek Sachsen-Anhalt <development@bibliothek.uni-halle.de>",
"download_url": "https://files.pythonhosted.org/packages/51/6b/6b1567caae7be27b986d824a14a6d7fe634c11fb4fcf239fb8e43e69d972/digital_eval-1.6.0.tar.gz",
"platform": null,
"description": "# digital eval\n\n\n[](https://pypi.org/project/digital-eval)   \n\nPython3 Tool to report evaluation outcomes from mass digitalization workflows.\n\n## Features\n\n* [OCR-D compliant](https://ocr-d.de/en/spec/ocrd_eval#character-error-rate-cer) normalized similarity for edit-distance based metrics based on characters, letters and words\n* choose from textual metrics based on characters or words plus common Information Retrieval\n* choose from different UTF-8 Python norms\n* match groundtruth (i.e. reference data) and candidates by filename start\n* use geometric information to evaluate only specific frame (i.e. specific column or region from large page) of\n candidates (requires ALTO or PAGE format)\n* aggregate evaluation outcomes on domain range (with multiple subdomains) according to folder layout\n* formats: ALTO, PAGE or plain text for both groundtruth and candidates\n* speedup with parallel execution\n* additional OCR util:\n * filter custom areas of single OCR files of ALTO files\n\n## Installation\n\n```bash\npip install digital-eval\n```\n\n## Usage\n\n### Metrics\n\n#### Edit-Distance based Strin Similarity\n\nCalculate similarity for each single reference/groundtruth and test/candidate item.\nComplete haracter-based text string (`Cs`, `Characters`) or Letter-based (`Ls`, `Letters`) minus whitespaces,\npunctuation and common digits (arabic, persian). \nWord/Token-based edit-distance of single tokens identified by Word or String elements or whitespaces, depending on data.\n\n#### Set based\n\nCalculate union of sets of tokens/words (`BoW`, `BagOfWords`).\nOperate on sets of tokens/words with respect to language specific stopwords using [nltk](https://www.nltk.org/)\n-framework for:\n\n* Precision (`IRPre`, `Pre`, `Precision`): How many tokens from candidate are in groundtruth reference?\n* Recall (`IRRec`, `Rec`, `Recall`): How many tokens from groundtruth reference should candidate include?\n* F-Measure (`IRFMeasure`, `FM`): weighted ratio Precision / Recall\n\n### UTF-8 Normalisations\n\nUse standard Python Implementation of UTF-8 normalizations; default: `NFKD`.\n\n### Statistics\n\nStatistics calculated via [numpy](https://numpy.org/) include arithmetic mean, median and outlier detection with\ninterquartile range and are based on the specific groundtruth/reference (ref) for each metric, i.e. char, letters or\ntokens.\n\n### Evaluate treelike structures\n\nTo evaluate OCR-candidate-data batch-like versus existing Groundtruth, please make sure that your structures fit this\nway:\n\n```bash\ngroundtruth root/\n\u251c\u2500\u2500 <domain>/ \n\u2502 \u2514\u2500\u2500 <subdomain>/\n\u2502 \u2514\u2500\u2500 <page-01>.gt.xml\ncandidate root/\n\u251c\u2500\u2500 <domain>/ \n\u2502 \u2514\u2500\u2500 <subdomain>/\n\u2502 \u2514\u2500\u2500 <page-01>.xml\n```\n\nNow call via:\n\n```bash\ndigital-eval <path-candidate-root>/domain/ -ref <path-groundtruth>/domain/\n```\n\nfor an aggregated overview on stdout. Feel free to increase verbosity via `-v` (or even `-vv`) to get detailed\ninformation about each single data set which was evaluated.\n\nStructured OCR is considered to contain valid geometrical and textual data on word level, even though for recent PAGE\nalso line level is possible.\n\n### Data problems\n\nInconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONTENT or PAGE without TextEquiv) or\ninvalid geometrical coordinates (less than 3 points or even empty) will lead to evaluation errors if geometry must be\nrespected.\n\n## Additional OCR Utils\n\n### Filter Area\n\nYou can filter a custom area of a page of an OCR file by providing the points of an arbitrary shape.\nThe format of the `-p, --points` argument is `<pt_1_x>,<pt_1_y> <pt_2_x>,<pt_2_y> <pt_3_x>,<pt_3_y> ... <pt_n_x>,<pt_n_y>` . For simple rectangular areas this can be expressed also with two points, with first point as top left and second point as bottom right: `<pt_top_left_x>,<pt_top_left_y> <pt_bottom_right_x>,<pt_bottom_right_y>`.\n\nThe following example filters a rectangular area of 600x400 pixels of a page, which is described by an input ALTO file and saves the result to an output ALTO file\n\n```bash\nocr-util frame -i page_1.alto.xml -p \"0,0 600,0 600,400 0,400\" -o page_1_area.alto.xml\n```\n\nShort version with top left and bottom right:\n\n```bash\nocr-util frame -i page_1.alto.xml -p \"0,0 600,400\" -o page_1_area.alto.xml\n```\n\n## Development\n\nPlattform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 20.04 LTS, Python 3.8.\n\n```bash\n# clone local\ngit clone <repository-url> <local-dir>\ncd <local-dir>\n\n# enable virtual python 3 environment (linux)\n# and update pip itself\npython3 -m venv venv\n. venv/bin/activate\npython -m pip install -U pip\n\n# install\npython -m pip install -e .\n\n# install additional development dependencies\npython -m pip install -r tests/test_requirements.txt\n\n# run tests\npython -m pytest -v\n```\n\n## Contribute\n\nContributions, suggestions and proposals welcome!\n\n## License\n\nUnder terms of the [MIT license](https://opensource.org/licenses/MIT).\n\n**NOTE**: This software depends on packages that _might_ be licensed under different terms.\n",
"bugtrack_url": null,
"license": null,
"summary": "Evaluate Digitalization Data",
"version": "1.6.0",
"project_urls": {
"Homepage": "https://github.com/ulb-sachsen-anhalt/digital-eval"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ae37e1c86024a1bb3937f677298fe50243e6e3bca2395f87f274befd8df652d3",
"md5": "33ffe1a5bc05d0452d84763cf08a84b8",
"sha256": "b3eb1d9f394f22e6bc6ce237695f5699e33a95372445fc57d1511df6a7cbaa03"
},
"downloads": -1,
"filename": "digital_eval-1.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "33ffe1a5bc05d0452d84763cf08a84b8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 26852,
"upload_time": "2024-05-24T17:42:59",
"upload_time_iso_8601": "2024-05-24T17:42:59.403236Z",
"url": "https://files.pythonhosted.org/packages/ae/37/e1c86024a1bb3937f677298fe50243e6e3bca2395f87f274befd8df652d3/digital_eval-1.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "516b6b1567caae7be27b986d824a14a6d7fe634c11fb4fcf239fb8e43e69d972",
"md5": "0d015e073c3d46100d8f8be6b5dcf5d1",
"sha256": "45832128fb6e716a9467beed683daca0a2aacb147ef6a30612cc4fb9409ecef0"
},
"downloads": -1,
"filename": "digital_eval-1.6.0.tar.gz",
"has_sig": false,
"md5_digest": "0d015e073c3d46100d8f8be6b5dcf5d1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 33824,
"upload_time": "2024-05-24T17:43:01",
"upload_time_iso_8601": "2024-05-24T17:43:01.890379Z",
"url": "https://files.pythonhosted.org/packages/51/6b/6b1567caae7be27b986d824a14a6d7fe634c11fb4fcf239fb8e43e69d972/digital_eval-1.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-24 17:43:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ulb-sachsen-anhalt",
"github_project": "digital-eval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "digital-eval"
}
