| Name | monocleaner JSON |
| Version |
1.6.2
JSON |
| download |
| home_page | |
| Summary | Monolingual corpus fluency filter |
| upload_time | 2023-09-06 11:16:27 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | GNU General Public License v3.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# monocleaner
Monocleaner is a Python tool that aims to detect disfluent sentences in a monolingual corpus.
Each sentence is assigned a fluency score between 0 and 1, with higher scores indicating more fluency.
In addition to a continuous score, several handwritten rules assign a score of 0 to obviously poor sentences.
Although a training tool (`monocleaner-train`) is provided, you may want to use the available ready-to-use language packages.
Please, visit https://github.com/bitextor/monocleaner-data/releases/latest or use `monocleaner-download` to download the latest language packages.
## Citation
If you find Monocleaner useful, please consider citing the following papers:
> V. M. Sánchez-Cartagena, M. Bañón, S. Ortiz-Rojas and G. Ramírez-Sánchez,\
> "[Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task](http://www.statmt.org/wmt18/pdf/WMT116.pdf)",\
>in *Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers*.\
>Brussels, Belgium: Association for Computational Linguistics, October 2018
```latex
@InProceedings{prompsit:2018:WMT,
author = { V\'{i}ctor M. S\'{a}nchez-Cartagena and Marta Ba{\~n}\'{o}n and Sergio Ortiz-Rojas and Gema Ram\'{i}rez-S\'{a}nchez},
title = {Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task},
booktitle = {Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers},
month = {October},
address = {Brussels, Belgium},
publisher = {Association for Computational Linguistics}
}
```
## Installation & Requirements
Monocleaner uses [FastSpell](https://github.com/mbanon/fastspell) that requires `python-dev`:
```bash
sudo apt install python-dev
```
Monocleaner can be installed using `pip`:
```bash
python3 -m pip install monocleaner
```
Monocleaner requires the [KenLM](https://github.com/kpu/kenlm) Python bindings with support for 7-gram language models. You can easily install it by running the following commands:
```bash
git clone https://github.com/kpu/kenlm
cd kenlm
pip install --config-settings="--build-option=--max_order=7" .
mkdir -p build && cd build
cmake .. -DKENLM_MAX_ORDER=7 -DCMAKE_INSTALL_PREFIX:PATH=/your/prefix/path
make -j all install
```
The remaining extra modules required by Monocleaner will be automatically downloaded and installed/upgraded (if required) with the first command.
After installation, two binary files (`monocleaner-train` and `monocleaner`) will be located in your `python/installation/prefix/bin` directory. This is usually `$HOME/.local/bin` or `/usr/local/bin/`.
## Scoring
`monocleaner` aims to detect disfluent sentences in a monolingual corpus.
Each sentence is assigned a fluency score between 0 and 1, with higher scores indicating more fluency.
In addition to a continuous score, several handwritten `hardrules` assign a score of 0 to obviously poor sentences.
The input file (monolingual corpus) must contain one sentence per line text.
The generated output file will contain the same lines adding a column containing the Monocleaner fluency score.
This tool can be run with
```bash
monocleaner [-h]
[--scol SCOL]
[--disable_lang_ident]
[--disable_hardrules]
[--disable_minimal_length]
[--disable_hbs]
[--score_only]
[--annotated_output]
[--add_lang_ident]
[--detect_script]
[--run_all_rules]
[--debug]
[-q]
[-v]
model_dir [input] [output]
```
If input and output are omitted, it will read from stdin and write to stdout.
The output file will contain the following columns which will appear in the strict order indicated below depending on the previous parameters:
| Column | Value | Parameters |
| ------ | ---------------- | ----------------------------- |
| 1 | Sentence | Disabled by --score_only |
| 2 | Score | - |
| 3 | Language Code | Enabled by --add_lang_ident |
| 4 | Hardrule Tag | Enabled by --annotated_output |
### Parameters
* Positional arguments:
* `model_dir`: Directory where the model is stored.
* `input`: Input text file, one sentence per line. When omitted jointly with output, it will read from stdin.
* `output`: Output tab-separated text file adding monocleaner score. When omitted output will be written to stdout.
* Optional arguments:
* `--scol`: Sentence column (starting in 1) (default: 1)
* `--disable_lang_ident`: Disables language identification in hardrules. (default: False)
* `--disable_hardrules`: Disables the hardrules filtering (only monocleaner fluency scoring is applied) (default: False)
* `--disable_minimal_length` : Don't apply minimal length rule (default: False).
* `--disable_hbs`: Don't group Serbo-Croatian under 'hbs' tag. (default: False)
* `--score_only`: Only output one column which is the monocleaner score (default: False)
* `--annotated_output`: Add hardrules annotation for each sentence. (default: False)
* `--add_lang_ident`: Add another column with the identified language if it's not disabled. (default: False)
* `--detect_script`: Detect writing script with FastSpell (only Serbo-Croatian is supported) (default: False)
* `--run_all_rules`: Run all hardrules for each sentence instead of stopping at the first one discarded. (default: False)
* Logging:
* `--debug`: Debug logging mode (default: False)
* `-q, --quiet`: Silent logging mode (default: False)
* `-v, --version`: show version of this script and exit
### Example
```bash
monocleaner models/es mono.es.txt mono.es.scored.txt
```
This will use the Spanish model located at `models/es`, read `mono.es.txt` file and write the sentences to `mono.es.scored.txt` adding the monocleaner score column.
## Monocleaner hard-rules
`monocleaner-hardrules` is an optional pre-filtering step for obvious noise based on rules and incorrect language identified by [FastSpell](https://github.com/mbanon/fastspell). It can be used integrated into the `monocleaner` endpoint, or separately.
### Cleaning
`monocleaner-hardrules` aims at detecting obvious noisey sentences in a monolingual corpus. Sentences that are considered noisy will be tagged with a `0` and the rest will be tagged with a `1`. By default, the input monolingual file must contain at least one column with the sentences needed to be cleaned. If more columns are present, the column index of the sentences desired to be cleaned can be customized via the `--scol` parameter.
By default, the generated output file will contain the same lines and columns that the original input file has, however, an extra column containing the Monocleaner hard-rules score is always added. The amount of newly inserted columns will vary depending on which parameters are enabled.
This tool can be run with:
```bash
monocleaner-hardrules [-h]
[--scol SCOL]
[--disable_lang_ident]
[--disable_minimal_length]
[--disable_hbs]
[--score_only]
[--add_lang_ident]
[--detect_script]
[--annotated_output]
[--run_all_rules]
[--debug]
[-q]
[-v]
language [input] [output]
```
The output file will contain the following columns which will appear in the strict order indicated below depending on the previous parameters:
| Column | Value | Parameters |
| ------ | ---------------- | ----------------------------- |
| 1 | Sentence | Disabled by --score_only |
| 2 | Score | - |
| 3 | Language Code | Enabled by --add_lang_ident |
| 4 | Hardrule Tag | Enabled by --annotated_output |
### Parameters
* Positional arguments:
* `language`: Language code of corpus in ISO 639-1 format (2-char code).
* `input`: Input text file, one sentence per line. When omitted jointly with output, it will read from stdin.
* `output`: Output tab-separated text file adding monocleaner score. When omitted output will be written to stdout.
* Optional arguments:
* `--scol`: Sentence column (starting in 1) (default: 1)
* `--disable_lang_ident`: Disables language identification in hardrules. (default: False)
* `--disable_minimal_length` : Don't apply minimal length rule (default: False).
* `--disable_hbs`: Don't group Serbo-Croatian under 'hbs' tag. (default: False)
* `--score_only`: Only output one column which is the monocleaner score (default: False)
* `--add_lang_ident`: Add another column with the identified language if it's not disabled. (default: False)
* `--detect_script`: Detect writing script with FastSpell (only Serbo-Croatian is supported) (default: False)
* `--annotated_output`: Add hardrules annotation for each sentence. (default: False)
* `--run_all_rules`: Run all hardrules for each sentence instead of stopping at the first one discarded. (default: False)
* Logging:
* `--debug`: Debug logging mode (default: False)
* `-q, --quiet`: Silent logging mode (default: False)
* `-v, --version`: show version of this script and exit
### Example
```bash
monocleaner-hardrules en mono.en.txt mono.en.scored.txt
```
### Understanding annotated output
When using the `--annotated_output` flag, an extra column with each sentence's evaluation is added to the output. If the evaluation returns the `keep` tag (with score column: 1), it means that the sentence is considered good and passed all filters. However, any other tag value (with score column: 0) in the extra column means that the sentence should be rejected. The rejection reasons, their meaning, and the order in which hard-rules are applied, is shown below:
```
no_empty Sentence is empty
no_titles All words in source sentence or target sentence are uppercased or in titlecase
not_too_long Sentence is more than 1024 characters long
not_too_short Sentence is less than 3 words long
no_bad_encoding Source sentence or target sentence contains mojibake
no_only_symbols The ratio of non-alphabetic characters in source sentence is more than 90%
no_only_numbers The ratio of numeric characters in source sentence is too high
no_urls There are URLs (disabled by default)
no_breadcrumbs There are more than 2 breadcrumb characters in the sentence
no_unicode_noise Too many characters from unwanted unicode in source sentence
no_space_noise Too many consecutive single characters separated by spaces in the sentence (excludes digits)
no_paren Too many parenthesis or brackets in sentence
no_literals Unwanted literals: "Re:","{{", "%s", "}}", "+++", "***", '=\"'
no_escaped_unicode There is unescaped unicode characters in sentence
no_glued_words There are words in the sentence containing too many uppercased characters between lowercased characters
no_repeated_words There are more than 1 consecutive words repeated
no_wrong_language Sentence is not in the desired language specifide in the cleaning command
```
___
All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.
Raw data
{
"_id": null,
"home_page": "",
"name": "monocleaner",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Jaume Zaragoza <jzaragoza@prompsit.com>",
"keywords": "",
"author": "",
"author_email": "Prompsit Language Engineering <info@prompsit.com>",
"download_url": "https://files.pythonhosted.org/packages/23/f7/26c28631572378bd7a9166795ededf5bf2b521c879ded0a47b1e12d9d4a1/monocleaner-1.6.2.tar.gz",
"platform": null,
"description": "\n# monocleaner\n\n\n\nMonocleaner is a Python tool that aims to detect disfluent sentences in a monolingual corpus.\nEach sentence is assigned a fluency score between 0 and 1, with higher scores indicating more fluency.\nIn addition to a continuous score, several handwritten rules assign a score of 0 to obviously poor sentences.\n\nAlthough a training tool (`monocleaner-train`) is provided, you may want to use the available ready-to-use language packages.\nPlease, visit https://github.com/bitextor/monocleaner-data/releases/latest or use `monocleaner-download` to download the latest language packages.\n\n## Citation\n\nIf you find Monocleaner useful, please consider citing the following papers:\n\n> V. M. S\u00e1nchez-Cartagena, M. Ba\u00f1\u00f3n, S. Ortiz-Rojas and G. Ram\u00edrez-S\u00e1nchez,\\\n> \"[Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task](http://www.statmt.org/wmt18/pdf/WMT116.pdf)\",\\\n>in *Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers*.\\\n>Brussels, Belgium: Association for Computational Linguistics, October 2018\n\n```latex\n@InProceedings{prompsit:2018:WMT,\n author = { V\\'{i}ctor M. S\\'{a}nchez-Cartagena and Marta Ba{\\~n}\\'{o}n and Sergio Ortiz-Rojas and Gema Ram\\'{i}rez-S\\'{a}nchez},\n title = {Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task},\n booktitle = {Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers},\n month = {October},\n address = {Brussels, Belgium},\n publisher = {Association for Computational Linguistics}\n}\n```\n\n## Installation & Requirements\nMonocleaner uses [FastSpell](https://github.com/mbanon/fastspell) that requires `python-dev`:\n```bash\nsudo apt install python-dev\n```\n\nMonocleaner can be installed using `pip`:\n\n```bash\npython3 -m pip install monocleaner\n```\n\nMonocleaner requires the [KenLM](https://github.com/kpu/kenlm) Python bindings with support for 7-gram language models. You can easily install it by running the following commands:\n\n```bash\ngit clone https://github.com/kpu/kenlm\ncd kenlm\npip install --config-settings=\"--build-option=--max_order=7\" .\nmkdir -p build && cd build\ncmake .. -DKENLM_MAX_ORDER=7 -DCMAKE_INSTALL_PREFIX:PATH=/your/prefix/path\nmake -j all install\n```\n\nThe remaining extra modules required by Monocleaner will be automatically downloaded and installed/upgraded (if required) with the first command.\n\nAfter installation, two binary files (`monocleaner-train` and `monocleaner`) will be located in your `python/installation/prefix/bin` directory. This is usually `$HOME/.local/bin` or `/usr/local/bin/`.\n\n## Scoring\n`monocleaner` aims to detect disfluent sentences in a monolingual corpus.\nEach sentence is assigned a fluency score between 0 and 1, with higher scores indicating more fluency.\nIn addition to a continuous score, several handwritten `hardrules` assign a score of 0 to obviously poor sentences.\n\nThe input file (monolingual corpus) must contain one sentence per line text.\nThe generated output file will contain the same lines adding a column containing the Monocleaner fluency score.\n\nThis tool can be run with\n```bash\nmonocleaner [-h]\n [--scol SCOL]\n [--disable_lang_ident] \n [--disable_hardrules]\n [--disable_minimal_length]\n [--disable_hbs]\n [--score_only]\n [--annotated_output]\n [--add_lang_ident]\n [--detect_script]\n [--run_all_rules]\n [--debug]\n [-q]\n [-v]\n model_dir [input] [output]\n```\nIf input and output are omitted, it will read from stdin and write to stdout.\n\nThe output file will contain the following columns which will appear in the strict order indicated below depending on the previous parameters:\n\n| Column | Value | Parameters |\n| ------ | ---------------- | ----------------------------- | \n| 1 | Sentence | Disabled by --score_only |\n| 2 | Score | - |\n| 3 | Language Code | Enabled by --add_lang_ident |\n| 4 | Hardrule Tag | Enabled by --annotated_output |\n\n### Parameters\n* Positional arguments:\n * `model_dir`: Directory where the model is stored.\n * `input`: Input text file, one sentence per line. When omitted jointly with output, it will read from stdin.\n * `output`: Output tab-separated text file adding monocleaner score. When omitted output will be written to stdout.\n* Optional arguments:\n * `--scol`: Sentence column (starting in 1) (default: 1)\n * `--disable_lang_ident`: Disables language identification in hardrules. (default: False)\n * `--disable_hardrules`: Disables the hardrules filtering (only monocleaner fluency scoring is applied) (default: False)\n * `--disable_minimal_length` : Don't apply minimal length rule (default: False).\n * `--disable_hbs`: Don't group Serbo-Croatian under 'hbs' tag. (default: False)\n * `--score_only`: Only output one column which is the monocleaner score (default: False)\n * `--annotated_output`: Add hardrules annotation for each sentence. (default: False)\n * `--add_lang_ident`: Add another column with the identified language if it's not disabled. (default: False)\n * `--detect_script`: Detect writing script with FastSpell (only Serbo-Croatian is supported) (default: False)\n * `--run_all_rules`: Run all hardrules for each sentence instead of stopping at the first one discarded. (default: False)\n* Logging:\n * `--debug`: Debug logging mode (default: False)\n * `-q, --quiet`: Silent logging mode (default: False)\n * `-v, --version`: show version of this script and exit\n\n### Example\n```bash\nmonocleaner models/es mono.es.txt mono.es.scored.txt\n```\n\nThis will use the Spanish model located at `models/es`, read `mono.es.txt` file and write the sentences to `mono.es.scored.txt` adding the monocleaner score column.\n\n## Monocleaner hard-rules\n`monocleaner-hardrules` is an optional pre-filtering step for obvious noise based on rules and incorrect language identified by [FastSpell](https://github.com/mbanon/fastspell). It can be used integrated into the `monocleaner` endpoint, or separately.\n\n### Cleaning\n`monocleaner-hardrules` aims at detecting obvious noisey sentences in a monolingual corpus. Sentences that are considered noisy will be tagged with a `0` and the rest will be tagged with a `1`. By default, the input monolingual file must contain at least one column with the sentences needed to be cleaned. If more columns are present, the column index of the sentences desired to be cleaned can be customized via the `--scol` parameter.\n\nBy default, the generated output file will contain the same lines and columns that the original input file has, however, an extra column containing the Monocleaner hard-rules score is always added. The amount of newly inserted columns will vary depending on which parameters are enabled.\n\nThis tool can be run with:\n```bash\nmonocleaner-hardrules [-h]\n [--scol SCOL]\n [--disable_lang_ident]\n [--disable_minimal_length]\n [--disable_hbs]\n [--score_only]\n [--add_lang_ident]\n [--detect_script]\n [--annotated_output]\n [--run_all_rules]\n [--debug]\n [-q]\n [-v]\n language [input] [output]\n```\n\nThe output file will contain the following columns which will appear in the strict order indicated below depending on the previous parameters:\n\n| Column | Value | Parameters |\n| ------ | ---------------- | ----------------------------- | \n| 1 | Sentence | Disabled by --score_only |\n| 2 | Score | - |\n| 3 | Language Code | Enabled by --add_lang_ident |\n| 4 | Hardrule Tag | Enabled by --annotated_output |\n\n### Parameters\n* Positional arguments:\n * `language`: Language code of corpus in ISO 639-1 format (2-char code).\n * `input`: Input text file, one sentence per line. When omitted jointly with output, it will read from stdin.\n * `output`: Output tab-separated text file adding monocleaner score. When omitted output will be written to stdout.\n* Optional arguments:\n * `--scol`: Sentence column (starting in 1) (default: 1)\n * `--disable_lang_ident`: Disables language identification in hardrules. (default: False)\n * `--disable_minimal_length` : Don't apply minimal length rule (default: False).\n * `--disable_hbs`: Don't group Serbo-Croatian under 'hbs' tag. (default: False)\n * `--score_only`: Only output one column which is the monocleaner score (default: False)\n * `--add_lang_ident`: Add another column with the identified language if it's not disabled. (default: False)\n * `--detect_script`: Detect writing script with FastSpell (only Serbo-Croatian is supported) (default: False)\n * `--annotated_output`: Add hardrules annotation for each sentence. (default: False)\n * `--run_all_rules`: Run all hardrules for each sentence instead of stopping at the first one discarded. (default: False)\n* Logging:\n * `--debug`: Debug logging mode (default: False)\n * `-q, --quiet`: Silent logging mode (default: False)\n * `-v, --version`: show version of this script and exit\n\n### Example\n```bash\nmonocleaner-hardrules en mono.en.txt mono.en.scored.txt\n```\n\n### Understanding annotated output\nWhen using the `--annotated_output` flag, an extra column with each sentence's evaluation is added to the output. If the evaluation returns the `keep` tag (with score column: 1), it means that the sentence is considered good and passed all filters. However, any other tag value (with score column: 0) in the extra column means that the sentence should be rejected. The rejection reasons, their meaning, and the order in which hard-rules are applied, is shown below:\n\n```\nno_empty\tSentence is empty\nno_titles\tAll words in source sentence or target sentence are uppercased or in titlecase\nnot_too_long\tSentence is more than 1024 characters long\nnot_too_short\tSentence is less than\t3 words long\nno_bad_encoding\tSource sentence or target sentence contains mojibake\nno_only_symbols\tThe ratio of non-alphabetic characters in source sentence is more than 90%\nno_only_numbers\tThe ratio of numeric characters in source sentence is too high\nno_urls\tThere are URLs (disabled by default)\nno_breadcrumbs\tThere are more than 2 breadcrumb characters in the sentence\nno_unicode_noise\tToo many characters from unwanted unicode in source sentence\nno_space_noise\tToo many consecutive single characters separated by spaces in the sentence (excludes digits)\nno_paren\tToo many parenthesis or brackets in sentence\nno_literals\tUnwanted literals: \"Re:\",\"{{\", \"%s\", \"}}\", \"+++\", \"***\", '=\\\"'\nno_escaped_unicode\tThere is unescaped unicode characters in sentence\nno_glued_words\tThere are words in the sentence containing too many uppercased characters between lowercased characters\nno_repeated_words There are more than 1 consecutive words repeated\nno_wrong_language\tSentence is not in the desired language specifide in the cleaning command\n```\n\n\n___\n\n\n\nAll documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.\n",
"bugtrack_url": null,
"license": "GNU General Public License v3.0",
"summary": "Monolingual corpus fluency filter",
"version": "1.6.2",
"project_urls": {
"Homepage": "https://github.com/bitextor/monocleaner",
"Macocu": "https://macocu.eu/",
"Monocleaner on GitHub": "https://github.com/bitextor/monocleaner",
"Prompsit Language Engineering": "http://www.prompsit.com"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "00eddf451d610d6743afb6615e6cf262d9b36f1920dddd5df251c1aa7a1974d3",
"md5": "d9770f606fce75095c3498c6bf026b53",
"sha256": "ac2a813b0b28366d04e89d8b4d3af9d9cfeb4a228447a34bff7a05f28e697c1a"
},
"downloads": -1,
"filename": "monocleaner-1.6.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d9770f606fce75095c3498c6bf026b53",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 34657,
"upload_time": "2023-09-06T11:16:25",
"upload_time_iso_8601": "2023-09-06T11:16:25.595814Z",
"url": "https://files.pythonhosted.org/packages/00/ed/df451d610d6743afb6615e6cf262d9b36f1920dddd5df251c1aa7a1974d3/monocleaner-1.6.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "23f726c28631572378bd7a9166795ededf5bf2b521c879ded0a47b1e12d9d4a1",
"md5": "909710831f945dbba0be21df3156b63a",
"sha256": "7578834645026200ffe8c146056731aaf7d8a941e6e8f740eee78fe73777ee9b"
},
"downloads": -1,
"filename": "monocleaner-1.6.2.tar.gz",
"has_sig": false,
"md5_digest": "909710831f945dbba0be21df3156b63a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 34113,
"upload_time": "2023-09-06T11:16:27",
"upload_time_iso_8601": "2023-09-06T11:16:27.735111Z",
"url": "https://files.pythonhosted.org/packages/23/f7/26c28631572378bd7a9166795ededf5bf2b521c879ded0a47b1e12d9d4a1/monocleaner-1.6.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-06 11:16:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bitextor",
"github_project": "monocleaner",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "monocleaner"
}
