| Name | nakdimon JSON |
| Version |
0.1.2
JSON |
| download |
| home_page | None |
| Summary | Diacritic restoration for Hebrew |
| upload_time | 2024-08-03 21:49:44 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | Copyright 2022, Elazar Gershuni Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
numpy
requests
requests-oauthlib
Flask
keras
tensorflowjs
tensorflow
wandb
prettytable
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Nakdimon: a simple Hebrew diacritizer
Repository for the paper [Restoring Hebrew Diacritics Without a Dictionary](https://arxiv.org/abs/2105.05209) by Elazar Gershuni and Yuval Pinter.
Demo: https://nakdimon.org/
## Building and running docker container
Build the docker container:
```
$ docker build -t nakdimon .
```
Run the docker container:
```
$ docker run --rm --gpus all --user 1000:1000 -it nakdimon /bin/bash
```
The `--gpus all` flag is required to run the container with GPU support.
## Training and evaluating
To train, test and evaluate the system, run the following commands:
```
> python nakdimon train --model=models/Nakdimon.h5
> python nakdimon run_test --test_set=tests/new --model=models/Nakdimon.h5
> python nakdimon results --test_set=tests/new --systems Snopi Morfix Dicta MajAllWithDicta Nakdimon
```
The first step trains the model and create a file named `Nakdimon.h5` in the `models` directory.
By default, the model is the one described in the paper: `models/Nakdimon.h5`.
If the model already exists, you may skip this step.
The second step asks the Nakdimon server to predict the diacritics for the test set. You may skip this step.
A folder for the results is created in the chosen test folder, with the same name as the model; in this case, `tests/new/NakdimonNew`.
By default, the test set is the one used in the paper (`tests/new`); you can use `tests/dicta` instead.
If the test results already exist, you may skip this step. If you are not sure, you can use the `--skip_existing` flag.
The third step calculates and prints the results (DEC, CHA, WOR and VOC metrics, as well as OOV_WOR and OOV_VOC).
By default, the systems are the folders in the chosen test folder.
For the Dicta test set (`/tests/dicta`) you should use `MajAllNoDicta` instead of `MajAllWithDicta`, otherwise the vocabulary for the Majority would include the test set itself.
## Diacritizing a single file
```
> python nakdimon predict input_file.txt output_file.txt
```
## Using other systems
You can use the `run_test` command to run the test set on other systems, such as Dicta:
```
> python nakdimon run_test --test_set=tests/new --system=Dicta
```
This will create a folder named `Dicta` for the results in the `tests/new` folder.
Note that `Morfix` cannot be used in this manner, as its license prohibit automatic use.
## Running ablation tests
You can use the `--ablation` flag to train different models for the ablation tests and other experiments:
```
> python nakdimon train --model=models/SingleLayer.h5 --ablation=SingleLayer
```
See the file `ablation.py` for the list of available ablation parameters.
## Important folders
* `hebrew_diacritized` is the training set.
* `tests` contains three tests sets: `new`, `dicta` and `validation`.
Each test set has an `expected` folder that describes the ground truth.
The results of `python nakdimon run_test` are stored in sibling folder, named after the model.
* `models` contains the trained model.
* `nakdimon` holds the source code.
## Citation
```
@inproceedings{gershuni2022restoring,
title={Restoring Hebrew Diacritics Without a Dictionary},
author={Gershuni, Elazar and Pinter, Yuval},
booktitle={Findings of the Association for Computational Linguistics: NAACL 2022},
pages={1010--1018},
year={2022}
}
```
> Gershuni, Elazar, and Yuval Pinter. "Restoring Hebrew Diacritics Without a Dictionary." Findings of the Association for Computational Linguistics: NAACL 2022. 2022.
Raw data
{
"_id": null,
"home_page": null,
"name": "nakdimon",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Elazar Gershuni <elazarg@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/1d/c8/3148685b25c8d780c04d29b1173f0c468e804757b31e1367324ccfe86a17/nakdimon-0.1.2.tar.gz",
"platform": null,
"description": "# Nakdimon: a simple Hebrew diacritizer\n\nRepository for the paper [Restoring Hebrew Diacritics Without a Dictionary](https://arxiv.org/abs/2105.05209) by Elazar Gershuni and Yuval Pinter.\n\nDemo: https://nakdimon.org/\n\n## Building and running docker container\nBuild the docker container:\n```\n$ docker build -t nakdimon .\n```\n\nRun the docker container:\n```\n$ docker run --rm --gpus all --user 1000:1000 -it nakdimon /bin/bash\n```\n\nThe `--gpus all` flag is required to run the container with GPU support.\n\n## Training and evaluating\nTo train, test and evaluate the system, run the following commands:\n```\n> python nakdimon train --model=models/Nakdimon.h5\n> python nakdimon run_test --test_set=tests/new --model=models/Nakdimon.h5\n> python nakdimon results --test_set=tests/new --systems Snopi Morfix Dicta MajAllWithDicta Nakdimon\n```\nThe first step trains the model and create a file named `Nakdimon.h5` in the `models` directory.\nBy default, the model is the one described in the paper: `models/Nakdimon.h5`.\nIf the model already exists, you may skip this step. \n\nThe second step asks the Nakdimon server to predict the diacritics for the test set. You may skip this step.\nA folder for the results is created in the chosen test folder, with the same name as the model; in this case, `tests/new/NakdimonNew`.\nBy default, the test set is the one used in the paper (`tests/new`); you can use `tests/dicta` instead.\nIf the test results already exist, you may skip this step. If you are not sure, you can use the `--skip_existing` flag.\n\nThe third step calculates and prints the results (DEC, CHA, WOR and VOC metrics, as well as OOV_WOR and OOV_VOC).\nBy default, the systems are the folders in the chosen test folder.\nFor the Dicta test set (`/tests/dicta`) you should use `MajAllNoDicta` instead of `MajAllWithDicta`, otherwise the vocabulary for the Majority would include the test set itself.\n\n## Diacritizing a single file\n```\n> python nakdimon predict input_file.txt output_file.txt\n```\n\n## Using other systems\nYou can use the `run_test` command to run the test set on other systems, such as Dicta:\n```\n> python nakdimon run_test --test_set=tests/new --system=Dicta\n```\nThis will create a folder named `Dicta` for the results in the `tests/new` folder.\nNote that `Morfix` cannot be used in this manner, as its license prohibit automatic use.\n\n## Running ablation tests\nYou can use the `--ablation` flag to train different models for the ablation tests and other experiments:\n```\n> python nakdimon train --model=models/SingleLayer.h5 --ablation=SingleLayer\n```\nSee the file `ablation.py` for the list of available ablation parameters.\n\n## Important folders\n* `hebrew_diacritized` is the training set.\n* `tests` contains three tests sets: `new`, `dicta` and `validation`.\n Each test set has an `expected` folder that describes the ground truth.\n The results of `python nakdimon run_test` are stored in sibling folder, named after the model.\n* `models` contains the trained model.\n* `nakdimon` holds the source code.\n\n## Citation\n```\n@inproceedings{gershuni2022restoring,\n title={Restoring Hebrew Diacritics Without a Dictionary},\n author={Gershuni, Elazar and Pinter, Yuval},\n booktitle={Findings of the Association for Computational Linguistics: NAACL 2022},\n pages={1010--1018},\n year={2022}\n}\n```\n> Gershuni, Elazar, and Yuval Pinter. \"Restoring Hebrew Diacritics Without a Dictionary.\" Findings of the Association for Computational Linguistics: NAACL 2022. 2022.\n",
"bugtrack_url": null,
"license": "Copyright 2022, Elazar Gershuni Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Diacritic restoration for Hebrew",
"version": "0.1.2",
"project_urls": {
"Bug Tracker": "https://github.com/elazarg/nakdimon/issues",
"Homepage": "https://github.com/elazarg/nakdimon"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bbb49daaf336ff7e17c0dea891852b082bd66d3c01fcb63e8956ab3bfff0817d",
"md5": "10c84bd3a83492cd4e618cb7727b2a7d",
"sha256": "bc5f7220d468307a0d3b88db61f76b92dc4ee283c56ebebb4138613d183aa0e8"
},
"downloads": -1,
"filename": "nakdimon-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "10c84bd3a83492cd4e618cb7727b2a7d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 59139749,
"upload_time": "2024-08-03T21:49:37",
"upload_time_iso_8601": "2024-08-03T21:49:37.681354Z",
"url": "https://files.pythonhosted.org/packages/bb/b4/9daaf336ff7e17c0dea891852b082bd66d3c01fcb63e8956ab3bfff0817d/nakdimon-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1dc83148685b25c8d780c04d29b1173f0c468e804757b31e1367324ccfe86a17",
"md5": "6c7f1a26f70d7640178c5441a6332c67",
"sha256": "f4d76793325039f2d12f9fc5f9c0bcdf5bbaadf261f16292707f0b7efed7572c"
},
"downloads": -1,
"filename": "nakdimon-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "6c7f1a26f70d7640178c5441a6332c67",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 59136449,
"upload_time": "2024-08-03T21:49:44",
"upload_time_iso_8601": "2024-08-03T21:49:44.346029Z",
"url": "https://files.pythonhosted.org/packages/1d/c8/3148685b25c8d780c04d29b1173f0c468e804757b31e1367324ccfe86a17/nakdimon-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-03 21:49:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "elazarg",
"github_project": "nakdimon",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": [
[
"==",
"1.26.2"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.3"
]
]
},
{
"name": "requests-oauthlib",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "Flask",
"specs": [
[
"==",
"2.2.5"
]
]
},
{
"name": "keras",
"specs": [
[
"==",
"2.15.0"
]
]
},
{
"name": "tensorflowjs",
"specs": [
[
"==",
"4.20.0"
]
]
},
{
"name": "tensorflow",
"specs": [
[
"==",
"2.15.0"
]
]
},
{
"name": "wandb",
"specs": [
[
"==",
"0.17.0"
]
]
},
{
"name": "prettytable",
"specs": [
[
"==",
"3.6.0"
]
]
}
],
"lcname": "nakdimon"
}
