<p align="center">
<img src="https://user-images.githubusercontent.com/2049665/29219793-b4dcb942-7e7e-11e7-8785-761b0e784e04.png" alt="Wordninja picture" width="128">
<h1 align="center">Wordninja-Enhanced</h1>
<p align="center">
Split your merged words!
<br />
</p>
</p>
<br>
## ℹ About
This is a fork of the popular wordninja repository and improves on it in several aspects.
Language support has been extended to include the following languages out of the box:
- English (en)
- German (de)
- French (fr)
- Italian (it)
- Spanish (es)
- Portuguese (pt)
More functionalities were added aswell:
- A new rejoin() function was created. It splits merged words in a sentence and returns the whole sentence with the corrected words while retaining spacing rules for punctuation characters.
- A candidates() function was added that returns not only one result, but instead several results sorted by their cost.
- It is now possible to specify additional words that should be added to the dictionary or words that should be excluded while initializing the LanguageModel.
- The splitting behaviour can be actively influenced if a different split of a certain word is desired.
- Hyphenated words are now also supported.
- The algorithm now also preserves punctuation while spitting merged words and does no longer break down when encountering unknown characters.
More info about these functionalities can be found further down in the usage section.
## Requirements
- Python 3.9 or higher
## How to Install
```
pip install wordninja-enhanced
```
## Usage
The functionalities are explained in the following code snippet:
```python
import wordninja_enhanced as wordninja
# This function splits merged words
split_text= wordninja.split("Splitthesemergedwordsforme")
print(f"Example 1: {split_text}\n")
# This function returns multiple possible ways to split the input, ranked by lowest cost first.
candidates_list = wordninja.candidates("derekanderson", 3)
print("Example 2:")
for i, candidate in enumerate(candidates_list):
print(f"candidate {i+1}: {candidate}")
print()
# This function splits merged words and returns the correctly splitted string,
# while applying correct spacing rules for punctuation characters.
rejoined_text = wordninja.rejoin("That'sthesheriff's\"badge\" youarewearing!")
print(f"Example 3: {rejoined_text}\n")
# Without any further arguments the default language is set to english
lm = wordninja.LanguageModel()
joined_text = lm.rejoin("Thisisanotherpreviouslymergedtextexample.")
print(f"Example 4: {joined_text}\n")
# Another language can be specified via the language parameter.
lm = wordninja.LanguageModel(language='de')
joined_text = lm.rejoin("Wiegehtesdir?")
print(f"Example 5: {joined_text}\n")
# The LanguageModel also supports adding words to the dictionary. If a word is being split,
# it is likely missing from the dictionary. It can be added via the add_words parameter:
print("Example 6:")
joined_text = wordninja.rejoin("Palaeoloxodonisanextinctgenusofelephant.")
print(f"Before adding the word: {joined_text}")
custom_lm = wordninja.LanguageModel(
language="en",
add_words=['Palaeoloxodon'],
)
joined_text = custom_lm.rejoin("Palaeoloxodonisanextinctgenusofelephant.")
print(f"After adding the word: {joined_text}\n")
# The add_words parameter can also be used to actively influence the splitting behaviour.
# A string like "coinc" gets split with the default dictionary into "coin" and "c".
# If they should instead be split as "co" and "inc", it is possible to move existings words
# in the dictionary, in this case "inc", manually to the top of the dictionary, making it
# more common. This is done via the "overwrite" parameter and specifying add_to_top=True.
print("Example 7:")
joined_text = wordninja.rejoin("coinc")
print(f"Before modifying the dictionary: {joined_text}")
custom_lm = wordninja.LanguageModel(
language="en",
add_words=['inc'],
add_to_top=True,
overwrite=True,
)
joined_text = custom_lm.rejoin("coinc")
print(f"After modifying the dictionary: {joined_text}")
# The LanguageModel also allows the usage of a custom dictionary when the language is specified
# as 'custom'. Words can also be removed from the dictionary in use via a blacklist.
custom_lm = wordninja.LanguageModel(
language='custom',
word_file=r'path\to\your\custom_dict.txt.gz',
add_words=[],
blacklist=[],
add_to_top=True, # Default false
overwrite=False, # Default false
)
```
The output from the examples is the following:
```
Example 1: ['Split', 'these', 'merged', 'words', 'for', 'me']
Example 2:
candidate 1: ['derek', 'anderson']
candidate 2: ['derek', 'anders', 'on']
candidate 3: ['derek', 'and', 'ers', 'on']
Example 3: That's the sheriff's "badge" you are wearing!
Example 4: This is another previously merged text example.
Example 5: Wie geht es dir?
Example 6:
Before adding the word: Palaeo lox odon is an extinct genus of elephant.
After adding the word: Palaeoloxodon is an extinct genus of elephant.
Example 7:
Before modifying the dictionary: coin c
After modifying the dictionary: co inc
```
It can also handle long strings:
```
>>> wordninja.split('wethepeopleoftheunitedstatesinordertoformamoreperfectunionestablishjusticeinsuredomestictranquilityprovideforthecommondefencepromotethegeneralwelfareandsecuretheblessingsoflibertytoourselvesandourposteritydoordainandestablishthisconstitutionfortheunitedstatesofamerica')
['we', 'the', 'people', 'of', 'the', 'united', 'states', 'in', 'order', 'to', 'form', 'a', 'more', 'perfect', 'union', 'establish', 'justice', 'in', 'sure', 'domestic', 'tranquility', 'provide', 'for', 'the', 'common', 'defence', 'promote', 'the', 'general', 'welfare', 'and', 'secure', 'the', 'blessings', 'of', 'liberty', 'to', 'ourselves', 'and', 'our', 'posterity', 'do', 'ordain', 'and', 'establish', 'this', 'constitution', 'for', 'the', 'united', 'states', 'of', 'america']
```
## Further notes
The files and the script to create the dictionaries are also included in the Dictionaries folder.
They can be created by just running the script 'create_dictionaries.py'.
If you are interested in adding support for another language feel free to add your language to the language_config in the script and to create a corresponding corpus folder with the language data.
## Acknowledgements
The dictionaries were created using the Leipzig Corpora Collection. Without their work this project would have not been possible.
D. Goldhahn, T. Eckart & U. Quasthoff: Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.
In: Proceedings of the 8th International Language Resources and Evaluation (LREC'12), 2012
Raw data
{
"_id": null,
"home_page": null,
"name": "wordninja-enhanced",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "text-processing, nlp, language, segmentation",
"author": "Tim Lodemann",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/6f/52/59488dce74b0dc0dce767f06bdda88df6b7b818134909c7c8f7e133a3037/wordninja_enhanced-3.1.0.tar.gz",
"platform": null,
"description": "<p align=\"center\">\r\n<img src=\"https://user-images.githubusercontent.com/2049665/29219793-b4dcb942-7e7e-11e7-8785-761b0e784e04.png\" alt=\"Wordninja picture\" width=\"128\">\r\n <h1 align=\"center\">Wordninja-Enhanced</h1>\r\n <p align=\"center\">\r\n Split your merged words!\r\n <br />\r\n </p>\r\n</p>\r\n\r\n<br>\r\n\r\n## \u2139 About\r\n\r\nThis is a fork of the popular wordninja repository and improves on it in several aspects.\r\n\r\nLanguage support has been extended to include the following languages out of the box:\r\n\r\n- English (en)\r\n- German (de)\r\n- French (fr)\r\n- Italian (it)\r\n- Spanish (es)\r\n- Portuguese (pt)\r\n\r\nMore functionalities were added aswell:\r\n\r\n- A new rejoin() function was created. It splits merged words in a sentence and returns the whole sentence with the corrected words while retaining spacing rules for punctuation characters.\r\n- A candidates() function was added that returns not only one result, but instead several results sorted by their cost.\r\n- It is now possible to specify additional words that should be added to the dictionary or words that should be excluded while initializing the LanguageModel.\r\n- The splitting behaviour can be actively influenced if a different split of a certain word is desired.\r\n- Hyphenated words are now also supported.\r\n- The algorithm now also preserves punctuation while spitting merged words and does no longer break down when encountering unknown characters.\r\n\r\nMore info about these functionalities can be found further down in the usage section.\r\n\r\n## Requirements\r\n\r\n- Python 3.9 or higher\r\n\r\n## How to Install\r\n\r\n```\r\npip install wordninja-enhanced\r\n```\r\n\r\n## Usage\r\n\r\nThe functionalities are explained in the following code snippet:\r\n\r\n```python\r\nimport wordninja_enhanced as wordninja\r\n\r\n# This function splits merged words \r\nsplit_text= wordninja.split(\"Splitthesemergedwordsforme\")\r\nprint(f\"Example 1: {split_text}\\n\")\r\n\r\n# This function returns multiple possible ways to split the input, ranked by lowest cost first.\r\ncandidates_list = wordninja.candidates(\"derekanderson\", 3)\r\nprint(\"Example 2:\")\r\nfor i, candidate in enumerate(candidates_list):\r\n print(f\"candidate {i+1}: {candidate}\")\r\nprint()\r\n\r\n# This function splits merged words and returns the correctly splitted string,\r\n# while applying correct spacing rules for punctuation characters.\r\nrejoined_text = wordninja.rejoin(\"That'sthesheriff's\\\"badge\\\" youarewearing!\")\r\nprint(f\"Example 3: {rejoined_text}\\n\")\r\n\r\n# Without any further arguments the default language is set to english\r\nlm = wordninja.LanguageModel()\r\njoined_text = lm.rejoin(\"Thisisanotherpreviouslymergedtextexample.\")\r\nprint(f\"Example 4: {joined_text}\\n\")\r\n\r\n# Another language can be specified via the language parameter.\r\nlm = wordninja.LanguageModel(language='de')\r\njoined_text = lm.rejoin(\"Wiegehtesdir?\")\r\nprint(f\"Example 5: {joined_text}\\n\")\r\n\r\n# The LanguageModel also supports adding words to the dictionary. If a word is being split,\r\n# it is likely missing from the dictionary. It can be added via the add_words parameter:\r\nprint(\"Example 6:\")\r\njoined_text = wordninja.rejoin(\"Palaeoloxodonisanextinctgenusofelephant.\")\r\nprint(f\"Before adding the word: {joined_text}\")\r\n\r\ncustom_lm = wordninja.LanguageModel(\r\n language=\"en\",\r\n add_words=['Palaeoloxodon'],\r\n)\r\n\r\njoined_text = custom_lm.rejoin(\"Palaeoloxodonisanextinctgenusofelephant.\")\r\nprint(f\"After adding the word: {joined_text}\\n\")\r\n\r\n# The add_words parameter can also be used to actively influence the splitting behaviour.\r\n# A string like \"coinc\" gets split with the default dictionary into \"coin\" and \"c\".\r\n# If they should instead be split as \"co\" and \"inc\", it is possible to move existings words\r\n# in the dictionary, in this case \"inc\", manually to the top of the dictionary, making it\r\n# more common. This is done via the \"overwrite\" parameter and specifying add_to_top=True.\r\nprint(\"Example 7:\")\r\njoined_text = wordninja.rejoin(\"coinc\")\r\nprint(f\"Before modifying the dictionary: {joined_text}\")\r\n\r\ncustom_lm = wordninja.LanguageModel(\r\n language=\"en\",\r\n add_words=['inc'],\r\n add_to_top=True,\r\n overwrite=True,\r\n)\r\n\r\njoined_text = custom_lm.rejoin(\"coinc\")\r\nprint(f\"After modifying the dictionary: {joined_text}\")\r\n\r\n# The LanguageModel also allows the usage of a custom dictionary when the language is specified\r\n# as 'custom'. Words can also be removed from the dictionary in use via a blacklist.\r\ncustom_lm = wordninja.LanguageModel(\r\n language='custom',\r\n word_file=r'path\\to\\your\\custom_dict.txt.gz',\r\n add_words=[],\r\n blacklist=[],\r\n add_to_top=True, # Default false\r\n overwrite=False, # Default false\r\n)\r\n```\r\n\r\nThe output from the examples is the following:\r\n\r\n```\r\nExample 1: ['Split', 'these', 'merged', 'words', 'for', 'me']\r\n\r\nExample 2:\r\ncandidate 1: ['derek', 'anderson']\r\ncandidate 2: ['derek', 'anders', 'on']\r\ncandidate 3: ['derek', 'and', 'ers', 'on']\r\n\r\nExample 3: That's the sheriff's \"badge\" you are wearing!\r\n\r\nExample 4: This is another previously merged text example.\r\n\r\nExample 5: Wie geht es dir?\r\n\r\nExample 6:\r\nBefore adding the word: Palaeo lox odon is an extinct genus of elephant.\r\nAfter adding the word: Palaeoloxodon is an extinct genus of elephant.\r\n\r\nExample 7:\r\nBefore modifying the dictionary: coin c\r\nAfter modifying the dictionary: co inc\r\n```\r\n\r\n\r\nIt can also handle long strings:\r\n```\r\n>>> wordninja.split('wethepeopleoftheunitedstatesinordertoformamoreperfectunionestablishjusticeinsuredomestictranquilityprovideforthecommondefencepromotethegeneralwelfareandsecuretheblessingsoflibertytoourselvesandourposteritydoordainandestablishthisconstitutionfortheunitedstatesofamerica')\r\n['we', 'the', 'people', 'of', 'the', 'united', 'states', 'in', 'order', 'to', 'form', 'a', 'more', 'perfect', 'union', 'establish', 'justice', 'in', 'sure', 'domestic', 'tranquility', 'provide', 'for', 'the', 'common', 'defence', 'promote', 'the', 'general', 'welfare', 'and', 'secure', 'the', 'blessings', 'of', 'liberty', 'to', 'ourselves', 'and', 'our', 'posterity', 'do', 'ordain', 'and', 'establish', 'this', 'constitution', 'for', 'the', 'united', 'states', 'of', 'america']\r\n```\r\n\r\n## Further notes\r\n\r\nThe files and the script to create the dictionaries are also included in the Dictionaries folder.\r\nThey can be created by just running the script 'create_dictionaries.py'.\r\n\r\nIf you are interested in adding support for another language feel free to add your language to the language_config in the script and to create a corresponding corpus folder with the language data.\r\n\r\n\r\n\r\n## Acknowledgements\r\n\r\nThe dictionaries were created using the Leipzig Corpora Collection. Without their work this project would have not been possible.\r\n\r\nD. Goldhahn, T. Eckart & U. Quasthoff: Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.\r\nIn: Proceedings of the 8th International Language Resources and Evaluation (LREC'12), 2012\r\n",
"bugtrack_url": null,
"license": null,
"summary": "Probabilistically split concatenated words. Now with more functionality and languages!",
"version": "3.1.0",
"project_urls": {
"Homepage": "https://github.com/timminator/wordninja-enhanced",
"Issues": "https://github.com/timminator/wordninja-enhanced/issues",
"Repository": "https://github.com/timminator/wordninja-enhanced.git",
"documentation": "https://github.com/timminator/wordninja-enhanced/blob/master/README.md"
},
"split_keywords": [
"text-processing",
" nlp",
" language",
" segmentation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "5fcb66f22c48b6485373d6067cf696ab6e8f1be89f5c8c93996601c0c559bfcc",
"md5": "697288f72b88f69737d14c0a79c112f3",
"sha256": "d312c421fa99a1b4e500680fd59f759691029f171a51a31cb62b7badf038801d"
},
"downloads": -1,
"filename": "wordninja_enhanced-3.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "697288f72b88f69737d14c0a79c112f3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 21555936,
"upload_time": "2025-07-16T20:37:42",
"upload_time_iso_8601": "2025-07-16T20:37:42.848190Z",
"url": "https://files.pythonhosted.org/packages/5f/cb/66f22c48b6485373d6067cf696ab6e8f1be89f5c8c93996601c0c559bfcc/wordninja_enhanced-3.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6f5259488dce74b0dc0dce767f06bdda88df6b7b818134909c7c8f7e133a3037",
"md5": "b739530c551ca36b75ba476192c98c96",
"sha256": "e0a747c885ea3128c05a4edac12b09c02979f8f4f15fcd16f5e971b1fec10cde"
},
"downloads": -1,
"filename": "wordninja_enhanced-3.1.0.tar.gz",
"has_sig": false,
"md5_digest": "b739530c551ca36b75ba476192c98c96",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 21560688,
"upload_time": "2025-07-16T20:37:45",
"upload_time_iso_8601": "2025-07-16T20:37:45.775194Z",
"url": "https://files.pythonhosted.org/packages/6f/52/59488dce74b0dc0dce767f06bdda88df6b7b818134909c7c8f7e133a3037/wordninja_enhanced-3.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-16 20:37:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "timminator",
"github_project": "wordninja-enhanced",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "wordninja-enhanced"
}
JSON
