[](http://applejack.science.ru.nl/languagemachines/)
[](https://www.codacy.com/app/proycon/gecco)
[](https://www.repostatus.org/#inactive)
[](https://zenodo.org/badge/latestdoi/28971190)
========================================================================
GECCO - Generic Environment for Context-Aware Correction of Orthography
=======================================================================
by Maarten van Gompel
Centre for Language and Speech Technology, Radboud University Nijmegen
Sponsored by Revisely (http://revise.ly)
Licensed under the GNU Public License v3
Gecco is a generic modular and distributed framework for spelling correction. Aimed to
build a complete context-aware spelling correction system given your own data
set. Most modules will be language-independent and trainable from a source
corpus. Training is explicitly included in the framework. The framework aims to
easily extendible, modules can be written in Python 3. Moreover, the framework
is scalable and can be distributed over multiple servers.
Given an input text, Gecco will add various suggestions for correction.
The system can be invoked from the command-line, as a Python binding, as a
RESTful webservice, or through the web application (two interfaces).
**Modules**:
- Generic built-in modules:
- **Confusible Module**
- A confusible module is able to discern which version of often
confused word is correct given the context. For example, the words
"then" and "than" are commonly confused in English.
- Your configuration should specify between which confusibles the module disambiguates.
- The module is implemented using the IGTree classifier (a k-Nearest Neighbour
approximation) in Timbl.
- **Suffix Confusible Module**
- A variant of the confusible module that checks commonly confused morphological
suffixes, rather than words.
- Your configuration should specify between which suffixes the module disambiguates
- The module is implemented using the IGTree classifier (a k-Nearest Neighbour
approximation) in Timbl.
- **Language Model Module**
- A language model predicts what words are likely to follow others,
similar to predictive typing applications commonly found on
smartphones.
- The module is implemented using the IGTree classifier (a k-Nearest Neighbour
approximation) in Timbl.
- **Aspell Module**
- Aspell is open-source lexicon-based software for spelling correction.
This module enables aspell to be used from gecco. This is not a
context-sensitive method.
- **Hunspell Module**
- Hunspell is open-source lexicon-based software for spelling correction.
This module enables hunspell to be used from gecco. This is not a
context-sensitive method.
- **Lexicon Module**
- The lexicon module enables you to automatically generate a lexicon
from corpus data and use it. This is not a context-sensitive method.
- Typed words are matched against the lexicon and the module will come
with suggestions within a certain Levenshtein distance.
- **Errorlist Module**
- The errorlist module is a very simple module that checks whether a
word is in a known error list, and if so, provides the suggestions
from that list. This is not a context-sensitive method.
- **Split Module**
- The split module detects words that are split but should be written
together.
- Implemented using Colibri Core
- **Runon Module**
- The runon module detects words that are written as one but should be
split.
- Implemented using Colibri Core
- **Punctuation & Recase Module**
- The punctuation & recase module attempts to detect missing
punctuation, superfluous punctuation, and missing capitals.
- The module is implemented using the IGTree classifier (a k-Nearest Neighbour
approximation) in Timbl.
- Modules suggested but not implemented yet:
- *Language Detection Module*
- (Not written yet, option for later)
- *Sound-alike Module*
- (Not written yet, option for later)
**Features**
- Easily extendible by adding modules using the gecco module API
- Language independent
- Built-in training pipeline (given corpus input): Create models from sources
- Built-in testing pipeline (given an error-annotated test corpus), returns report of evaluation metrics per module
- **Distributed**, **Multithreaded** & **Scalable**:
- Load balancing: backend servers can run on multiple hosts, master process distributes amongst these
- Multithreaded, modules can be invoked in parallel, module servers themselves may be multithreaded too
- Input and output is **FoLiA XML** (http://proycon.github.io/folia)
- Automatic input conversion from plain text using ucto
Gecco is the successor of Valkuil.net and Fowlt.net.
-----------------------
Installation
-----------------------
Gecco relies on a large number of dependencies, including but not limited to:
Dependencies:
- *Generic*:
- python 3.3 or higher
- [PyNLPl](https://github.com/proycon/pynlpl), needed for FoLiA support (https://proycon.github.io/folia)
- [python-ucto](http://proycon.github.com/python-ucto) & [ucto](https://languagemachines.github.io/ucto) (in turn depending on libfolia, ticcutils)
- *Module-specific*:
- [Timbl](https://languagemachines.github.io/timbl) *(mandatory)*
- [python-timbl](https://github.com/proycon/python-timbl)
- [Colibri Core](https://github.com/proycon/colibri-core/) *(mandatory)*
- For the Aspell Module: *(optional)*
- [Aspell](http://aspell.net)
- aspell-python-py3
- For the Hunspell Module: *(optional)*
- [Hunspell](http://hunspell.github.io)
- [PyHunspell](https://github.com/smathot/pyhunspell) *(not supported out of the box on Mac OS X)*
- *Webservice*: *(optional)*
- [CLAM](https://proycon.github.io/clam)
To install Gecco, we *strongly* recommend you to use our LaMachine
distribution, which can be obtained from https://github.com/proycon/lamachine .
LaMachine includes Gecco and can be run in multiple ways: as a virtual machine,
as a docker app, or as a compilation script setting up a Python virtual
environment.
Gecco uses memory-based technologies, and depending on the models you train,
may take up considerable memory. Therefore we recommend *at least* 16GB RAM,
training may require even more. For various modules, model size may be reduced
by increasing frequency thresholds, but this will come at the cost of reduced
accuracy.
Gecco will only run on POSIX-complaint operating systems (i.e. Linux, BSD, Mac OS X), not on Windows.
----------------
Configuration
----------------
To build an actual spelling correction system, you need to have corpus sources
and create a gecco configuration that enable the modules you desire with the
parameters you want.
A Gecco system consists of a configuration, either in the form of a simple Python
script or an external YAML configuration file.
Example YAML configuration:
name: fowlt
path: /path/to/fowlt
language: en
modules:
- module: gecco.modules.confusibles.TIMBLWordConfusibleModule
id: confusibles
source:
- train.txt
model:
- confusible.model
confusibles: [then,than]
To list all available modules and the parameters they may take, run ``gecco --helpmodules``.
Alternatively, the configuration can be done in Python directly, in which case
the script will be the tool that exposes all functionality:
from gecco import Corrector
from gecco.modules.confusibles import TIMBLWordConfusibleModule
corrector = Corrector(id="fowlt", root="/path/to/fowlt/")
corrector.append( TIMBLWordConfusibleModule("thenthan", source="train.txt",test_crossvalidate=True,test=0.1,tune=0.1,model="confusibles.model", confusible=('then','than')))
corrector.append( TIMBLWordConfusibleModule("its", source="train.txt",test_crossvalidate=True,test=0.1,tune=0.1,model="confusibles.model", confusible=('its',"it's")))
corrector.append( TIMBLWordConfusibleModule("errorlist", source="errorlist.txt",model="errorlist.model", servers=[("blah",1234),("blah2",1234)] )
corrector.append( TIMBLWordConfusibleModule("lexicon", source=["lexicon.txt","lexicon2.txt"],model=["lexicon.model","lexicon2.model"], servers=[("blah",1235)] )
corrector.main()
It is recommended to adopt a file/directory structure as described below. If you plan on using multiple hosts, you should store it on a shared network drive so all hosts can access the models:
- yourconfiguration.yml
- sources/
- models/
An example system spelling correction system for English is provided with Gecco and resides in the ``example/`` directory.
----------------
Server setup
----------------
`gecco <yourconfig.yml> run <input.folia.xml>` is executed to process a given
FoLiA document or plaintext document, it starts a master process that will
invoke all the modules, which may be distributed over multiple servers. If
multiple server instances of the same module are available, the load will be
distributed over them. Output will be delivered in the FoLiA XML format and
will contain suggestions for correction.
To start module servers on a host, issue `gecco <yourconfig.yml> startservers`.
You can optionally specify which servers you want to start, if you do not want
to start all. You can start servers multiple times, either on the same or on
multiple hosts. The master process will distribute the load amongst all
servers.
To stop the servers, run `gecco <yourconfig.yml> stopservers` on each host that
has servers running. A list of all running servers can be obtained by `gecco
<yourconfig.yml> listservers`.
Modules can also run locally within the master process rather than as servers,
this is done by either by adding `local: true` in the configuration, or by
adding the ``--local`` option when starting a run. But this will have a
significant negative impact on performance and should therefore be avoided.
-----------------
Architecture
-----------------
---------------------
Command line usage
---------------------
Invoke all gecco functionality through a single command line tool
$ gecco myconfig.yml [subcommand]
or
$ myspellingcorrector.py [subcommand]
Syntax:
usage: gecco [-h]
{run,startservers,stopservers,startserver,train,evaluate,reset}
...
Gecco is a generic, scalable and modular spelling correction framework
Commands:
{run,startservers,stopservers,startserver,train,evaluate,reset}
run Run the spelling corrector on the specified input file
startservers Starts all the module servers that are configured to
run on the current host. Issue once for each host.
stopservers Stops all the module servers that are configured to
run on the current host. Issue once for each host.
listservers Lists all the module servers on all hosts
startserver Start one module's server on the specified port, use
'startservers' instead
train Train modules
evaluate Runs the spelling corrector on input data and compares
it to reference data, produces an evaluation report
reset Reset modules, deletes all trained models that have
sources. Issue prior to train if you want to start
anew.
Vital documentation regarding all modules and the settings they take can be obtained through:
$ gecco --helpmodules
----------------------------------------
Gecco as a webservice
----------------------------------------
RESTUL webservice access will be available through CLAM. We are still working
on better integration of this in Gecco. FOr now, an example implementation of
this can be seen here:
https://github.com/proycon/valkuil-gecco/tree/master/valkuilwebservice
------------------------------
Gecco as a web-application
------------------------------
A web-application will eventually be available, modelled after Valkuil.net/Fowlt.net.
Raw data
{
"_id": null,
"home_page": "https://github.com/proycon/gecco",
"name": "gecco",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "spelling corrector spell check nlp computational_linguistics rest",
"author": "Maarten van Gompel",
"author_email": "proycon@anaproy.nl",
"download_url": "https://files.pythonhosted.org/packages/3d/a4/d7108152c84290d06cc88007c29f4e5a43e953ce9f2000c2bd4dbf24fd98/Gecco-0.3.0.tar.gz",
"platform": "",
"description": "[](http://applejack.science.ru.nl/languagemachines/)\n[](https://www.codacy.com/app/proycon/gecco)\n[](https://www.repostatus.org/#inactive)\n[](https://zenodo.org/badge/latestdoi/28971190)\n\n========================================================================\nGECCO - Generic Environment for Context-Aware Correction of Orthography\n=======================================================================\n\n by Maarten van Gompel\n Centre for Language and Speech Technology, Radboud University Nijmegen\n Sponsored by Revisely (http://revise.ly)\n Licensed under the GNU Public License v3\n\nGecco is a generic modular and distributed framework for spelling correction. Aimed to\nbuild a complete context-aware spelling correction system given your own data\nset. Most modules will be language-independent and trainable from a source\ncorpus. Training is explicitly included in the framework. The framework aims to\neasily extendible, modules can be written in Python 3. Moreover, the framework\nis scalable and can be distributed over multiple servers.\n\nGiven an input text, Gecco will add various suggestions for correction.\n\nThe system can be invoked from the command-line, as a Python binding, as a\nRESTful webservice, or through the web application (two interfaces).\n\n**Modules**:\n - Generic built-in modules:\n - **Confusible Module**\n - A confusible module is able to discern which version of often\n confused word is correct given the context. For example, the words\n \"then\" and \"than\" are commonly confused in English.\n - Your configuration should specify between which confusibles the module disambiguates.\n - The module is implemented using the IGTree classifier (a k-Nearest Neighbour\n approximation) in Timbl.\n - **Suffix Confusible Module**\n - A variant of the confusible module that checks commonly confused morphological\n suffixes, rather than words.\n - Your configuration should specify between which suffixes the module disambiguates\n - The module is implemented using the IGTree classifier (a k-Nearest Neighbour\n approximation) in Timbl.\n - **Language Model Module**\n - A language model predicts what words are likely to follow others,\n similar to predictive typing applications commonly found on\n smartphones.\n - The module is implemented using the IGTree classifier (a k-Nearest Neighbour\n approximation) in Timbl.\n - **Aspell Module**\n - Aspell is open-source lexicon-based software for spelling correction.\n This module enables aspell to be used from gecco. This is not a\n context-sensitive method.\n - **Hunspell Module**\n - Hunspell is open-source lexicon-based software for spelling correction.\n This module enables hunspell to be used from gecco. This is not a\n context-sensitive method.\n - **Lexicon Module**\n - The lexicon module enables you to automatically generate a lexicon\n from corpus data and use it. This is not a context-sensitive method.\n - Typed words are matched against the lexicon and the module will come\n with suggestions within a certain Levenshtein distance.\n - **Errorlist Module**\n - The errorlist module is a very simple module that checks whether a\n word is in a known error list, and if so, provides the suggestions\n from that list. This is not a context-sensitive method.\n - **Split Module**\n - The split module detects words that are split but should be written\n together.\n - Implemented using Colibri Core\n - **Runon Module**\n - The runon module detects words that are written as one but should be\n split.\n - Implemented using Colibri Core\n - **Punctuation & Recase Module**\n - The punctuation & recase module attempts to detect missing\n punctuation, superfluous punctuation, and missing capitals.\n - The module is implemented using the IGTree classifier (a k-Nearest Neighbour\n approximation) in Timbl.\n - Modules suggested but not implemented yet:\n - *Language Detection Module*\n - (Not written yet, option for later)\n - *Sound-alike Module*\n - (Not written yet, option for later)\n\n**Features**\n - Easily extendible by adding modules using the gecco module API\n - Language independent\n - Built-in training pipeline (given corpus input): Create models from sources\n - Built-in testing pipeline (given an error-annotated test corpus), returns report of evaluation metrics per module\n - **Distributed**, **Multithreaded** & **Scalable**:\n - Load balancing: backend servers can run on multiple hosts, master process distributes amongst these\n - Multithreaded, modules can be invoked in parallel, module servers themselves may be multithreaded too\n - Input and output is **FoLiA XML** (http://proycon.github.io/folia)\n - Automatic input conversion from plain text using ucto\n\nGecco is the successor of Valkuil.net and Fowlt.net.\n\n-----------------------\nInstallation\n-----------------------\n\nGecco relies on a large number of dependencies, including but not limited to:\n\nDependencies:\n - *Generic*:\n - python 3.3 or higher\n - [PyNLPl](https://github.com/proycon/pynlpl), needed for FoLiA support (https://proycon.github.io/folia)\n - [python-ucto](http://proycon.github.com/python-ucto) & [ucto](https://languagemachines.github.io/ucto) (in turn depending on libfolia, ticcutils)\n - *Module-specific*:\n - [Timbl](https://languagemachines.github.io/timbl) *(mandatory)*\n - [python-timbl](https://github.com/proycon/python-timbl)\n - [Colibri Core](https://github.com/proycon/colibri-core/) *(mandatory)*\n - For the Aspell Module: *(optional)*\n - [Aspell](http://aspell.net)\n - aspell-python-py3\n - For the Hunspell Module: *(optional)*\n - [Hunspell](http://hunspell.github.io)\n - [PyHunspell](https://github.com/smathot/pyhunspell) *(not supported out of the box on Mac OS X)*\n - *Webservice*: *(optional)*\n - [CLAM](https://proycon.github.io/clam)\n\nTo install Gecco, we *strongly* recommend you to use our LaMachine\ndistribution, which can be obtained from https://github.com/proycon/lamachine .\n\nLaMachine includes Gecco and can be run in multiple ways: as a virtual machine,\nas a docker app, or as a compilation script setting up a Python virtual\nenvironment.\n\nGecco uses memory-based technologies, and depending on the models you train,\nmay take up considerable memory. Therefore we recommend *at least* 16GB RAM,\ntraining may require even more. For various modules, model size may be reduced\nby increasing frequency thresholds, but this will come at the cost of reduced\naccuracy.\n\nGecco will only run on POSIX-complaint operating systems (i.e. Linux, BSD, Mac OS X), not on Windows.\n\n----------------\nConfiguration\n----------------\n\nTo build an actual spelling correction system, you need to have corpus sources\nand create a gecco configuration that enable the modules you desire with the\nparameters you want.\n\nA Gecco system consists of a configuration, either in the form of a simple Python\nscript or an external YAML configuration file.\n\nExample YAML configuration:\n\n name: fowlt\n path: /path/to/fowlt\n language: en\n modules:\n - module: gecco.modules.confusibles.TIMBLWordConfusibleModule\n id: confusibles\n source:\n - train.txt\n model:\n - confusible.model\n confusibles: [then,than]\n\nTo list all available modules and the parameters they may take, run ``gecco --helpmodules``.\n\nAlternatively, the configuration can be done in Python directly, in which case\nthe script will be the tool that exposes all functionality:\n\n from gecco import Corrector\n from gecco.modules.confusibles import TIMBLWordConfusibleModule\n\n\tcorrector = Corrector(id=\"fowlt\", root=\"/path/to/fowlt/\")\n\tcorrector.append( TIMBLWordConfusibleModule(\"thenthan\", source=\"train.txt\",test_crossvalidate=True,test=0.1,tune=0.1,model=\"confusibles.model\", confusible=('then','than')))\n\tcorrector.append( TIMBLWordConfusibleModule(\"its\", source=\"train.txt\",test_crossvalidate=True,test=0.1,tune=0.1,model=\"confusibles.model\", confusible=('its',\"it's\")))\n\tcorrector.append( TIMBLWordConfusibleModule(\"errorlist\", source=\"errorlist.txt\",model=\"errorlist.model\", servers=[(\"blah\",1234),(\"blah2\",1234)] )\n\tcorrector.append( TIMBLWordConfusibleModule(\"lexicon\", source=[\"lexicon.txt\",\"lexicon2.txt\"],model=[\"lexicon.model\",\"lexicon2.model\"], servers=[(\"blah\",1235)] )\n\tcorrector.main()\n\n\nIt is recommended to adopt a file/directory structure as described below. If you plan on using multiple hosts, you should store it on a shared network drive so all hosts can access the models:\n\n - yourconfiguration.yml\n - sources/\n - models/\n\nAn example system spelling correction system for English is provided with Gecco and resides in the ``example/`` directory.\n\n\n\n----------------\nServer setup\n----------------\n\n`gecco <yourconfig.yml> run <input.folia.xml>` is executed to process a given\nFoLiA document or plaintext document, it starts a master process that will\ninvoke all the modules, which may be distributed over multiple servers. If\nmultiple server instances of the same module are available, the load will be\ndistributed over them. Output will be delivered in the FoLiA XML format and\nwill contain suggestions for correction.\n\nTo start module servers on a host, issue `gecco <yourconfig.yml> startservers`.\nYou can optionally specify which servers you want to start, if you do not want\nto start all. You can start servers multiple times, either on the same or on\nmultiple hosts. The master process will distribute the load amongst all\nservers.\n\nTo stop the servers, run `gecco <yourconfig.yml> stopservers` on each host that\nhas servers running. A list of all running servers can be obtained by `gecco\n<yourconfig.yml> listservers`.\n\nModules can also run locally within the master process rather than as servers,\nthis is done by either by adding `local: true` in the configuration, or by\nadding the ``--local`` option when starting a run. But this will have a\nsignificant negative impact on performance and should therefore be avoided.\n\n-----------------\nArchitecture\n-----------------\n\n![Gecco Architecture](https://raw.github.com/proycon/gecco/master/gecco_architecture.png \"Gecco Architecture\")\n\n---------------------\nCommand line usage\n---------------------\n\nInvoke all gecco functionality through a single command line tool\n\n $ gecco myconfig.yml [subcommand]\n\nor\n\n $ myspellingcorrector.py [subcommand]\n\n\nSyntax:\n\n usage: gecco [-h]\n {run,startservers,stopservers,startserver,train,evaluate,reset}\n ...\n\n Gecco is a generic, scalable and modular spelling correction framework\n\n Commands:\n {run,startservers,stopservers,startserver,train,evaluate,reset}\n run Run the spelling corrector on the specified input file\n startservers Starts all the module servers that are configured to\n run on the current host. Issue once for each host.\n stopservers Stops all the module servers that are configured to\n run on the current host. Issue once for each host.\n listservers Lists all the module servers on all hosts\n startserver Start one module's server on the specified port, use\n 'startservers' instead\n train Train modules\n evaluate Runs the spelling corrector on input data and compares\n it to reference data, produces an evaluation report\n reset Reset modules, deletes all trained models that have\n sources. Issue prior to train if you want to start\n anew.\n\n\nVital documentation regarding all modules and the settings they take can be obtained through:\n\n $ gecco --helpmodules\n\n----------------------------------------\nGecco as a webservice\n----------------------------------------\n\nRESTUL webservice access will be available through CLAM. We are still working\non better integration of this in Gecco. FOr now, an example implementation of\nthis can be seen here:\nhttps://github.com/proycon/valkuil-gecco/tree/master/valkuilwebservice\n\n------------------------------\nGecco as a web-application\n------------------------------\n\nA web-application will eventually be available, modelled after Valkuil.net/Fowlt.net.",
"bugtrack_url": null,
"license": "GPL",
"summary": "Generic Environment for Context-Aware Correction of Orthography",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/proycon/gecco"
},
"split_keywords": [
"spelling",
"corrector",
"spell",
"check",
"nlp",
"computational_linguistics",
"rest"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3da4d7108152c84290d06cc88007c29f4e5a43e953ce9f2000c2bd4dbf24fd98",
"md5": "566139369c88d629ebd9e050fb46ca53",
"sha256": "6e4db39e6fa0d69229893a3e9732f6997f1eb626684d823ba572bdd9aac6d855"
},
"downloads": -1,
"filename": "Gecco-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "566139369c88d629ebd9e050fb46ca53",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 64180,
"upload_time": "2020-07-11T11:30:58",
"upload_time_iso_8601": "2020-07-11T11:30:58.129055Z",
"url": "https://files.pythonhosted.org/packages/3d/a4/d7108152c84290d06cc88007c29f4e5a43e953ce9f2000c2bd4dbf24fd98/Gecco-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-07-11 11:30:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "proycon",
"github_project": "gecco",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "Cython",
"specs": [
[
">=",
"0.20"
]
]
},
{
"name": "lxml",
"specs": [
[
">=",
"2.2"
]
]
},
{
"name": "pynlpl",
"specs": [
[
">=",
"0.7.6"
]
]
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "colibricore",
"specs": [
[
">=",
"0.5.9"
]
]
},
{
"name": "python3-timbl",
"specs": []
},
{
"name": "aspell-python-py3",
"specs": []
}
],
"lcname": "gecco"
}
JSON
