🗽 CityLex: a free English lexical database
===========================================
[](https://pypi.org/project/citylex)
[](https://pypi.org/project/citylex)
[](https://circleci.com/gh/kylebgorman/citylex/tree/master)
CityLex is an English lexical database intended to replace or enhance databases
like [CELEX](https://catalog.ldc.upenn.edu/LDC96L14). It combines data from up
to seven unique sources, including frequency norms, morphological analyses, and
pronunciations. Since these have varying license conditions (some are
proprietary, others restrict redistribution), we do not provide the database as
is. Rather the user must generate a personal copy by executing a Python script,
enabling whatever sources they wish to use.
Building your own CityLex
-------------------------
To install CityLex execute
```bash
pip install citylex
```
To see the available data sources and options, execute `citylex --help`.
To generate the lexicon, execute `citylex` with at least one source enabled
using command-line flags. As most of the data is downloaded from outline
sources, an internet connection is normally required. The process takes roughly
four minutes with all sources enabled; much of the time is spent downloading
large files.
To generate a lexicon with all the sources that don't require manual downloads,
execute
```bash
citylex --all-free
```
File formats
------------
Two files are produced. The first, by default `citylex.tsv`, is a standard
wide-format "tab separated values" (TSV) file, of the sort that can be read into
Excel or R. Some fields (particularly pronunciations and morphological analyses)
can have multiple entries per wordform. In this case, they are separated using
the `^` character.
Advanced users may wish to make use of the second file, by default
`citylex.textproto`, a
[text-format](https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module)
[protocol buffer](https://developers.google.com/protocol-buffers/) which
provides a better representation of the repeated fields. To parse this file in
Python, use the following snippet:
```python
import citylex
lexicon = citylex.read_textproto("citylex.textproto")
```
This will parse the text-format data and populate `lexicon`. One can then
iterate over `lexicon.entry` like a Python dictionary.
Non-redistributable data sources
--------------------------------
Not all CityLex data can be obtained automatically from online sources. If you
wish to enable CELEX features, follow the instructions below.
This proprietary resource must be obtained from the [Linguistic Data
Consortium](https://catalog.ldc.upenn.edu/LDC96L14) as `LDC96L14.tgz`. The file
should be decompressed using
```bash
tar -xzf LDC96L14.tgz
```
This will produce a directory named `celex2`. To enable CELEX2 features, use
`--celex` and pass the local path of this directory as an argument to
`--celex_path`.
For more information
--------------------
- [`citylex.proto`](citylex.proto) for the protocol buffer data structure
- [`citylex.bib`](citylex.bib) for references to the data sources used
For contributors
----------------
To regenerate [`citylex_pb2.py`](citylex_pb2.py) you will need to install the
[Protocol Buffers C++ runtime](https://github.com/protocolbuffers/protobuf)
for your platform, making sure the version number (e.g., the one returned by
`protoc --version` matches that of `protobuf` in `requirements.txt`. Then, run
`protoc --python_out=. citylex.proto`.
License
-------
The CityLex codebase are distributed under the Apache 2.0 license. Please see
[`License.txt`](LICENSE.txt) for details.
All other data sources bear their original licenses chosen by their creators;
see `citylex --help` for more information.
Author
------
CityLex was created by [Kyle Gorman](http://wellformedness.com).
Raw data
{
"_id": null,
"home_page": "",
"name": "citylex",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "computational linguistics,morphology,natural language processing,phonology,phonetics,speech,language",
"author": "Kyle Gorman",
"author_email": "kylebgorman@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/2f/b9/4f6cb165637696003c9ec390c9dcb52c620adced56fa970ec31bbe7fcd68/citylex-0.1.15.tar.gz",
"platform": null,
"description": "\ud83d\uddfd CityLex: a free English lexical database\n===========================================\n\n[](https://pypi.org/project/citylex)\n[](https://pypi.org/project/citylex)\n[](https://circleci.com/gh/kylebgorman/citylex/tree/master)\n\nCityLex is an English lexical database intended to replace or enhance databases\nlike [CELEX](https://catalog.ldc.upenn.edu/LDC96L14). It combines data from up\nto seven unique sources, including frequency norms, morphological analyses, and\npronunciations. Since these have varying license conditions (some are\nproprietary, others restrict redistribution), we do not provide the database as\nis. Rather the user must generate a personal copy by executing a Python script,\nenabling whatever sources they wish to use.\n\nBuilding your own CityLex\n-------------------------\n\nTo install CityLex execute\n\n```bash\npip install citylex\n```\n\nTo see the available data sources and options, execute `citylex --help`.\n\nTo generate the lexicon, execute `citylex` with at least one source enabled\nusing command-line flags. As most of the data is downloaded from outline\nsources, an internet connection is normally required. The process takes roughly\nfour minutes with all sources enabled; much of the time is spent downloading\nlarge files.\n\nTo generate a lexicon with all the sources that don't require manual downloads,\nexecute\n\n```bash\ncitylex --all-free\n```\n\nFile formats\n------------\n\nTwo files are produced. The first, by default `citylex.tsv`, is a standard\nwide-format \"tab separated values\" (TSV) file, of the sort that can be read into\nExcel or R. Some fields (particularly pronunciations and morphological analyses)\ncan have multiple entries per wordform. In this case, they are separated using\nthe `^` character.\n\nAdvanced users may wish to make use of the second file, by default\n`citylex.textproto`, a\n[text-format](https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module)\n[protocol buffer](https://developers.google.com/protocol-buffers/) which\nprovides a better representation of the repeated fields. To parse this file in\nPython, use the following snippet:\n\n```python\nimport citylex\n\nlexicon = citylex.read_textproto(\"citylex.textproto\")\n```\n\nThis will parse the text-format data and populate `lexicon`. One can then\niterate over `lexicon.entry` like a Python dictionary.\n\nNon-redistributable data sources\n--------------------------------\n\nNot all CityLex data can be obtained automatically from online sources. If you\nwish to enable CELEX features, follow the instructions below.\n\nThis proprietary resource must be obtained from the [Linguistic Data\nConsortium](https://catalog.ldc.upenn.edu/LDC96L14) as `LDC96L14.tgz`. The file\nshould be decompressed using\n\n```bash\ntar -xzf LDC96L14.tgz\n```\n\nThis will produce a directory named `celex2`. To enable CELEX2 features, use\n`--celex` and pass the local path of this directory as an argument to\n`--celex_path`.\n\nFor more information\n--------------------\n\n- [`citylex.proto`](citylex.proto) for the protocol buffer data structure\n- [`citylex.bib`](citylex.bib) for references to the data sources used\n\nFor contributors\n----------------\n\nTo regenerate [`citylex_pb2.py`](citylex_pb2.py) you will need to install the\n[Protocol Buffers C++ runtime](https://github.com/protocolbuffers/protobuf)\nfor your platform, making sure the version number (e.g., the one returned by\n`protoc --version` matches that of `protobuf` in `requirements.txt`. Then, run\n`protoc --python_out=. citylex.proto`.\n\nLicense\n-------\n\nThe CityLex codebase are distributed under the Apache 2.0 license. Please see\n[`License.txt`](LICENSE.txt) for details.\n\nAll other data sources bear their original licenses chosen by their creators;\nsee `citylex --help` for more information.\n\nAuthor\n------\n\nCityLex was created by [Kyle Gorman](http://wellformedness.com).\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Builds a multi-source English lexicon",
"version": "0.1.15",
"project_urls": {
"homepage": "https://github.com/kylebgorman/citylex"
},
"split_keywords": [
"computational linguistics",
"morphology",
"natural language processing",
"phonology",
"phonetics",
"speech",
"language"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "46c116cc464e6cb619f347c80ada1e3820cdf8fabeacd1c47150f232893efd67",
"md5": "61c40043e11131ed34dc8274096d14dc",
"sha256": "a042a739dce2842acf61d4cf3640efd6fa4a73404c7f4a80a1d2bf4feba6b174"
},
"downloads": -1,
"filename": "citylex-0.1.15-py3-none-any.whl",
"has_sig": false,
"md5_digest": "61c40043e11131ed34dc8274096d14dc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13574,
"upload_time": "2024-02-20T17:46:51",
"upload_time_iso_8601": "2024-02-20T17:46:51.140051Z",
"url": "https://files.pythonhosted.org/packages/46/c1/16cc464e6cb619f347c80ada1e3820cdf8fabeacd1c47150f232893efd67/citylex-0.1.15-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2fb94f6cb165637696003c9ec390c9dcb52c620adced56fa970ec31bbe7fcd68",
"md5": "d210fd2d907b37243f9697b3ee2457c0",
"sha256": "a24dfe367968eb433e997975c64dde36bb15da7be227c214de0609645001883b"
},
"downloads": -1,
"filename": "citylex-0.1.15.tar.gz",
"has_sig": false,
"md5_digest": "d210fd2d907b37243f9697b3ee2457c0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 13049,
"upload_time": "2024-02-20T17:46:52",
"upload_time_iso_8601": "2024-02-20T17:46:52.243595Z",
"url": "https://files.pythonhosted.org/packages/2f/b9/4f6cb165637696003c9ec390c9dcb52c620adced56fa970ec31bbe7fcd68/citylex-0.1.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-20 17:46:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kylebgorman",
"github_project": "citylex",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"circle": true,
"requirements": [],
"lcname": "citylex"
}
JSON
