| Name | dphon JSON |
| Version |
2.0.4
JSON |
| download |
| home_page | None |
| Summary | Tools and algorithms for phonology-aware Early Chinese NLP. |
| upload_time | 2024-04-18 04:15:22 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2018 direct-phonology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
old chinese
early chinese
phonology
linguistics
nlp
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
spacy
lingpy
docopt
python-levenshtein
networkx
jsonlines
rich
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
# dphon
[](https://github.com/direct-phonology/dphon/actions?query=workflow%3Aci)
[](https://codecov.io/gh/direct-phonology/dphon)
[](https://zenodo.org/record/4641277)
[](https://spacy.io)
[](https://github.com/psf/black)
## installation
this software is tested on the latest versions of macOS, windows, and ubuntu. you will need a supported version of python (above), along with `pip`.
```sh
$ pip install dphon
```
if you're on windows and are seeing incorrectly formatted output in your terminal, have a look at this [stackoverflow answer](https://stackoverflow.com/questions/49476326/displaying-unicode-in-powershell/49481797#49481797).
## usage
### basics
the main function of `dphon` is to look for instances of text reuse in a corpus of old chinese texts. instead of relying purely on graphemes, it does this by performing grapheme-to-phoneme conversion, and determining possible reuse based on whether passages are likely to have _sounded_ similar (or rhymed) when spoken aloud.
you will need to have files stored locally as utf-8 encoded plain-text (`.txt`) or json-lines (`.jsonl`) format. for the former, one file is assumed to represent one document. for the latter, one file can contain any number of lines, each of which is a document, with required keys `id` (a unique identifier) and `text` (text content) and any number of optional keys. you can obtain a representative corpus of old chinese sourced from the kanseki repository via [`direct-phonology/ect-krp`](https://github.com/direct-phonology/ect-krp).
a simple invocation of `dphon` might look like:
```sh
$ dphon text_a.txt text_b.txt
```
which would look for phonetically similar passages between `text_a` and `text_b`. the output will be a list of sequences and their phonemic transcriptions, with an identifier based on the file's name and an indicator of where in the text the sequence occurs:
```sh
1. text_a (2208–2216)：
夏后啟曰以為可為故為之為之天下弗能
*ləʔ ɢʷraj kʰˤajʔ ɢʷraj kˤaʔs ɢʷraj tə ɢʷraj tə
2. text_b (3340–3348)：
不可弗爲以爲可 故爲之爲之繇其道物
*ləʔ ɢʷraj kʰˤajʔ kˤaʔs ɢʷraj tə ɢʷraj tə pit
```
the numbers next to the identifiers are _token indices_, and may vary depending on how the text is tokenized – `dphon` currently uses character-based tokenization. whitespace will be removed, and the output will be aligned to make it easier to spot differences between the two sequences. by default, insertions are highlighted in green, and mismatches (differences between the two sequences) are highlighted in red. additional (non-matching) context added to either side of match sequences is displayed using a dimmed color (see "advanced usage" below for more information on colorization).
matches are sorted by the ratio of their phomenic similarity to their graphic similarity – in other words, matches between texts that sound highly similar but were written very differently will be at the top of the list.
by default, `dphon` only returns matches that display at least one instance of _graphic variation_ – a case where two different graphemes are used in the same place to represent the same sound. these cases are highlighted in blue. if you're interested in all instances of reuse, regardless of graphic variation, you can use the `--all` flag:
```sh
$ dphon --all text_a.txt text_b.txt
```
you can view the full list of command options with:
```sh
$ dphon --help
```
this tool is under active development, and results may vary. to find the version you are running:
```sh
$ dphon --version
```
### advanced usage
by default, `dphon` uses your system's `$PAGER` to display output, since the results can be quite long. on MacOS and Linux, this will likely be `less`, which supports additional options like searching through the output once it's displayed. for more information, see the man page:
```sh
$ man less
```
`dphon` can colorize output for nicer display in the terminal if your pager supports it. to enable this behavior on MacOS and Linux, set `LESS=R`:
```sh
$ export LESS=R
```
if you want to save the results of the run to a file, you can use redirection. this is useful when writing structured formats like .csv and .jsonl. you can also write html to preserve colors:
```sh
$ dphon -o html files/*.txt > results.html
```
alternatively, you can pipe the output of `dphon` to another utility like `sed` for filtering the results further. for example, you could strip out the ideographic space ` ` from results to remove the alignments:
```sh
$ dphon files/*.txt | sed 's/ //g'
```
## methodology
matching sequences are determined by a "dictionary" file that represents a particular reconstruction of old chinese phonology. these data structures perform grapheme-to-phoneme conversion, yielding the associated sound for each character:
```
"埃": "qˤə"
"哀": "ʔˤəj"
"藹": "qˤats"
...
```
if two characters have the same phonemes, they're treated as a match. for characters with multiple readings, `dphon` currently chooses the first available reading for comparison. more work is planned for version 3.0 to address this shortcoming.
in version 1.0, `dphon`'s default reconstruction was based on Schuessler 2007[<sup>1</sup>](#note1), but used a single "dummy" character to represent all the lexemes in a rhyming group. [the dictionary](dphon/data/sound_table_v1.json) was compiled by John O'Leary ([@valgrinderror](https://github.com/valgrinderror)) and Gian Duri Rominger ([@GDRom](https://github.com/GDRom)). since version 2.0, `dphon` uses [a dictionary](dphon/data/sound_table_v2.json) based on the Baxter-Sagart 2014 reconstruction[<sup>2</sup>](#note2), with additional work by Rominger.
the matching algorithm is based on Paul Vierthaler's [`chinesetextreuse`](https://github.com/vierth/chinesetextreuse) project[<sup>3</sup>](#note3), with some modifications. it uses a [BLAST](<https://en.wikipedia.org/wiki/BLAST_(biotechnology)>)-like strategy to identify initial match candidates, and then extend them via phonetic [edit distance](https://en.wikipedia.org/wiki/Edit_distance) comparison. finally, the results are aligned using a version of the [Smith-Waterman algorithm](https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm) that operates on phonemes, powered by the `lingpy` library[<sup>4</sup>](#note4).
## development setup
first, clone the repository:
```sh
$ git clone https://github.com/direct-phonology/dphon.git
$ cd dphon
```
then, to create and activate a virtual environment (recommended):
```sh
$ python -m venv venv
$ source venv/bin/activate
```
install dependencies:
```sh
$ pip install -r dev-requirements.txt
```
finally, install the package itself in development mode:
```sh
$ pip install -e .
```
now your changes will be automatically picked up when you run `dphon`.
pull requests can be made against `main`.
## code documentation
code documentation is [available on github pages](https://direct-phonology.github.io/dphon) and is generated with `pdoc3`.
to build the docs:
```sh
$ pdoc --html --output-dir docs dphon
```
## tests
unit tests are written with `unittest`. you can run them with:
```sh
$ python -m unittest
```
## releases
the package is built and published to pyPI automatically using `twine` when using GitHub's release functionality.
**make sure the version number in `dphon/__init__.py` is correct!**
<hr/>
<sup id="note1">1</sup> Schuessler, Axel (2007), _ABC Etymological Dictionary of Old Chinese_, Honolulu: University of Hawaii Press, ISBN 978-0-8248-2975-9.
<sup id="note2">2</sup> Baxter, William H.; Sagart, Laurent (2014), _Old Chinese: A New Reconstruction_, Oxford University Press, ISBN 978-0-19-994537-5.
<sup id="note3">3</sup> Vierthaler, Paul, and Mees Gelein. “A BLAST-Based, Language-Agnostic Text Reuse Algorithm with a MARKUS Implementation and Sequence Alignment Optimized for Large Chinese Corpora,” April 26, 2019. https://doi.org/10.31235/osf.io/7xpqe.
<sup id="note4">4</sup> List, Johann-Mattis; Greenhill, Simon; Tresoldi, Tiago; and Forkel, Robert (2019): **LingPy. A Python library for historical linguistics**. Version 2.6.5. URL: http://lingpy.org, DOI: https://zenodo.org/badge/latestdoi/5137/lingpy/lingpy. With contributions by Christoph Rzymski, Gereon Kaiping, Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel. Jena: Max Planck Institute for the Science of Human History.
Raw data
{
"_id": null,
"home_page": null,
"name": "dphon",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "old chinese, early chinese, phonology, linguistics, nlp",
"author": null,
"author_email": "Nick Budak <budak@stanford.edu>, Gian Duri Rominger <gromin@uw.edu>",
"download_url": "https://files.pythonhosted.org/packages/59/52/d4c93d2307bb33f5e4f6eac4eb87e2c14a04460621b8683529d2cd982042/dphon-2.0.4.tar.gz",
"platform": null,
"description": "# dphon\n\n[](https://github.com/direct-phonology/dphon/actions?query=workflow%3Aci)\n[](https://codecov.io/gh/direct-phonology/dphon)\n\n[](https://zenodo.org/record/4641277)\n[](https://spacy.io)\n[](https://github.com/psf/black)\n\n## installation\n\nthis software is tested on the latest versions of macOS, windows, and ubuntu. you will need a supported version of python (above), along with `pip`.\n\n```sh\n$ pip install dphon\n```\n\nif you're on windows and are seeing incorrectly formatted output in your terminal, have a look at this [stackoverflow answer](https://stackoverflow.com/questions/49476326/displaying-unicode-in-powershell/49481797#49481797).\n\n## usage\n\n### basics\n\nthe main function of `dphon` is to look for instances of text reuse in a corpus of old chinese texts. instead of relying purely on graphemes, it does this by performing grapheme-to-phoneme conversion, and determining possible reuse based on whether passages are likely to have _sounded_ similar (or rhymed) when spoken aloud.\n\nyou will need to have files stored locally as utf-8 encoded plain-text (`.txt`) or json-lines (`.jsonl`) format. for the former, one file is assumed to represent one document. for the latter, one file can contain any number of lines, each of which is a document, with required keys `id` (a unique identifier) and `text` (text content) and any number of optional keys. you can obtain a representative corpus of old chinese sourced from the kanseki repository via [`direct-phonology/ect-krp`](https://github.com/direct-phonology/ect-krp).\n\na simple invocation of `dphon` might look like:\n\n```sh\n$ dphon text_a.txt text_b.txt\n```\n\nwhich would look for phonetically similar passages between `text_a` and `text_b`. the output will be a list of sequences and their phonemic transcriptions, with an identifier based on the file's name and an indicator of where in the text the sequence occurs:\n\n```sh\n1. text_a (2208\u20132216)\uff1a\n \u590f\u540e\u555f\u66f0\u4ee5\u70ba\u53ef\u70ba\u6545\u70ba\u4e4b\u70ba\u4e4b\u5929\u4e0b\u5f17\u80fd\n *l\u0259\u0294 \u0262\u02b7raj k\u02b0\u02e4aj\u0294 \u0262\u02b7raj k\u02e4a\u0294s \u0262\u02b7raj t\u0259 \u0262\u02b7raj t\u0259\n2. text_b (3340\u20133348)\uff1a\n \u4e0d\u53ef\u5f17\u7232\u4ee5\u7232\u53ef\u3000\u6545\u7232\u4e4b\u7232\u4e4b\u7e47\u5176\u9053\u7269\n *l\u0259\u0294 \u0262\u02b7raj k\u02b0\u02e4aj\u0294 k\u02e4a\u0294s \u0262\u02b7raj t\u0259 \u0262\u02b7raj t\u0259 pit\n```\n\nthe numbers next to the identifiers are _token indices_, and may vary depending on how the text is tokenized \u2013 `dphon` currently uses character-based tokenization. whitespace will be removed, and the output will be aligned to make it easier to spot differences between the two sequences. by default, insertions are highlighted in green, and mismatches (differences between the two sequences) are highlighted in red. additional (non-matching) context added to either side of match sequences is displayed using a dimmed color (see \"advanced usage\" below for more information on colorization).\n\nmatches are sorted by the ratio of their phomenic similarity to their graphic similarity \u2013\u00a0in other words, matches between texts that sound highly similar but were written very differently will be at the top of the list.\n\nby default, `dphon` only returns matches that display at least one instance of _graphic variation_ \u2013 a case where two different graphemes are used in the same place to represent the same sound. these cases are highlighted in blue. if you're interested in all instances of reuse, regardless of graphic variation, you can use the `--all` flag:\n\n```sh\n$ dphon --all text_a.txt text_b.txt\n```\n\nyou can view the full list of command options with:\n\n```sh\n$ dphon --help\n```\n\nthis tool is under active development, and results may vary. to find the version you are running:\n\n```sh\n$ dphon --version\n```\n\n### advanced usage\n\nby default, `dphon` uses your system's `$PAGER` to display output, since the results can be quite long. on MacOS and Linux, this will likely be `less`, which supports additional options like searching through the output once it's displayed. for more information, see the man page:\n\n```sh\n$ man less\n```\n\n`dphon` can colorize output for nicer display in the terminal if your pager supports it. to enable this behavior on MacOS and Linux, set `LESS=R`:\n\n```sh\n$ export LESS=R\n```\n\nif you want to save the results of the run to a file, you can use redirection. this is useful when writing structured formats like .csv and .jsonl. you can also write html to preserve colors:\n\n```sh\n$ dphon -o html files/*.txt > results.html\n```\n\nalternatively, you can pipe the output of `dphon` to another utility like `sed` for filtering the results further. for example, you could strip out the ideographic space `\u3000` from results to remove the alignments:\n\n```sh\n$ dphon files/*.txt | sed 's/\u3000//g'\n```\n\n## methodology\n\nmatching sequences are determined by a \"dictionary\" file that represents a particular reconstruction of old chinese phonology. these data structures perform grapheme-to-phoneme conversion, yielding the associated sound for each character:\n\n```\n\"\u57c3\": \"q\u02e4\u0259\"\n\"\u54c0\": \"\u0294\u02e4\u0259j\"\n\"\u85f9\": \"q\u02e4ats\"\n...\n```\n\nif two characters have the same phonemes, they're treated as a match. for characters with multiple readings, `dphon` currently chooses the first available reading for comparison. more work is planned for version 3.0 to address this shortcoming.\n\nin version 1.0, `dphon`'s default reconstruction was based on Schuessler 2007[<sup>1</sup>](#note1), but used a single \"dummy\" character to represent all the lexemes in a rhyming group. [the dictionary](dphon/data/sound_table_v1.json) was compiled by John O'Leary ([@valgrinderror](https://github.com/valgrinderror)) and Gian Duri Rominger ([@GDRom](https://github.com/GDRom)). since version 2.0, `dphon` uses [a dictionary](dphon/data/sound_table_v2.json) based on the Baxter-Sagart 2014 reconstruction[<sup>2</sup>](#note2), with additional work by Rominger.\n\nthe matching algorithm is based on Paul Vierthaler's [`chinesetextreuse`](https://github.com/vierth/chinesetextreuse) project[<sup>3</sup>](#note3), with some modifications. it uses a [BLAST](<https://en.wikipedia.org/wiki/BLAST_(biotechnology)>)-like strategy to identify initial match candidates, and then extend them via phonetic [edit distance](https://en.wikipedia.org/wiki/Edit_distance) comparison. finally, the results are aligned using a version of the [Smith-Waterman algorithm](https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm) that operates on phonemes, powered by the `lingpy` library[<sup>4</sup>](#note4).\n\n## development setup\n\nfirst, clone the repository:\n\n```sh\n$ git clone https://github.com/direct-phonology/dphon.git\n$ cd dphon\n```\n\nthen, to create and activate a virtual environment (recommended):\n\n```sh\n$ python -m venv venv\n$ source venv/bin/activate\n```\n\ninstall dependencies:\n\n```sh\n$ pip install -r dev-requirements.txt\n```\n\nfinally, install the package itself in development mode:\n\n```sh\n$ pip install -e .\n```\n\nnow your changes will be automatically picked up when you run `dphon`.\n\npull requests can be made against `main`.\n\n## code documentation\n\ncode documentation is [available on github pages](https://direct-phonology.github.io/dphon) and is generated with `pdoc3`.\n\nto build the docs:\n\n```sh\n$ pdoc --html --output-dir docs dphon\n```\n\n## tests\n\nunit tests are written with `unittest`. you can run them with:\n\n```sh\n$ python -m unittest\n```\n\n## releases\n\nthe package is built and published to pyPI automatically using `twine` when using GitHub's release functionality.\n\n**make sure the version number in `dphon/__init__.py` is correct!**\n\n<hr/>\n<sup id=\"note1\">1</sup> Schuessler, Axel (2007), _ABC Etymological Dictionary of Old Chinese_, Honolulu: University of Hawaii Press, ISBN 978-0-8248-2975-9.\n\n<sup id=\"note2\">2</sup> Baxter, William H.; Sagart, Laurent (2014), _Old Chinese: A New Reconstruction_, Oxford University Press, ISBN 978-0-19-994537-5.\n\n<sup id=\"note3\">3</sup> Vierthaler, Paul, and Mees Gelein. \u201cA BLAST-Based, Language-Agnostic Text Reuse Algorithm with a MARKUS Implementation and Sequence Alignment Optimized for Large Chinese Corpora,\u201d April 26, 2019. https://doi.org/10.31235/osf.io/7xpqe.\n\n<sup id=\"note4\">4</sup> List, Johann-Mattis; Greenhill, Simon; Tresoldi, Tiago; and Forkel, Robert (2019): **LingPy. A Python library for historical linguistics**. Version 2.6.5. URL: http://lingpy.org, DOI: https://zenodo.org/badge/latestdoi/5137/lingpy/lingpy. With contributions by Christoph Rzymski, Gereon Kaiping, Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel. Jena: Max Planck Institute for the Science of Human History.\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2018 direct-phonology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Tools and algorithms for phonology-aware Early Chinese NLP.",
"version": "2.0.4",
"project_urls": {
"Documentation": "https://direct-phonology.github.io/dphon/",
"Issues": "https://github.com/direct-phonology/dphon/issues",
"Repository": "https://github.com/direct-phonology/dphon"
},
"split_keywords": [
"old chinese",
" early chinese",
" phonology",
" linguistics",
" nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c7bb6d146a64cfb1760962243ebb707858d62d9c6953a3dc5b60c195e0a77701",
"md5": "dc68d40559234bb1fe00518097b649cc",
"sha256": "432c03e78c6dfd7c695ab836c9eb40260c8b8f4c82b158286995ad39dc451dff"
},
"downloads": -1,
"filename": "dphon-2.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dc68d40559234bb1fe00518097b649cc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 165035,
"upload_time": "2024-04-18T04:15:21",
"upload_time_iso_8601": "2024-04-18T04:15:21.603082Z",
"url": "https://files.pythonhosted.org/packages/c7/bb/6d146a64cfb1760962243ebb707858d62d9c6953a3dc5b60c195e0a77701/dphon-2.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5952d4c93d2307bb33f5e4f6eac4eb87e2c14a04460621b8683529d2cd982042",
"md5": "ba5c865cfd2c5803db6dd5aacd07b667",
"sha256": "057df332e7f0bb5d0004cb76075ffa95041f1a432b6d4dd035ad3b9d69299d80"
},
"downloads": -1,
"filename": "dphon-2.0.4.tar.gz",
"has_sig": false,
"md5_digest": "ba5c865cfd2c5803db6dd5aacd07b667",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 149621,
"upload_time": "2024-04-18T04:15:22",
"upload_time_iso_8601": "2024-04-18T04:15:22.976691Z",
"url": "https://files.pythonhosted.org/packages/59/52/d4c93d2307bb33f5e4f6eac4eb87e2c14a04460621b8683529d2cd982042/dphon-2.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-18 04:15:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "direct-phonology",
"github_project": "dphon",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "spacy",
"specs": [
[
">=",
"3"
]
]
},
{
"name": "lingpy",
"specs": [
[
"==",
"2.6.13"
]
]
},
{
"name": "docopt",
"specs": [
[
"==",
"0.6.2"
]
]
},
{
"name": "python-levenshtein",
"specs": [
[
"==",
"0.25.1"
]
]
},
{
"name": "networkx",
"specs": [
[
">=",
"2.5"
]
]
},
{
"name": "jsonlines",
"specs": [
[
"==",
"4.0.0"
]
]
},
{
"name": "rich",
"specs": [
[
"==",
"13.7.1"
]
]
}
],
"lcname": "dphon"
}
