textmate-grammar-python


Nametextmate-grammar-python JSON
Version 0.6.1 PyPI version JSON
download
home_pagehttps://github.com/watermarkhu/textmate-grammar-python
SummaryA lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python.
upload_time2024-07-31 18:43:24
maintainerNone
docs_urlNone
authorMark Shui Hu
requires_python<4.0,>=3.9
licenseMIT
keywords textmate tokenization
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            [![PyPI - Version](https://img.shields.io/pypi/v/textmate-grammar-python.svg)](https://pypi.python.org/pypi/textmate-grammar-python)
[![PyPI - License](https://img.shields.io/pypi/l/textmate-grammar-python.svg)](https://github.com/watermarkhu/textmate-grammar-python/tree/main?tab=MIT-1-ov-file)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue)](http://mypy-lang.org/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![Python versions](https://img.shields.io/pypi/pyversions/textmate-grammar-python.svg)](https://pypi.python.org/pypi/textmate-grammar-python)
[![CI/CD](https://github.com/watermarkhu/textmate-grammar-python/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/watermarkhu/textmate-grammar-python/blob/main/.github/workflows/ci.yml)
[![readthedocs](https://readthedocs.org/projects/textmate-grammar-python/badge/?version=latest)](https://textmate-grammar-python.readthedocs.io)

# textmate-grammar-python

A lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python. 

Textmate grammars are made for [vscode-texmate](https://github.com/microsoft/vscode-textmate), allowing for syntax highlighting in VSCode after tokenization. This presents textmate-grammar-python with a large list of potentially supported languages. 

## Usage
Install the module with:
```bash
pip install textmate-grammar-python
```

Before tokenization is possible, a `LanguageParser` needs to be initialized using a loaded grammar. 

```python
from textmate_grammar.parsers.matlab import MatlabParser
parser = MatlabParser()
```

After this, one can either choose to call [`parser.parsing_string`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_string) to parse a input string directly, or call [`parser.parse_file`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_file) with the path to the appropiate source file as the first argument, such as in the example [`example.py`](https://github.com/watermarkhu/textmate-grammar-python/blob/main/example.py). 

The parsed `element` object can be displayed directly by calling the [`print`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.elements.html#textmate_grammar.elements.ContentElement.print) method. By default the element is printed as an element tree in a dictionary format. 

```python
>>> element = parser.parse_string("value = num2str(10);")
>>> element.print()

{'token': 'source.matlab',
 'children': [{'token': 'meta.assignment.variable.single.matlab', 
               'children': [{'token': 'variable.other.readwrite.matlab', 'content': 'value'}]},
              {'token': 'keyword.operator.assignment.matlab', 'content': '='},
              {'token': 'meta.function-call.parens.matlab',
               'begin': [{'token': 'entity.name.function.matlab', 'content': 'num2str'},
                         {'token': 'punctuation.section.parens.begin.matlab', 'content': '('}],
               'end': [{'token': 'punctuation.section.parens.end.matlab', 'content': ')'}],
               'children': [{'token': 'constant.numeric.decimal.matlab', 'content': '10'}]},
              {'token': 'punctuation.terminator.semicolon.matlab', 'content': ';'}]}

```
Alternatively, with the keyword argument `flatten` the element is displayed as a list per unique token. Here the first item in the list is the starting position (line, column) of the unique tokenized element. 

```python
>>> element.print(flatten=True)

[[(0, 0), 'value', ['source.matlab', 'meta.assignment.variable.single.matlab', 'variable.other.readwrite.matlab']],
 [(0, 5), ' ', ['source.matlab']],
 [(0, 6), '=', ['source.matlab', 'keyword.operator.assignment.matlab']],
 [(0, 7), ' ', ['source.matlab']],
 [(0, 8), 'num2str', ['source.matlab', 'meta.function-call.parens.matlab', 'entity.name.function.matlab']],
 [(0, 15), '(', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.begin.matlab']],
 [(0, 16), '10', ['source.matlab', 'meta.function-call.parens.matlab', 'constant.numeric.decimal.matlab']],
 [(0, 18), ')', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.end.matlab']],
 [(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]
```

## Information

- For further information, please checkout the [documentation](https://textmate-grammar-python.readthedocs.io/en/latest/). 
- To setup an environment for development, see [CONTRIBUTING.md](https://github.com/watermarkhu/textmate-grammar-python/blob/main/CONTRIBUTING.md)

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/watermarkhu/textmate-grammar-python",
    "name": "textmate-grammar-python",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.9",
    "maintainer_email": null,
    "keywords": "textmate, tokenization",
    "author": "Mark Shui Hu",
    "author_email": "watermarkhu@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/78/73/a188ceb10f89bf6fdc1100a3372f619e2eed779bce5df8d35b6751705138/textmate_grammar_python-0.6.1.tar.gz",
    "platform": null,
    "description": "[![PyPI - Version](https://img.shields.io/pypi/v/textmate-grammar-python.svg)](https://pypi.python.org/pypi/textmate-grammar-python)\n[![PyPI - License](https://img.shields.io/pypi/l/textmate-grammar-python.svg)](https://github.com/watermarkhu/textmate-grammar-python/tree/main?tab=MIT-1-ov-file)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue)](http://mypy-lang.org/)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)\n[![Python versions](https://img.shields.io/pypi/pyversions/textmate-grammar-python.svg)](https://pypi.python.org/pypi/textmate-grammar-python)\n[![CI/CD](https://github.com/watermarkhu/textmate-grammar-python/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/watermarkhu/textmate-grammar-python/blob/main/.github/workflows/ci.yml)\n[![readthedocs](https://readthedocs.org/projects/textmate-grammar-python/badge/?version=latest)](https://textmate-grammar-python.readthedocs.io)\n\n# textmate-grammar-python\n\nA lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python. \n\nTextmate grammars are made for [vscode-texmate](https://github.com/microsoft/vscode-textmate), allowing for syntax highlighting in VSCode after tokenization. This presents textmate-grammar-python with a large list of potentially supported languages. \n\n## Usage\nInstall the module with:\n```bash\npip install textmate-grammar-python\n```\n\nBefore tokenization is possible, a `LanguageParser` needs to be initialized using a loaded grammar. \n\n```python\nfrom textmate_grammar.parsers.matlab import MatlabParser\nparser = MatlabParser()\n```\n\nAfter this, one can either choose to call [`parser.parsing_string`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_string) to parse a input string directly, or call [`parser.parse_file`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_file) with the path to the appropiate source file as the first argument, such as in the example [`example.py`](https://github.com/watermarkhu/textmate-grammar-python/blob/main/example.py). \n\nThe parsed `element` object can be displayed directly by calling the [`print`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.elements.html#textmate_grammar.elements.ContentElement.print) method. By default the element is printed as an element tree in a dictionary format. \n\n```python\n>>> element = parser.parse_string(\"value = num2str(10);\")\n>>> element.print()\n\n{'token': 'source.matlab',\n 'children': [{'token': 'meta.assignment.variable.single.matlab', \n               'children': [{'token': 'variable.other.readwrite.matlab', 'content': 'value'}]},\n              {'token': 'keyword.operator.assignment.matlab', 'content': '='},\n              {'token': 'meta.function-call.parens.matlab',\n               'begin': [{'token': 'entity.name.function.matlab', 'content': 'num2str'},\n                         {'token': 'punctuation.section.parens.begin.matlab', 'content': '('}],\n               'end': [{'token': 'punctuation.section.parens.end.matlab', 'content': ')'}],\n               'children': [{'token': 'constant.numeric.decimal.matlab', 'content': '10'}]},\n              {'token': 'punctuation.terminator.semicolon.matlab', 'content': ';'}]}\n\n```\nAlternatively, with the keyword argument `flatten` the element is displayed as a list per unique token. Here the first item in the list is the starting position (line, column) of the unique tokenized element. \n\n```python\n>>> element.print(flatten=True)\n\n[[(0, 0), 'value', ['source.matlab', 'meta.assignment.variable.single.matlab', 'variable.other.readwrite.matlab']],\n [(0, 5), ' ', ['source.matlab']],\n [(0, 6), '=', ['source.matlab', 'keyword.operator.assignment.matlab']],\n [(0, 7), ' ', ['source.matlab']],\n [(0, 8), 'num2str', ['source.matlab', 'meta.function-call.parens.matlab', 'entity.name.function.matlab']],\n [(0, 15), '(', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.begin.matlab']],\n [(0, 16), '10', ['source.matlab', 'meta.function-call.parens.matlab', 'constant.numeric.decimal.matlab']],\n [(0, 18), ')', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.end.matlab']],\n [(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]\n```\n\n## Information\n\n- For further information, please checkout the [documentation](https://textmate-grammar-python.readthedocs.io/en/latest/). \n- To setup an environment for development, see [CONTRIBUTING.md](https://github.com/watermarkhu/textmate-grammar-python/blob/main/CONTRIBUTING.md)",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "A lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python.",
    "version": "0.6.1",
    "project_urls": {
        "Documentation": "https://textmate-grammar-python.readthedocs.io",
        "Homepage": "https://github.com/watermarkhu/textmate-grammar-python",
        "Repository": "https://github.com/watermarkhu/textmate-grammar-python"
    },
    "split_keywords": [
        "textmate",
        " tokenization"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "92b0b181687b29a014634ffcebbf978906c1927a294558d555c8da322abd5cb4",
                "md5": "4ba158f04ee3e4297f67424c4d67308b",
                "sha256": "f88ccb0e8d6de3279b9dacff6486639b823e87e5102f460371c4525bf02fcb0c"
            },
            "downloads": -1,
            "filename": "textmate_grammar_python-0.6.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4ba158f04ee3e4297f67424c4d67308b",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.9",
            "size": 37471,
            "upload_time": "2024-07-31T18:43:23",
            "upload_time_iso_8601": "2024-07-31T18:43:23.079051Z",
            "url": "https://files.pythonhosted.org/packages/92/b0/b181687b29a014634ffcebbf978906c1927a294558d555c8da322abd5cb4/textmate_grammar_python-0.6.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "7873a188ceb10f89bf6fdc1100a3372f619e2eed779bce5df8d35b6751705138",
                "md5": "b6cdac59da730744eebd3bcfca7e9bc5",
                "sha256": "b1cd0e5a8eb59d003080367409b2c2b4bdeca0d2abec39425bf9e528d8299f92"
            },
            "downloads": -1,
            "filename": "textmate_grammar_python-0.6.1.tar.gz",
            "has_sig": false,
            "md5_digest": "b6cdac59da730744eebd3bcfca7e9bc5",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.9",
            "size": 34072,
            "upload_time": "2024-07-31T18:43:24",
            "upload_time_iso_8601": "2024-07-31T18:43:24.766803Z",
            "url": "https://files.pythonhosted.org/packages/78/73/a188ceb10f89bf6fdc1100a3372f619e2eed779bce5df8d35b6751705138/textmate_grammar_python-0.6.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-07-31 18:43:24",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "watermarkhu",
    "github_project": "textmate-grammar-python",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "textmate-grammar-python"
}
Mark Shui Hu
Elapsed time: 2.29123s