|Workflows| |Pypi Installs| |Latest Version| |Supported Python Versions|
|Packaging status|
Mathics Character Tables and Scanner
=====================================
This repository really contains two things:
* extensive tables describing WL symbols, operators, and their properties
* a tokenizer or scanner portion for the Wolfram Language.
With respect to the first item, there are two commented YAML files that contain a
full set of translation between:
* Wolfram Language named characters,
* Unicode/ASCII equivalents and Unicode and WL code-points,
* AMSLaTeX equivalent names,
* Operator name (if symbol is an operator),
* Operator arity (if an operator)
* Operator precedence (if an operator)
* Operator associativity (if an operator)
* Keyboard escape sequences for the symbol
Uses
----
The scanner and character tables are used inside `Mathics <https://mathics.org>`_. However information can
also be used by other programs for tokenizing and formatting Wolfram Language code.
For example, tables are used in `mathics-pygments <https://pypi.org/project/Mathics-Scanner/>`_, a Pygments-based
lexer and highlighter for Mathematica/Wolfram Language source code.
This library may be useful if you need to work with Wolfram Language
named character and convert them to various formats.
Usage
-----
- For tokenizing and scanning Wolfram Language code, use the
``mathics_scanner.tokenizer.Tokenizer`` class.
- To convert between Wolfram Language named characters and Unicode/ASCII, use
the ``mathics_scanner.characters.replace_wl_with_plain_text`` and
``mathics_scanner.characters.replace_unicode_with_wl`` functions.
- To convert between qualified names of named characters (such ``FormalA`` for
``\[FormalA]``) and Wolfram's internal representation use the
``m̀athics_scanner.characters.named_characters`` dictionary.
To regenerate JSON-format tables run:
::
$ mathics3-generate-json-table
$ mathics3-generate-operator-json-table
Without options ``mathics3-generate-json-table`` produces the maximum set of correspondences.
In most applications though you may need just a few of these. The
``--field`` option can be used to narrow the list of entries to output in JSON. Run
``mathics3-generate-json-table --help`` for a full list of fields.
Utility for showing token parsing
---------------------------------
There is a command-line utility, ``mathics3-tokens`` for showing how
an input stream is tokenized. The ``--CodeTokenize`` or ``-C`` option
will try to show the token as it would appear using CodeTokenize. type
``mathics3-tokens --help`` information on command-line options.
Implementation
--------------
For notes on the implementation of the packages or details on the conversion
scheme please read `Implementation <https://mathics-scanner.readthedocs.io/en/latest/implementation.html>`_.
Contributing
------------
Please feel encouraged to contribute to this package or Mathics3! Create your
own fork, make the desired changes, commit, and make a pull request.
License
-------
Mathics3 is released under the GNU General Public License Version 3 (GPL3).
.. |Workflows| image:: https://github.com/Mathics3/mathics-scanner/actions/workflows/ubuntu.yml/badge.svg
.. |Packaging status| image:: https://repology.org/badge/vertical-allrepos/mathics-scanner.svg
:target: https://repology.org/project/mathics-scanner/versions
.. |Latest Version| image:: https://badge.fury.io/py/Mathics-Scanner.svg
:target: https://badge.fury.io/py/Mathics-Scanner
.. |Pypi Installs| image:: https://pepy.tech/badge/Mathics-Scanner
.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/Mathics-Scanner.svg
Raw data
{
"_id": null,
"home_page": null,
"name": "Mathics-Scanner",
"maintainer": "Mathics3 Group",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Mathematica, Wolfram, Interpreter, Shell, Math, CAS",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/b4/a9/8410a1d37a360fcd27735b2bdc1ee20dd29744c21365ad71fe92cf6dddff/mathics_scanner-1.4.1.tar.gz",
"platform": null,
"description": "|Workflows| |Pypi Installs| |Latest Version| |Supported Python Versions|\n\n|Packaging status|\n\nMathics Character Tables and Scanner\n=====================================\n\nThis repository really contains two things:\n\n* extensive tables describing WL symbols, operators, and their properties\n* a tokenizer or scanner portion for the Wolfram Language.\n\nWith respect to the first item, there are two commented YAML files that contain a\nfull set of translation between:\n\n* Wolfram Language named characters,\n* Unicode/ASCII equivalents and Unicode and WL code-points,\n* AMSLaTeX equivalent names,\n* Operator name (if symbol is an operator),\n* Operator arity (if an operator)\n* Operator precedence (if an operator)\n* Operator associativity (if an operator)\n* Keyboard escape sequences for the symbol\n\nUses\n----\n\nThe scanner and character tables are used inside `Mathics <https://mathics.org>`_. However information can\nalso be used by other programs for tokenizing and formatting Wolfram Language code.\n\nFor example, tables are used in `mathics-pygments <https://pypi.org/project/Mathics-Scanner/>`_, a Pygments-based\nlexer and highlighter for Mathematica/Wolfram Language source code.\n\nThis library may be useful if you need to work with Wolfram Language\nnamed character and convert them to various formats.\n\nUsage\n-----\n\n- For tokenizing and scanning Wolfram Language code, use the\n ``mathics_scanner.tokenizer.Tokenizer`` class.\n- To convert between Wolfram Language named characters and Unicode/ASCII, use\n the ``mathics_scanner.characters.replace_wl_with_plain_text`` and\n ``mathics_scanner.characters.replace_unicode_with_wl`` functions.\n- To convert between qualified names of named characters (such ``FormalA`` for\n ``\\[FormalA]``) and Wolfram's internal representation use the\n ``m\u0300athics_scanner.characters.named_characters`` dictionary.\n\nTo regenerate JSON-format tables run:\n\n::\n\n $ mathics3-generate-json-table\n $ mathics3-generate-operator-json-table\n\nWithout options ``mathics3-generate-json-table`` produces the maximum set of correspondences.\n\nIn most applications though you may need just a few of these. The\n``--field`` option can be used to narrow the list of entries to output in JSON. Run\n``mathics3-generate-json-table --help`` for a full list of fields.\n\n\nUtility for showing token parsing\n---------------------------------\n\nThere is a command-line utility, ``mathics3-tokens`` for showing how\nan input stream is tokenized. The ``--CodeTokenize`` or ``-C`` option\nwill try to show the token as it would appear using CodeTokenize. type\n``mathics3-tokens --help`` information on command-line options.\n\nImplementation\n--------------\n\nFor notes on the implementation of the packages or details on the conversion\nscheme please read `Implementation <https://mathics-scanner.readthedocs.io/en/latest/implementation.html>`_.\n\nContributing\n------------\n\nPlease feel encouraged to contribute to this package or Mathics3! Create your\nown fork, make the desired changes, commit, and make a pull request.\n\nLicense\n-------\n\nMathics3 is released under the GNU General Public License Version 3 (GPL3).\n\n.. |Workflows| image:: https://github.com/Mathics3/mathics-scanner/actions/workflows/ubuntu.yml/badge.svg\n.. |Packaging status| image:: https://repology.org/badge/vertical-allrepos/mathics-scanner.svg\n\t\t\t :target: https://repology.org/project/mathics-scanner/versions\n.. |Latest Version| image:: https://badge.fury.io/py/Mathics-Scanner.svg\n\t\t :target: https://badge.fury.io/py/Mathics-Scanner\n.. |Pypi Installs| image:: https://pepy.tech/badge/Mathics-Scanner\n.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/Mathics-Scanner.svg\n",
"bugtrack_url": null,
"license": "GPL-3.0-only",
"summary": "Character Tables and Tokenizer for Mathics and the Wolfram Language.",
"version": "1.4.1",
"project_urls": {
"Homepage": "https://mathics.org/"
},
"split_keywords": [
"mathematica",
" wolfram",
" interpreter",
" shell",
" math",
" cas"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fff94e20ccd1a51d6078c640f9a420b5e42415f3c1f7f36c68941215a0066ef4",
"md5": "2b28cd83b60b4bae3439db9933b4e4e3",
"sha256": "5955cf75e3e6b71e30107f5d45d046e9cc45926ac96f0a26049be590abac830a"
},
"downloads": -1,
"filename": "Mathics_Scanner-1.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2b28cd83b60b4bae3439db9933b4e4e3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 153640,
"upload_time": "2025-01-26T17:39:36",
"upload_time_iso_8601": "2025-01-26T17:39:36.419328Z",
"url": "https://files.pythonhosted.org/packages/ff/f9/4e20ccd1a51d6078c640f9a420b5e42415f3c1f7f36c68941215a0066ef4/Mathics_Scanner-1.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b4a98410a1d37a360fcd27735b2bdc1ee20dd29744c21365ad71fe92cf6dddff",
"md5": "18b1ff27645ebb4db0b2c2c2e09edc2e",
"sha256": "a98a5ee11815bf94edf8a6f665e788915c09b9cd4fd96ed4fc26583973dc6fb1"
},
"downloads": -1,
"filename": "mathics_scanner-1.4.1.tar.gz",
"has_sig": false,
"md5_digest": "18b1ff27645ebb4db0b2c2c2e09edc2e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 182389,
"upload_time": "2025-01-26T17:39:37",
"upload_time_iso_8601": "2025-01-26T17:39:37.681599Z",
"url": "https://files.pythonhosted.org/packages/b4/a9/8410a1d37a360fcd27735b2bdc1ee20dd29744c21365ad71fe92cf6dddff/mathics_scanner-1.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-26 17:39:37",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "mathics-scanner"
}
JSON