strip-hints
===========
This package provides a command-line command and a corresponding importable
function that strips type hints from Python code files. The stripping process
leaves runnable code, assuming the rest of the code is runnable in the
interpreter version. The program tries to make as few changes as possible to
the processed code so that line and column numbers in error messages for the
processed code file also correspond to those for the original code file. In
most cases, with the default options, both the line and column numbers are
preserved.
This program was originally written to strip hints from Python 2 code to allow
for developing such code on Python 3 but running it with Python 2. Since
Python 2 is no longer maintained the program is no longer tested when run with
Python 2 (but it probably still works).
This project also contains a general-purpose class named ``TokenList`` which
allows lists of Python tokens to be operated on using an interface similar to
that of Python strings. In particular, a ``split`` method is used for much of
the processing in stripping hints. This module could be useful for doing other
things with Python at the token level.
Installing the code
-------------------
To install from PyPI using pip use::
pip install strip-hints
To install the most-recent development version first clone or download the
project from `this GitHub repository
<https://github.com/abarker/strip-hints>`_.
Running the code
----------------
After installing with pip you can run the console script ``strip-hints``::
strip-hints your_file_with_hints.py
The code runs with Python 2 and Python 3. The processed code is written to
stdout. The AST checker that is run on the processed code checks the code
against whatever version of Python the script is run with.
The command-line options are as follows:
``--outfile`` (``-o``)
Write the output to a file with the pathname passed in. Files will be
silently overwritten if they already exist.
If this argument is omitted the output is written to stdout.
``--inplace``
Modify the input code file inplace; code will be replaced with the stripped
code. This is the same as passing in the code file's name as the output file.
``--to-empty``
Map removed code to empty strings rather than spaces. This is easier to read,
but does not preserve columns. Default is false.
``--strip-nl``
Also strip non-logical newline tokens inside type hints. These occur, for
example, when a type-hint function like ``List`` in a function parameter
list has line breaks inside its own arguments list. The default is to keep
the newline tokens in order to preserve line numbers between the stripped
and non-stripped files. Selecting this option no longer guarantees a direct
correspondence.
``--no-ast``
Do not parse the resulting code with the Python ``ast`` module to check it.
Default is false.
``--only-assigns-and-defs``
Only strip annotated assignments and standalone type definitions, keeping
function signature annotations. Python 3.5 and earlier do not implement
these; they first appeared in Python 3.6. The default is false.
``--only-test-for-changes``
Only test if any changes would be made. If any stripping would be done then
it prints ``True`` and exits with code 0. Otherwise it prints ``False`` and
exits with code 1.
``--no-colon-move``
Do not move colons to fix line breaks that occur in the hints for the
function return type. Default is false. See the Limitations section below
for more information. Not recommended.
``--no-equal-move``
Do not move the assignment with ``=`` when needed to fix annotated
assignments that include newlines in the type hints. When they are moved
the total number of lines is kept the same in order to preserve line number
correspondence between the stripped and non-stripped files. If this option
is selected and such a situation occurs an exception is raised. See the
Limitations section below for more information. Not recommended.
If you are using the development repo you can just run the file
``strip_hints.py`` in the ``bin`` directory of the repo::
python strip_hints.py your_file_with_hints.py
Alternately, you can install the development repo with pip::
cd <pathToMainProjectDirectory>
pip install . # use -e for development mode
Automatically running on import
-------------------------------
A function can be called to automatically strip the type hints from all future
imports that are in the same directory as the calling module. For a package
the function call can be placed in ``__init__.py``, for example.
The function can be called as follows, with options set as desired (these
are the default settings):
.. code-block:: python
from strip_hints import strip_on_import
strip_on_import(__file__, to_empty=False, no_ast=False, no_colon_move=False,
only_assigns_and_defs=False, py3_also=False)
By default Python 3 code is ignored unless ``py3_also`` is set. The first
argument is the file path of the calling module.
Calling from a Python program
-----------------------------
To strip the comments from a source file from within a Python program,
returning a string containing the code, the functional interface is as follows.
The option settings here are all the default values:
.. code-block:: python
from strip_hints import strip_file_to_string
code_string = strip_file_to_string(filename, to_empty=False, strip_nl=False,
no_ast=False, no_colon_move=False,
no_equal_move=False,
only_assigns_and_defs=False,
only_test_for_changes=False)
To strip code that is originally in a string, rather than reading from a file,
the function ``strip_string_to_string`` takes the same arguments as
``strip_file_to_string`` except that the first argument is ``code_string``.
If ``only_test_for_changes`` is true then a boolean is returned which is true iff
some changes would be made.
Limitations
-----------
Ordinarily the program simply converts type hints to whitespace and the
resulting code is still syntactically correct. There are a couple of
situations, though, where further transformations are required to preserve
syntactical correctness.
One example is when a line break occurs in the argument list of a type
hint in an annotated assignment:
.. code-block:: python
x: List[int,
int] = [1,2]
The program currently handles this by removing the newlines up to the ``=``
sign. Any comments on those lines are also stripped, since otherwise they
cause syntax errors. Empty lines are added to the end to keep to total number
of lines the same. The ``--no-equal-move`` argument turns this off, in which
case situations like those above raise exceptions. (As a workaround if
necessary to use ``--no-equal-move``, using an explicit backslash line
continuation seems to work.)
A similar situation can occur in return type specifications:
.. code-block:: python
def f() -> List[int,
int]:
pass
This is handled by moving the colon up to the line with the closing paren. The
situation does not occur inside function parameter lists because they are
always nested inside parentheses.
The program currently only handles simple annotated expressions (e.g.,
it handles ``my_class.x: int`` and ``my_list[2]: int`` but not ``(x): int``).
How it works
------------
Rather than doing a full, roundtrip parse, this module works on the tokens
produced by the Python tokenizer. Locating the relevant parts to remove is a
much simpler task than parsing a program in full generality. This allows an ad
hoc approach based on splitting groups of tokens, taking into account the
nesting level of the tokens to potentially split on. Nesting level is based on
the level count inside parentheses, brackets, and curly braces.
* The tokenizer for Python 2 also works on code with type hints, as introduced in
Python 3.
* Type hints can be removed, in most cases, simply by turning some tokens into
whitespace. This preserves line and column numbers in the files. Whiting-out a
section of code with a non-nested line break either raises an exception or
performs a slightly more-complicated transformation.
In the most basic usage the sequence of tokens originally read from the file is
never changed; some tokens just have their string values set to whitespace or
to a pound sign before the untokenize operation.
The gory details of the algorithm are discussed in the docstring for
``strip_hints_main.py``. The method should be fairly robust.
Bugs
----
The code has been run on the Mypy source code and on some other examples, with
the results parsed into ASTs and also visually inspected via diff. Some edge
cases may well remain to cause problems. There is a Bash script in the ``test``
directory which runs the program on files and shows the diffs.
Possible enhancements
---------------------
* Formal tests.
* Better argument-handling, help, etc. with argparse.
* Better error warnings (raising exceptions with messages rather than just failing
assertions in some places).
Raw data
{
"_id": null,
"home_page": "https://github.com/abarker/strip-hints",
"name": "strip-hints",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "type",
"author": "Allen Barker",
"author_email": "Allen.L.Barker@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/9c/d4/312ddce71ee10f7e0ab762afc027e07a918f1c0e1be5b0069db5b0e7542d/strip-hints-0.1.10.tar.gz",
"platform": "",
"description": "\nstrip-hints\n===========\n\nThis package provides a command-line command and a corresponding importable\nfunction that strips type hints from Python code files. The stripping process\nleaves runnable code, assuming the rest of the code is runnable in the\ninterpreter version. The program tries to make as few changes as possible to\nthe processed code so that line and column numbers in error messages for the\nprocessed code file also correspond to those for the original code file. In\nmost cases, with the default options, both the line and column numbers are\npreserved.\n\nThis program was originally written to strip hints from Python 2 code to allow\nfor developing such code on Python 3 but running it with Python 2. Since\nPython 2 is no longer maintained the program is no longer tested when run with\nPython 2 (but it probably still works).\n\nThis project also contains a general-purpose class named ``TokenList`` which\nallows lists of Python tokens to be operated on using an interface similar to\nthat of Python strings. In particular, a ``split`` method is used for much of\nthe processing in stripping hints. This module could be useful for doing other\nthings with Python at the token level.\n\nInstalling the code\n-------------------\n\nTo install from PyPI using pip use::\n\n pip install strip-hints\n\nTo install the most-recent development version first clone or download the\nproject from `this GitHub repository\n<https://github.com/abarker/strip-hints>`_.\n\nRunning the code\n----------------\n\nAfter installing with pip you can run the console script ``strip-hints``::\n\n strip-hints your_file_with_hints.py\n\nThe code runs with Python 2 and Python 3. The processed code is written to\nstdout. The AST checker that is run on the processed code checks the code\nagainst whatever version of Python the script is run with.\n\nThe command-line options are as follows:\n\n``--outfile`` (``-o``)\n Write the output to a file with the pathname passed in. Files will be\n silently overwritten if they already exist.\n If this argument is omitted the output is written to stdout.\n \n``--inplace``\n Modify the input code file inplace; code will be replaced with the stripped\n code. This is the same as passing in the code file's name as the output file.\n\n``--to-empty``\n Map removed code to empty strings rather than spaces. This is easier to read,\n but does not preserve columns. Default is false.\n\n``--strip-nl``\n Also strip non-logical newline tokens inside type hints. These occur, for\n example, when a type-hint function like ``List`` in a function parameter\n list has line breaks inside its own arguments list. The default is to keep\n the newline tokens in order to preserve line numbers between the stripped\n and non-stripped files. Selecting this option no longer guarantees a direct\n correspondence.\n\n``--no-ast``\n Do not parse the resulting code with the Python ``ast`` module to check it.\n Default is false.\n\n``--only-assigns-and-defs``\n Only strip annotated assignments and standalone type definitions, keeping\n function signature annotations. Python 3.5 and earlier do not implement\n these; they first appeared in Python 3.6. The default is false.\n\n``--only-test-for-changes``\n Only test if any changes would be made. If any stripping would be done then\n it prints ``True`` and exits with code 0. Otherwise it prints ``False`` and\n exits with code 1.\n\n``--no-colon-move``\n Do not move colons to fix line breaks that occur in the hints for the\n function return type. Default is false. See the Limitations section below\n for more information. Not recommended.\n\n``--no-equal-move``\n Do not move the assignment with ``=`` when needed to fix annotated\n assignments that include newlines in the type hints. When they are moved\n the total number of lines is kept the same in order to preserve line number\n correspondence between the stripped and non-stripped files. If this option\n is selected and such a situation occurs an exception is raised. See the\n Limitations section below for more information. Not recommended.\n\nIf you are using the development repo you can just run the file\n``strip_hints.py`` in the ``bin`` directory of the repo::\n\n python strip_hints.py your_file_with_hints.py\n\nAlternately, you can install the development repo with pip::\n\n cd <pathToMainProjectDirectory> \n pip install . # use -e for development mode\n\nAutomatically running on import\n-------------------------------\n\nA function can be called to automatically strip the type hints from all future\nimports that are in the same directory as the calling module. For a package\nthe function call can be placed in ``__init__.py``, for example.\n\nThe function can be called as follows, with options set as desired (these\nare the default settings):\n\n.. code-block:: python\n\n from strip_hints import strip_on_import\n strip_on_import(__file__, to_empty=False, no_ast=False, no_colon_move=False,\n only_assigns_and_defs=False, py3_also=False)\n\nBy default Python 3 code is ignored unless ``py3_also`` is set. The first\nargument is the file path of the calling module.\n\nCalling from a Python program\n-----------------------------\n\nTo strip the comments from a source file from within a Python program,\nreturning a string containing the code, the functional interface is as follows.\nThe option settings here are all the default values:\n\n.. code-block:: python\n\n from strip_hints import strip_file_to_string\n code_string = strip_file_to_string(filename, to_empty=False, strip_nl=False,\n no_ast=False, no_colon_move=False,\n no_equal_move=False,\n only_assigns_and_defs=False,\n only_test_for_changes=False)\n\nTo strip code that is originally in a string, rather than reading from a file,\nthe function ``strip_string_to_string`` takes the same arguments as\n``strip_file_to_string`` except that the first argument is ``code_string``.\n\nIf ``only_test_for_changes`` is true then a boolean is returned which is true iff\nsome changes would be made.\n\nLimitations\n-----------\n\nOrdinarily the program simply converts type hints to whitespace and the\nresulting code is still syntactically correct. There are a couple of\nsituations, though, where further transformations are required to preserve\nsyntactical correctness.\n\nOne example is when a line break occurs in the argument list of a type\nhint in an annotated assignment:\n\n.. code-block:: python\n \n x: List[int,\n int] = [1,2]\n\nThe program currently handles this by removing the newlines up to the ``=``\nsign. Any comments on those lines are also stripped, since otherwise they\ncause syntax errors. Empty lines are added to the end to keep to total number\nof lines the same. The ``--no-equal-move`` argument turns this off, in which\ncase situations like those above raise exceptions. (As a workaround if\nnecessary to use ``--no-equal-move``, using an explicit backslash line\ncontinuation seems to work.)\n\nA similar situation can occur in return type specifications:\n\n.. code-block:: python\n\n def f() -> List[int,\n int]:\n pass\n\nThis is handled by moving the colon up to the line with the closing paren. The\nsituation does not occur inside function parameter lists because they are\nalways nested inside parentheses.\n\nThe program currently only handles simple annotated expressions (e.g.,\nit handles ``my_class.x: int`` and ``my_list[2]: int`` but not ``(x): int``).\n\nHow it works\n------------\n\nRather than doing a full, roundtrip parse, this module works on the tokens\nproduced by the Python tokenizer. Locating the relevant parts to remove is a\nmuch simpler task than parsing a program in full generality. This allows an ad\nhoc approach based on splitting groups of tokens, taking into account the\nnesting level of the tokens to potentially split on. Nesting level is based on\nthe level count inside parentheses, brackets, and curly braces.\n\n* The tokenizer for Python 2 also works on code with type hints, as introduced in\n Python 3.\n\n* Type hints can be removed, in most cases, simply by turning some tokens into\n whitespace. This preserves line and column numbers in the files. Whiting-out a\n section of code with a non-nested line break either raises an exception or\n performs a slightly more-complicated transformation.\n\nIn the most basic usage the sequence of tokens originally read from the file is\nnever changed; some tokens just have their string values set to whitespace or\nto a pound sign before the untokenize operation.\n\nThe gory details of the algorithm are discussed in the docstring for\n``strip_hints_main.py``. The method should be fairly robust.\n\nBugs\n----\n\nThe code has been run on the Mypy source code and on some other examples, with\nthe results parsed into ASTs and also visually inspected via diff. Some edge\ncases may well remain to cause problems. There is a Bash script in the ``test``\ndirectory which runs the program on files and shows the diffs.\n\nPossible enhancements\n---------------------\n\n* Formal tests.\n \n* Better argument-handling, help, etc. with argparse.\n\n* Better error warnings (raising exceptions with messages rather than just failing\n assertions in some places).\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Function and command-line program to strip Python type hints.",
"version": "0.1.10",
"split_keywords": [
"type"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "497ee170c5e12d8ba21eef86e70c6c27",
"sha256": "307c2bd147cd35997c8ed2e9a3bdca48ad9c9617e04ea46599095201b4ce998f"
},
"downloads": -1,
"filename": "strip-hints-0.1.10.tar.gz",
"has_sig": false,
"md5_digest": "497ee170c5e12d8ba21eef86e70c6c27",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 29388,
"upload_time": "2021-07-28T02:06:39",
"upload_time_iso_8601": "2021-07-28T02:06:39.223941Z",
"url": "https://files.pythonhosted.org/packages/9c/d4/312ddce71ee10f7e0ab762afc027e07a918f1c0e1be5b0069db5b0e7542d/strip-hints-0.1.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2021-07-28 02:06:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "abarker",
"github_project": "strip-hints",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "strip-hints"
}
JSON
