pyastgrep
=========
.. image:: https://badge.fury.io/py/pyastgrep.svg
:target: https://badge.fury.io/py/pyastgrep
.. image:: https://github.com/spookylukey/pyastgrep/actions/workflows/tests.yml/badge.svg
:target: https://github.com/spookylukey/pyastgrep/actions/workflows/tests.yml
A command-line utility for grepping Python files using XPath syntax (or CSS
selectors) against the Python AST (Abstract Syntax Tree).
In other words, this allows you to search Python code against specific syntax
elements (function definitions, arguments, assignments, variables etc), instead
of grepping for string matches.
The interface and behaviour is designed to match grep and ripgrep as far as it
makes sense to do so.
.. contents:: Contents
Installation
------------
Python 3.8+ required.
We recommend `pipx <https://pipxproject.github.io/pipx/>`_ to install it
conveniently in an isolated environment:
::
pipx install pyastgrep
You can also use pip:
::
pip install pyastgrep
Basic usage
-----------
::
pyastgrep <xpath expression> [files or directories]
See ``pyastgrep --help`` for more options.
pyastgrep in action
-------------------
Here is an example showing everywhere in the ``pyastgrep`` code base that uses a
variable whose name contains ``idx``:
.. image:: https://raw.githubusercontent.com/spookylukey/pyastgrep/master/pyastgrep_screenshot.png
:align: center
:width: 800
:alt: Screenshot showing a terminal running ``pyastgrep './/Name[contains(@id, 'idx')]’``
Understanding the XML structure
-------------------------------
To get started, you’ll need some understanding of how Python AST is structured,
and how that is mapped to XML. Some methods for doing that are below:
1. Use `Python AST Explorer <https://python-ast-explorer.com/>`_ to play around
with what AST looks like.
2. Dump out the AST and/or XML structure of the top-level statements in a Python
file. The easiest way to do this is to use the provided ``pyastdump``
command, passing in either a Python filename, ``pyastdump yourfile.py``, or
piping in Python fragments as below:
.. code:: bash
$ echo 'x = 1' | pyastdump -
<Module>
<body>
<Assign lineno="1" col_offset="0">
<targets>
<Name lineno="1" col_offset="0" type="str" id="x">
<ctx>
<Store/>
</ctx>
</Name>
</targets>
<value>
<Constant lineno="1" col_offset="4" type="int" value="1"/>
</value>
</Assign>
</body>
<type_ignores/>
</Module>
(When piping input in this way, code will be automatically dedented, making
this easier to do from partial Python snippets.)
You can also use the ``pyastgrep`` command, but since the top-level XML
elements are ``<Module><body>``, and don’t correspond to actual source lines,
you’ll need to use an XPath expression ``./*/*`` to get a match for each
statement within the body, and pass ``--xml`` and/or ``--ast`` to dump the
XML/AST structure:
.. code:: bash
$ pyastgrep --xml --ast './*/*' myfile.py
myfile.py:1:1:import os
Import(
lineno=1,
col_offset=0,
end_lineno=1,
end_col_offset=9,
names=[alias(lineno=1, col_offset=7, end_lineno=1, end_col_offset=9, name='os', asname=None)],
)
<Import lineno="1" col_offset="0">
<names>
<alias lineno="1" col_offset="7" type="str" name="os"/>
</names>
</Import>
...
Note that the XML format is a very direct translation of the Python AST as
produced by the `ast module <https://docs.python.org/3/library/ast.html>`_ (with
some small additions made to improve usability for a few cases). This AST is not
stable across Python versions, so the XML is not stable either. Normally changes
in the AST correspond to new syntax that is added to Python, but in some cases a
new Python version will make significant changes made to the AST generated for
the same code.
You’ll also need some understanding of how to write XPath expressions (see links
at the bottom), but the examples below should get you started.
Examples
--------
Usages of a function called ``open``:
.. code:: bash
$ pyastgrep './/Call/func/Name[@id="open"]'
src/pyastgrep/search.py:88:18: with open(path) as f:
Literal numbers:
.. code:: bash
$ pyastgrep './/Constant[@type="int" or @type="float"]'
tests/examples/test_xml/everything.py:5:20: assigned_int = 123
tests/examples/test_xml/everything.py:6:22: assigned_float = 3.14
Function calls where:
* the function is named ``open``:
* the second positional argument is a string literal containing the character ``b``:
.. code:: bash
pyastgrep './/Call[./func/Name[@id="open"]][./args/Constant[position()=1][contains(@value, "b")]]'
Usages of ``open`` that are **not** in a ``with`` item expression:
.. code:: bash
pyastgrep './/Call[not(ancestor::withitem)]/func/Name[@id="open"]'
Names longer than 42 characters:
.. code:: bash
$ pyastgrep './/Name[string-length(@id) > 42]'
``except`` clauses that raise a different exception class than they catch:
.. code:: bash
$ pyastgrep "//ExceptHandler[body//Raise/exc//Name and not(contains(body//Raise/exc//Name/@id, type/Name/@id))]"
Functions whose name contain a certain substring:
.. code:: bash
$ pyastgrep './/FunctionDef[contains(@name, "something")]'
Classes whose name matches a regular expression:
.. code:: bash
$ pyastgrep ".//ClassDef[re:match('M.*', @name)]"
The above uses the Python `re.match
<https://docs.python.org/3/library/re.html#re.match>`_ method. You can also use
``re:search`` to use the Python `re.search
<https://docs.python.org/3/library/re.html#re.search>`_ method.
Case-insensitive match of names on the left hand side of an assignment
containing a certain string. This can be achieved using the ``lower-case``
function from XPath2:
.. code:: bash
$ pyastgrep './/Assign/targets//Name[contains(lower-case(@id), "something")]' --xpath2
You can also use regexes, passing the ``i`` (case-insensitive flag) as below, as
described in the Python `Regular Expression Syntax docs
<https://docs.python.org/3/library/re.html#regular-expression-syntax>`_
.. code:: bash
$ pyastgrep './/Assign/targets//Name[re:search("(?i)something", @id)]'
Assignments to the name ``foo``, including type annotated assignments, which
use ``AnnAssign``, and tuple unpacking assignments (while avoiding things like
``foo.bar = ...``). Note the use of the ``|`` operator to do a union.
.. code:: bash
$ pyastgrep '(.//AnnAssign/target|.//Assign/targets|.//Assign/targets/Tuple/elts)/Name[@id="foo"]'
Docstrings of functions/methods whose value contains “hello”:
.. code:: bash
$ pyastgrep './/FunctionDef/body/Expr[1]/value/Constant[@type="str"][contains(@value, "hello")]'
For-loop variables called ``i`` or ``j`` (including those created by tuple unpacking):
.. code:: bash
$ pyastgrep './/For/target//Name[@id="i" or @id="j"]'
Method calls: These are actually “calls” on objects that are attributes of other
objects. This will match the top-level object:
.. code:: bash
$ pyastgrep './/Call/func/Attribute'
Individual positional arguments to a method call named ``encode``, where the
arguments are literal strings or numbers. Note the use of ``Call[…]`` to match
“Call nodes that have descendants that match …”, rather than matching those
descendant nodes themselves.
.. code:: bash
$ pyastgrep './/Call[./func/Attribute[@attr="encode"]]/args/Constant'
For a Django code base, find all ``.filter`` and ``.exclude`` method calls, and
all ``Q`` object calls, which have a keyword argument where the name contains
the string ``"user"``, for finding ORM calls like
``.filter(user__id__in=...)`` or ``Q(thing__user=...)``:
.. code:: bash
pyastgrep '(.//Call[./func/Attribute[@attr="filter" or @attr="exclude"]] | .//Call[./func/Name[@id="Q"]]) [./keywords/keyword[contains(@arg, "user")]]'
Ignoring files
--------------
Files/directories matching ``.gitignore`` entries (global and local) are
automatically ignored, unless specified as paths on the command line.
Currently there are no other methods to add or remove this ignoring logic.
Please open a ticket if you want this feature. Most likely we should try to make
it work like `ripgrep filtering
<https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#manual-filtering-globs>`_
if that makes sense.
CSS selectors
-------------
In general, XPath expressions are more powerful than CSS selectors, and CSS
selectors have some things that are specific to HTML (such as specific selectors
for ``id`` and ``class``). However, it may be easier to get started using CSS
selectors, and for some things CSS selectors are easier. In that case, just pass
``--css`` and the expression will be interpreted as a CSS selector instead.
For example, to get the first statement in each ``for`` statement body:
.. code:: bash
$ pyastgrep --css 'For > body > *:first-child'
The CSS selector will converted to an XPath expression with a prefix of ``.//``
— that is, it will be interpreted as a query over all the document.
Note that unlike CSS selectors in HTML, the expression will be interpreted
case-sensitively.
You can also use the online tool `css2xpath <https://css2xpath.github.io/>`_ to
do translations before passing to ``pyastgrep``. This tool also supports some
things that our `cssselect (our dependency) does not yet support
<https://github.com/scrapy/cssselect/issues>`_.
Tips
----
Command line flags
~~~~~~~~~~~~~~~~~~
There are a growing number of command line flags – see ``pyastgrep --help``
Extracting code snippets
~~~~~~~~~~~~~~~~~~~~~~~~
If you want to extract standalone snippets of code, try ``--context=statement
--heading`` which does automatic dedenting. e.g. to extract all functions and
methods, with leading whitespace removed, do:
.. code-block:: bash
$ pyastgrep --heading -C statement './/FunctionDef'
Absolute paths
~~~~~~~~~~~~~~
To get pyastgrep to print absolute paths in results, pass the current absolute
path as the directory to search::
pyastgrep "..." $(pwd)
Debugging XPath expressions
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the ``--xml`` option to see the XML for matches. If you need to see more
context, you can use things like the ``parent`` or ``ancestor`` selector. For
example, you might do the following but get back more results than you want:
.. code:: bash
$ pyastgrep './/Assign/targets//Name[@id="foo"]
example.py:1:1:foo = 1
example.py:2:2:(foo, bar) = (3, 4)
example.py:3:1:foo.bar = 2
Here you might be interested in the first two results, which both assign to
the name ``foo``, but not the last one since it does not. You can get the XML for the
whole matching assignment expressions like this:
.. code:: bash
$ pyastgrep './/Assign/targets//Name[@id="foo"]/ancestor::Assign' --xml
example.py:1:1:foo = 1
<Assign lineno="1" col_offset="0">
<targets>
<Name lineno="1" col_offset="0" type="str" id="foo">
<ctx>
<Store/>
</ctx>
</Name>
</targets>
<value>
<Constant lineno="1" col_offset="6" type="int" value="1"/>
</value>
</Assign>
...
You could also go the other way and change the XPath expression to match on the
parent ``Assign`` node — this matches “all ``Assign`` nodes that are parents of
a ``target`` node that is a parent of a ``Name`` node with attribute ``id``
equal to ``"foo"``:
.. code:: bash
$ pyastgrep './/Assign[./targets//Name[@id="foo"]]' --xml
Bugs
----
Due to limitations in what characters can be stored in an XML document, null
bytes (``\x00``) and other characters such as escape codes in string and byte
literals get stripped, and can’t be searched for.
Limitations and other tools
---------------------------
pyastgrep is useful for grepping Python code at a fairly low level. It can be
used for various refactoring or linting tasks. Some linting tasks require higher
level understanding of a code base. For example, to detect use of a certain
function, you need to cope with various ways that the function may be imported
and used, and avoid detecting a function with the same name but from a different
module. For these kinds of tasks, you might be interested in:
* `Semgrep <https://semgrep.dev/>`_
* `Fixit <https://github.com/Instagram/Fixit>`_
If you are looking for something simpler, try:
* Simon Willison’s `symbex <https://github.com/simonw/symbex/>`_ which can
extract functions/methods/classes.
If you are using this as a library, you should note that while AST works well
for linting, it’s not as good for rewriting code, because AST does not contain
or preserve things like formatting and comments. For a better approach, have a
look at `libCST <https://github.com/Instagram/LibCST>`_.
Use as a library
----------------
pyastgrep is structured internally to make it easy to use as a library as well
as a CLI, with a clear separation of the different layers. However, while we
will try not to break things without good reason, at this point we are not
documenting or guaranteeing API stability for these functions. Please contribute
to `the discussion <https://github.com/spookylukey/pyastgrep/discussions/18>`_
if you have needs here.
Editor integration
------------------
Emacs
~~~~~
pyastgrep works very well with ``compilation-mode`` and wrappers like
``projectile-compile-project`` from `Projectile
<https://docs.projectile.mx/projectile/usage.html#basic-usage>`_. We recommend
setting up a keyboard shortcut for ``next-error`` to enable you to step through
results easily.
Visual Studio Code
~~~~~~~~~~~~~~~~~~
Run pyastgrep from a terminal and results will be hyperlinked automatically.
PyCharm
~~~~~~~
Run pyastgrep from a terminal and results will be hyperlinked automatically.
Others
~~~~~~
Contributions to this section gladly accepted!
Contributing
------------
Get test suite running::
pip install -r requirements-test.txt
pytest
Run tests against all versions::
pip install tox
tox
Please install `pre-commit <https://pre-commit.com/>`_ in the repo::
pre-commit install
This will add Git hooks to run linters when committing, which ensures our style
(black) and other things.
You can manually run these linters using::
pre-commit run --all --all-files
Run mypy (we only expect it to pass on Python 3.10)::
mypy .
Bug fixes and other changes can be submitted using pull requests on GitHub. For
large changes, it’s worth opening an issue first to discuss the approach.
Links
-----
- `Green tree snakes <https://greentreesnakes.readthedocs.io/en/latest/>`__ - a very readable overview of Python ASTs.
- `ast module documentation <https://docs.python.org/3/library/ast.html>`__.
- `Python AST Explorer <https://python-ast-explorer.com/>`__ for worked examples of ASTs.
- A `brief guide to XPath <http://www.w3schools.com/xml/xpath_syntax.asp>`__.
See also the `XPath Axes <https://www.w3schools.com/xml/xpath_axes.asp>`_ guide
which can be very helpful for matching related AST nodes.
- `Online XPath Tester <https://extendsclass.com/xpath-tester.html>`_
History
-------
This project was forked from https://github.com/hchasestevens/astpath by `H.
Chase Stevens <http://www.chasestevens.com>`__. Main changes:
* Added a test suite
* Many bugs fixed
* Significant rewrite of parts of code
* Changes to match grep/ripgrep, including formatting and automatic filtering.
Raw data
{
"_id": null,
"home_page": "https://github.com/spookylukey/pyastgrep",
"name": "pyastgrep",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "xpath xml ast asts syntax query css grep",
"author": "Luke Plant",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/45/80/b60669bec74b5e1371713637c92bf07d2401e5b6ed0389dfeb1af6569518/pyastgrep-1.3.1.tar.gz",
"platform": null,
"description": "pyastgrep\n=========\n\n\n.. image:: https://badge.fury.io/py/pyastgrep.svg\n :target: https://badge.fury.io/py/pyastgrep\n\n.. image:: https://github.com/spookylukey/pyastgrep/actions/workflows/tests.yml/badge.svg\n :target: https://github.com/spookylukey/pyastgrep/actions/workflows/tests.yml\n\nA command-line utility for grepping Python files using XPath syntax (or CSS\nselectors) against the Python AST (Abstract Syntax Tree).\n\nIn other words, this allows you to search Python code against specific syntax\nelements (function definitions, arguments, assignments, variables etc), instead\nof grepping for string matches.\n\nThe interface and behaviour is designed to match grep and ripgrep as far as it\nmakes sense to do so.\n\n.. contents:: Contents\n\n\nInstallation\n------------\n\nPython 3.8+ required.\n\nWe recommend `pipx <https://pipxproject.github.io/pipx/>`_ to install it\nconveniently in an isolated environment:\n\n::\n\n pipx install pyastgrep\n\n\nYou can also use pip:\n\n::\n\n pip install pyastgrep\n\n\nBasic usage\n-----------\n\n::\n\n pyastgrep <xpath expression> [files or directories]\n\n\nSee ``pyastgrep --help`` for more options.\n\n\npyastgrep in action\n-------------------\n\nHere is an example showing everywhere in the ``pyastgrep`` code base that uses a\nvariable whose name contains ``idx``:\n\n.. image:: https://raw.githubusercontent.com/spookylukey/pyastgrep/master/pyastgrep_screenshot.png\n :align: center\n :width: 800\n :alt: Screenshot showing a terminal running ``pyastgrep './/Name[contains(@id, 'idx')]\u2019``\n\n\nUnderstanding the XML structure\n-------------------------------\n\nTo get started, you\u2019ll need some understanding of how Python AST is structured,\nand how that is mapped to XML. Some methods for doing that are below:\n\n1. Use `Python AST Explorer <https://python-ast-explorer.com/>`_ to play around\n with what AST looks like.\n\n2. Dump out the AST and/or XML structure of the top-level statements in a Python\n file. The easiest way to do this is to use the provided ``pyastdump``\n command, passing in either a Python filename, ``pyastdump yourfile.py``, or\n piping in Python fragments as below:\n\n .. code:: bash\n\n $ echo 'x = 1' | pyastdump -\n <Module>\n <body>\n <Assign lineno=\"1\" col_offset=\"0\">\n <targets>\n <Name lineno=\"1\" col_offset=\"0\" type=\"str\" id=\"x\">\n <ctx>\n <Store/>\n </ctx>\n </Name>\n </targets>\n <value>\n <Constant lineno=\"1\" col_offset=\"4\" type=\"int\" value=\"1\"/>\n </value>\n </Assign>\n </body>\n <type_ignores/>\n </Module>\n\n (When piping input in this way, code will be automatically dedented, making\n this easier to do from partial Python snippets.)\n\n You can also use the ``pyastgrep`` command, but since the top-level XML\n elements are ``<Module><body>``, and don\u2019t correspond to actual source lines,\n you\u2019ll need to use an XPath expression ``./*/*`` to get a match for each\n statement within the body, and pass ``--xml`` and/or ``--ast`` to dump the\n XML/AST structure:\n\n .. code:: bash\n\n $ pyastgrep --xml --ast './*/*' myfile.py\n myfile.py:1:1:import os\n Import(\n lineno=1,\n col_offset=0,\n end_lineno=1,\n end_col_offset=9,\n names=[alias(lineno=1, col_offset=7, end_lineno=1, end_col_offset=9, name='os', asname=None)],\n )\n <Import lineno=\"1\" col_offset=\"0\">\n <names>\n <alias lineno=\"1\" col_offset=\"7\" type=\"str\" name=\"os\"/>\n </names>\n </Import>\n ...\n\n\nNote that the XML format is a very direct translation of the Python AST as\nproduced by the `ast module <https://docs.python.org/3/library/ast.html>`_ (with\nsome small additions made to improve usability for a few cases). This AST is not\nstable across Python versions, so the XML is not stable either. Normally changes\nin the AST correspond to new syntax that is added to Python, but in some cases a\nnew Python version will make significant changes made to the AST generated for\nthe same code.\n\nYou\u2019ll also need some understanding of how to write XPath expressions (see links\nat the bottom), but the examples below should get you started.\n\nExamples\n--------\n\nUsages of a function called ``open``:\n\n.. code:: bash\n\n $ pyastgrep './/Call/func/Name[@id=\"open\"]'\n src/pyastgrep/search.py:88:18: with open(path) as f:\n\nLiteral numbers:\n\n.. code:: bash\n\n $ pyastgrep './/Constant[@type=\"int\" or @type=\"float\"]'\n tests/examples/test_xml/everything.py:5:20: assigned_int = 123\n tests/examples/test_xml/everything.py:6:22: assigned_float = 3.14\n\nFunction calls where:\n\n* the function is named ``open``:\n* the second positional argument is a string literal containing the character ``b``:\n\n.. code:: bash\n\n pyastgrep './/Call[./func/Name[@id=\"open\"]][./args/Constant[position()=1][contains(@value, \"b\")]]'\n\nUsages of ``open`` that are **not** in a ``with`` item expression:\n\n.. code:: bash\n\n pyastgrep './/Call[not(ancestor::withitem)]/func/Name[@id=\"open\"]'\n\nNames longer than 42 characters:\n\n.. code:: bash\n\n $ pyastgrep './/Name[string-length(@id) > 42]'\n\n``except`` clauses that raise a different exception class than they catch:\n\n.. code:: bash\n\n $ pyastgrep \"//ExceptHandler[body//Raise/exc//Name and not(contains(body//Raise/exc//Name/@id, type/Name/@id))]\"\n\nFunctions whose name contain a certain substring:\n\n.. code:: bash\n\n $ pyastgrep './/FunctionDef[contains(@name, \"something\")]'\n\nClasses whose name matches a regular expression:\n\n.. code:: bash\n\n $ pyastgrep \".//ClassDef[re:match('M.*', @name)]\"\n\n\nThe above uses the Python `re.match\n<https://docs.python.org/3/library/re.html#re.match>`_ method. You can also use\n``re:search`` to use the Python `re.search\n<https://docs.python.org/3/library/re.html#re.search>`_ method.\n\nCase-insensitive match of names on the left hand side of an assignment\ncontaining a certain string. This can be achieved using the ``lower-case``\nfunction from XPath2:\n\n.. code:: bash\n\n $ pyastgrep './/Assign/targets//Name[contains(lower-case(@id), \"something\")]' --xpath2\n\n\nYou can also use regexes, passing the ``i`` (case-insensitive flag) as below, as\ndescribed in the Python `Regular Expression Syntax docs\n<https://docs.python.org/3/library/re.html#regular-expression-syntax>`_\n\n.. code:: bash\n\n $ pyastgrep './/Assign/targets//Name[re:search(\"(?i)something\", @id)]'\n\n\nAssignments to the name ``foo``, including type annotated assignments, which\nuse ``AnnAssign``, and tuple unpacking assignments (while avoiding things like\n``foo.bar = ...``). Note the use of the ``|`` operator to do a union.\n\n.. code:: bash\n\n $ pyastgrep '(.//AnnAssign/target|.//Assign/targets|.//Assign/targets/Tuple/elts)/Name[@id=\"foo\"]'\n\nDocstrings of functions/methods whose value contains \u201chello\u201d:\n\n.. code:: bash\n\n $ pyastgrep './/FunctionDef/body/Expr[1]/value/Constant[@type=\"str\"][contains(@value, \"hello\")]'\n\nFor-loop variables called ``i`` or ``j`` (including those created by tuple unpacking):\n\n.. code:: bash\n\n $ pyastgrep './/For/target//Name[@id=\"i\" or @id=\"j\"]'\n\n\nMethod calls: These are actually \u201ccalls\u201d on objects that are attributes of other\nobjects. This will match the top-level object:\n\n.. code:: bash\n\n $ pyastgrep './/Call/func/Attribute'\n\n\nIndividual positional arguments to a method call named ``encode``, where the\narguments are literal strings or numbers. Note the use of ``Call[\u2026]`` to match\n\u201cCall nodes that have descendants that match \u2026\u201d, rather than matching those\ndescendant nodes themselves.\n\n.. code:: bash\n\n $ pyastgrep './/Call[./func/Attribute[@attr=\"encode\"]]/args/Constant'\n\n\nFor a Django code base, find all ``.filter`` and ``.exclude`` method calls, and\nall ``Q`` object calls, which have a keyword argument where the name contains\nthe string ``\"user\"``, for finding ORM calls like\n``.filter(user__id__in=...)`` or ``Q(thing__user=...)``:\n\n.. code:: bash\n\n pyastgrep '(.//Call[./func/Attribute[@attr=\"filter\" or @attr=\"exclude\"]] | .//Call[./func/Name[@id=\"Q\"]]) [./keywords/keyword[contains(@arg, \"user\")]]'\n\n\nIgnoring files\n--------------\n\nFiles/directories matching ``.gitignore`` entries (global and local) are\nautomatically ignored, unless specified as paths on the command line.\n\nCurrently there are no other methods to add or remove this ignoring logic.\nPlease open a ticket if you want this feature. Most likely we should try to make\nit work like `ripgrep filtering\n<https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#manual-filtering-globs>`_\nif that makes sense.\n\nCSS selectors\n-------------\n\nIn general, XPath expressions are more powerful than CSS selectors, and CSS\nselectors have some things that are specific to HTML (such as specific selectors\nfor ``id`` and ``class``). However, it may be easier to get started using CSS\nselectors, and for some things CSS selectors are easier. In that case, just pass\n``--css`` and the expression will be interpreted as a CSS selector instead.\n\nFor example, to get the first statement in each ``for`` statement body:\n\n.. code:: bash\n\n $ pyastgrep --css 'For > body > *:first-child'\n\nThe CSS selector will converted to an XPath expression with a prefix of ``.//``\n\u2014 that is, it will be interpreted as a query over all the document.\n\nNote that unlike CSS selectors in HTML, the expression will be interpreted\ncase-sensitively.\n\nYou can also use the online tool `css2xpath <https://css2xpath.github.io/>`_ to\ndo translations before passing to ``pyastgrep``. This tool also supports some\nthings that our `cssselect (our dependency) does not yet support\n<https://github.com/scrapy/cssselect/issues>`_.\n\nTips\n----\n\nCommand line flags\n~~~~~~~~~~~~~~~~~~\n\nThere are a growing number of command line flags \u2013 see ``pyastgrep --help``\n\nExtracting code snippets\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you want to extract standalone snippets of code, try ``--context=statement\n--heading`` which does automatic dedenting. e.g. to extract all functions and\nmethods, with leading whitespace removed, do:\n\n.. code-block:: bash\n\n $ pyastgrep --heading -C statement './/FunctionDef'\n\nAbsolute paths\n~~~~~~~~~~~~~~\nTo get pyastgrep to print absolute paths in results, pass the current absolute\npath as the directory to search::\n\n pyastgrep \"...\" $(pwd)\n\n\nDebugging XPath expressions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUse the ``--xml`` option to see the XML for matches. If you need to see more\ncontext, you can use things like the ``parent`` or ``ancestor`` selector. For\nexample, you might do the following but get back more results than you want:\n\n.. code:: bash\n\n $ pyastgrep './/Assign/targets//Name[@id=\"foo\"]\n example.py:1:1:foo = 1\n example.py:2:2:(foo, bar) = (3, 4)\n example.py:3:1:foo.bar = 2\n\nHere you might be interested in the first two results, which both assign to\nthe name ``foo``, but not the last one since it does not. You can get the XML for the\nwhole matching assignment expressions like this:\n\n.. code:: bash\n\n $ pyastgrep './/Assign/targets//Name[@id=\"foo\"]/ancestor::Assign' --xml\n example.py:1:1:foo = 1\n <Assign lineno=\"1\" col_offset=\"0\">\n <targets>\n <Name lineno=\"1\" col_offset=\"0\" type=\"str\" id=\"foo\">\n <ctx>\n <Store/>\n </ctx>\n </Name>\n </targets>\n <value>\n <Constant lineno=\"1\" col_offset=\"6\" type=\"int\" value=\"1\"/>\n </value>\n </Assign>\n ...\n\n\nYou could also go the other way and change the XPath expression to match on the\nparent ``Assign`` node \u2014 this matches \u201call ``Assign`` nodes that are parents of\na ``target`` node that is a parent of a ``Name`` node with attribute ``id``\nequal to ``\"foo\"``:\n\n.. code:: bash\n\n $ pyastgrep './/Assign[./targets//Name[@id=\"foo\"]]' --xml\n\nBugs\n----\n\nDue to limitations in what characters can be stored in an XML document, null\nbytes (``\\x00``) and other characters such as escape codes in string and byte\nliterals get stripped, and can\u2019t be searched for.\n\nLimitations and other tools\n---------------------------\n\npyastgrep is useful for grepping Python code at a fairly low level. It can be\nused for various refactoring or linting tasks. Some linting tasks require higher\nlevel understanding of a code base. For example, to detect use of a certain\nfunction, you need to cope with various ways that the function may be imported\nand used, and avoid detecting a function with the same name but from a different\nmodule. For these kinds of tasks, you might be interested in:\n\n* `Semgrep <https://semgrep.dev/>`_\n* `Fixit <https://github.com/Instagram/Fixit>`_\n\n\nIf you are looking for something simpler, try:\n\n* Simon Willison\u2019s `symbex <https://github.com/simonw/symbex/>`_ which can\n extract functions/methods/classes.\n\nIf you are using this as a library, you should note that while AST works well\nfor linting, it\u2019s not as good for rewriting code, because AST does not contain\nor preserve things like formatting and comments. For a better approach, have a\nlook at `libCST <https://github.com/Instagram/LibCST>`_.\n\n\nUse as a library\n----------------\n\npyastgrep is structured internally to make it easy to use as a library as well\nas a CLI, with a clear separation of the different layers. However, while we\nwill try not to break things without good reason, at this point we are not\ndocumenting or guaranteeing API stability for these functions. Please contribute\nto `the discussion <https://github.com/spookylukey/pyastgrep/discussions/18>`_\nif you have needs here.\n\nEditor integration\n------------------\n\nEmacs\n~~~~~\n\npyastgrep works very well with ``compilation-mode`` and wrappers like\n``projectile-compile-project`` from `Projectile\n<https://docs.projectile.mx/projectile/usage.html#basic-usage>`_. We recommend\nsetting up a keyboard shortcut for ``next-error`` to enable you to step through\nresults easily.\n\nVisual Studio Code\n~~~~~~~~~~~~~~~~~~\n\nRun pyastgrep from a terminal and results will be hyperlinked automatically.\n\nPyCharm\n~~~~~~~\n\nRun pyastgrep from a terminal and results will be hyperlinked automatically.\n\nOthers\n~~~~~~\n\nContributions to this section gladly accepted!\n\n\n\nContributing\n------------\n\nGet test suite running::\n\n pip install -r requirements-test.txt\n pytest\n\nRun tests against all versions::\n\n pip install tox\n tox\n\nPlease install `pre-commit <https://pre-commit.com/>`_ in the repo::\n\n pre-commit install\n\nThis will add Git hooks to run linters when committing, which ensures our style\n(black) and other things.\n\nYou can manually run these linters using::\n\n pre-commit run --all --all-files\n\nRun mypy (we only expect it to pass on Python 3.10)::\n\n mypy .\n\nBug fixes and other changes can be submitted using pull requests on GitHub. For\nlarge changes, it\u2019s worth opening an issue first to discuss the approach.\n\nLinks\n-----\n\n- `Green tree snakes <https://greentreesnakes.readthedocs.io/en/latest/>`__ - a very readable overview of Python ASTs.\n- `ast module documentation <https://docs.python.org/3/library/ast.html>`__.\n- `Python AST Explorer <https://python-ast-explorer.com/>`__ for worked examples of ASTs.\n- A `brief guide to XPath <http://www.w3schools.com/xml/xpath_syntax.asp>`__.\n See also the `XPath Axes <https://www.w3schools.com/xml/xpath_axes.asp>`_ guide\n which can be very helpful for matching related AST nodes.\n- `Online XPath Tester <https://extendsclass.com/xpath-tester.html>`_\n\nHistory\n-------\n\nThis project was forked from https://github.com/hchasestevens/astpath by `H.\nChase Stevens <http://www.chasestevens.com>`__. Main changes:\n\n* Added a test suite\n* Many bugs fixed\n* Significant rewrite of parts of code\n* Changes to match grep/ripgrep, including formatting and automatic filtering.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A query tool for Python abstract syntax trees",
"version": "1.3.1",
"project_urls": {
"Homepage": "https://github.com/spookylukey/pyastgrep"
},
"split_keywords": [
"xpath",
"xml",
"ast",
"asts",
"syntax",
"query",
"css",
"grep"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d3cd814b35466926df2dd2802fbc0a343f44d8c496047ad6afb390e32389bb39",
"md5": "5b99489c43c27b0da4f464ad2ad35fb5",
"sha256": "261c0926944f56d446657a3622f0f8ebd8c2c92a341cc21f57e59d5fe3d7bb2c"
},
"downloads": -1,
"filename": "pyastgrep-1.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5b99489c43c27b0da4f464ad2ad35fb5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 26228,
"upload_time": "2023-07-19T09:12:39",
"upload_time_iso_8601": "2023-07-19T09:12:39.570412Z",
"url": "https://files.pythonhosted.org/packages/d3/cd/814b35466926df2dd2802fbc0a343f44d8c496047ad6afb390e32389bb39/pyastgrep-1.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4580b60669bec74b5e1371713637c92bf07d2401e5b6ed0389dfeb1af6569518",
"md5": "c52d1cfd4d0bbb598da57103062668fc",
"sha256": "22994e1c288b5624cd8c3966fd411f36f1ce918f2b048173e9a18c27b9371fb3"
},
"downloads": -1,
"filename": "pyastgrep-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "c52d1cfd4d0bbb598da57103062668fc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 37232,
"upload_time": "2023-07-19T09:12:41",
"upload_time_iso_8601": "2023-07-19T09:12:41.360074Z",
"url": "https://files.pythonhosted.org/packages/45/80/b60669bec74b5e1371713637c92bf07d2401e5b6ed0389dfeb1af6569518/pyastgrep-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-19 09:12:41",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "spookylukey",
"github_project": "pyastgrep",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pyastgrep"
}
JSON
