<div align=center>
Specialist
==========
[](https://github.com/brandtbucher/specialist/releases)[](https://github.com/brandtbucher/specialist/actions)[](https://github.com/brandtbucher/specialist/issues)
<br>
</div>
<div align=justify>
Specialist uses [fine-grained location](https://peps.python.org/pep-0657/)
information to create visual representations of exactly *where* and *how*
CPython's new
[specializing, adaptive interpreter](https://peps.python.org/pep-0659/)
optimizes your code.
<div align=center>
</div>
Getting Started
---------------
Specialist supports CPython 3.11+ on all platforms.
To install, just run:
```sh
$ pip install specialist
```
If you normally use `pytest` to run your tests, then you can try using the
following command to run them instead:
```sh
$ specialist --output report --targets '**/*.py' -m pytest # any additional pytest options here...
```
After your tests complete, `specialist` will create a directory named `report`
and fill it with browsable HTML visualizations for each module in the current
directory tree.
Background
----------
While CPython is running your code, it identifies "hot" regions that are being
run often enough to spend time optimizing. It occasionally "quickens" these
regions, which `specialist` represents using color. **Dark, rich colors indicate
code with many quickened instructions (and, therefore, high specialization
potential), while light, pale colors indicate code with relatively few
specialization opportunities.**
Most of the time, quickening involves three phases:
- Replacing individual bytecode instructions with "adaptive" forms. These are
actually a bit slower than normal instructions, because they attempt to
"specialize" themselves periodically. If they are unable to specialize, they
will remain in their adaptive form. **`specialist` uses red to indicate the
presence of adaptive instructions.**
- Occasionaly, adaptive instructions will convert themselves into much faster
"specialized" instructions. Examples of specializations include attribute
accesses on a single object or type, certain pure-Python calls, or integer
addition. **`specialist` uses green to indicate the presence of specialized
instructions.**
- If a specialization becomes invalid after some time (for example, if an
expression that previously added two integers starts concatenating two strings
instead), the specialized instruction may be converted back into an adaptive
one. At this point, the cycle repeats itself.
Specialist aims to provide insight into this process for the maintainers of
CPython itself, as well as for users seeking an optimization profile for their
own code. If you're curious to learn more about specialization, check out [this
talk from PyCon US 2023](https://youtu.be/shQtrn1v7sQ).
Tutorial
--------
Suppose we have the following source file, `conversions.py`, which contains some
utilities and tests for converting between Fahrenheit and Celsius:
```py
import math
def f_to_c(f: float) -> float:
"""Convert Fahrenheit to Celsius."""
x = f - 32
return x * 5 / 9
def c_to_f(c: float) -> float:
"""Convert Celsius to Fahrenheit."""
x = c * 9 / 5
return x + 32
TEST_VALUES = [-459.67, -273.15, 0.0, 32.0, 42.0, 273.15, 100.0, 212.0, 373.15]
def test_conversions() -> None:
for t in TEST_VALUES:
assert_round_trip(t)
def assert_round_trip(t: float) -> None:
# Round-trip Fahrenheit through Celsius:
assert math.isclose(t, f_to_c(c_to_f(t))), f"{t} F -> C -> F failed!"
# Round-trip Celsius through Fahrenheit:
assert math.isclose(t, c_to_f(f_to_c(t))), f"{t} C -> F -> C failed!"
if __name__ == "__main__":
test_conversions()
```
We can run this file with CPython from the command-line using `specialist`:
```sh
$ specialist conversions.py
```
After the script has finished running, `specialist` will open a web browser and
display the annotated program source:
<div align=center>
</div>
The green areas indicate regions of code that were successfully specialized,
while the red areas indicate unsuccessful specializations (in the form of
"adaptive" instructions). Mixed results are indicated by colors along the
green-yellow-orange-red gradient, depending on the ratio of successes to
failures. Regions of code that don't contain any attempted specializations are
left white.
Focusing on `f_to_c` and `c_to_f` for a moment, we can see that CPython is
failing to specialize addition and subtraction by `32`. It doesn't currently
specialize binary operators between mixed `float` and `int` values, which is
exactly what the code here is doing.
It can, however, specialize addition and subtraction between two `float` values!
Replacing `32` with `32.0` results in successful specializations (confirmed by
re-running `specialist`):
<div align=center>
</div>
We can see that something similar is happening with `float` and `int`
multiplication as well. One option could be to continue converting constant
values to `float`:
<div align=center>
</div>
However, there's a better option! Notice that CPython doesn't attempt to
specialize division at all (it's left white in the visualization). We can take
advantage of CPython's constant folding optimizations by slightly changing the
order of operations, which allows our scaling factors (`5 / 9` and `9 / 5`) to
be computed at compile-time. When we do this, CPython is able to implement our
converters *entirely* using native floating-point operations:
<div align=center>
</div>
Modes
-----
Like `python` itself, `specialist` can run code a few different ways. It can be
given a file path:
```sh
$ specialist spam/eggs.py foo bar baz
```
Or a module name:
```sh
$ specialist -m spam.eggs foo bar baz
```
Or a command:
```sh
$ specialist -c 'import spam; spam.eggs()' foo bar baz
```
It also has a `-t`/`--targets` option to support discovery of arbitrary "target"
files to analyze after the script completes. This is useful if the script being
run is different from the code you want to visualize:
```sh
$ specialist --targets spam/eggs.py -c 'import uses_eggs; uses_eggs.run()'
```
Multiple files can be provided using "glob" style patterns:
```sh
$ specialist --targets 'spam/**/*.py' -m pytest
```
Specialist can also write the generated HTML files to the filesystem instead of
opening them in a browser. To do so, just provide an output directory path using
the `-o`/`--output` option:
```sh
$ specialist --output ../report --targets 'spam/**/*.py' -m pytest
/home/brandtbucher/sketch/spam/__init__.py -> /home/brandtbucher/report/__init__.html
/home/brandtbucher/sketch/spam/_spammy.py -> /home/brandtbucher/report/_spammy.html
/home/brandtbucher/sketch/spam/eggs/__init__.py -> /home/brandtbucher/report/eggs/__init__.html
/home/brandtbucher/sketch/spam/eggs/_eggy.py -> /home/brandtbucher/report/eggs/_eggy.html
```
Options
-------
### `-b`/`--blue`
Use blue (rather than green) to indicate specialized code. Some users may find
a blue-violet-magenta-red gradient easier to read than the default
green-yellow-orange-red one.
### `-d`/`--dark`
Use light text on a dark background. Some users may find a dark scheme makes
them feel cooler than the default light one.
</div>
Raw data
{
"_id": null,
"home_page": "https://github.com/brandtbucher/specialist",
"name": "specialist",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": null,
"author": "Brandt Bucher",
"author_email": "brandt@python.org",
"download_url": "https://files.pythonhosted.org/packages/bb/dc/b3f60b02f4356432790d52bb7db9e64d2e5ff799d246b757f3400cfcbd6e/specialist-0.7.0.tar.gz",
"platform": null,
"description": "<div align=center>\n\n\nSpecialist\n==========\n\n[](https://github.com/brandtbucher/specialist/releases)[](https://github.com/brandtbucher/specialist/actions)[](https://github.com/brandtbucher/specialist/issues)\n\n<br>\n\n</div>\n\n<div align=justify>\n\nSpecialist uses [fine-grained location](https://peps.python.org/pep-0657/)\ninformation to create visual representations of exactly *where* and *how*\nCPython's new\n[specializing, adaptive interpreter](https://peps.python.org/pep-0659/)\noptimizes your code.\n\n<div align=center>\n\n\n\n</div>\n\n\nGetting Started\n---------------\n\nSpecialist supports CPython 3.11+ on all platforms.\n\nTo install, just run:\n\n```sh\n$ pip install specialist\n```\n\nIf you normally use `pytest` to run your tests, then you can try using the\nfollowing command to run them instead:\n\n```sh\n$ specialist --output report --targets '**/*.py' -m pytest # any additional pytest options here...\n```\n\nAfter your tests complete, `specialist` will create a directory named `report`\nand fill it with browsable HTML visualizations for each module in the current\ndirectory tree.\n\n\nBackground\n----------\n\nWhile CPython is running your code, it identifies \"hot\" regions that are being\nrun often enough to spend time optimizing. It occasionally \"quickens\" these\nregions, which `specialist` represents using color. **Dark, rich colors indicate\ncode with many quickened instructions (and, therefore, high specialization\npotential), while light, pale colors indicate code with relatively few\nspecialization opportunities.**\n\nMost of the time, quickening involves three phases:\n\n- Replacing individual bytecode instructions with \"adaptive\" forms. These are\nactually a bit slower than normal instructions, because they attempt to\n\"specialize\" themselves periodically. If they are unable to specialize, they\nwill remain in their adaptive form. **`specialist` uses red to indicate the\npresence of adaptive instructions.**\n\n- Occasionaly, adaptive instructions will convert themselves into much faster\n\"specialized\" instructions. Examples of specializations include attribute\naccesses on a single object or type, certain pure-Python calls, or integer\naddition. **`specialist` uses green to indicate the presence of specialized\ninstructions.**\n\n- If a specialization becomes invalid after some time (for example, if an\nexpression that previously added two integers starts concatenating two strings\ninstead), the specialized instruction may be converted back into an adaptive\none. At this point, the cycle repeats itself.\n\nSpecialist aims to provide insight into this process for the maintainers of\nCPython itself, as well as for users seeking an optimization profile for their\nown code. If you're curious to learn more about specialization, check out [this\ntalk from PyCon US 2023](https://youtu.be/shQtrn1v7sQ).\n\n\nTutorial\n--------\n\nSuppose we have the following source file, `conversions.py`, which contains some\nutilities and tests for converting between Fahrenheit and Celsius:\n\n```py\nimport math\n\ndef f_to_c(f: float) -> float:\n \"\"\"Convert Fahrenheit to Celsius.\"\"\"\n x = f - 32\n return x * 5 / 9\n\ndef c_to_f(c: float) -> float:\n \"\"\"Convert Celsius to Fahrenheit.\"\"\"\n x = c * 9 / 5\n return x + 32\n\nTEST_VALUES = [-459.67, -273.15, 0.0, 32.0, 42.0, 273.15, 100.0, 212.0, 373.15]\n\ndef test_conversions() -> None:\n for t in TEST_VALUES:\n assert_round_trip(t)\n\ndef assert_round_trip(t: float) -> None:\n # Round-trip Fahrenheit through Celsius:\n assert math.isclose(t, f_to_c(c_to_f(t))), f\"{t} F -> C -> F failed!\"\n # Round-trip Celsius through Fahrenheit:\n assert math.isclose(t, c_to_f(f_to_c(t))), f\"{t} C -> F -> C failed!\"\n\nif __name__ == \"__main__\":\n test_conversions()\n```\n\nWe can run this file with CPython from the command-line using `specialist`:\n\n```sh\n$ specialist conversions.py\n```\n\nAfter the script has finished running, `specialist` will open a web browser and\ndisplay the annotated program source:\n\n<div align=center>\n\n\n\n</div>\n\nThe green areas indicate regions of code that were successfully specialized,\nwhile the red areas indicate unsuccessful specializations (in the form of\n\"adaptive\" instructions). Mixed results are indicated by colors along the\ngreen-yellow-orange-red gradient, depending on the ratio of successes to\nfailures. Regions of code that don't contain any attempted specializations are\nleft white.\n\nFocusing on `f_to_c` and `c_to_f` for a moment, we can see that CPython is\nfailing to specialize addition and subtraction by `32`. It doesn't currently\nspecialize binary operators between mixed `float` and `int` values, which is\nexactly what the code here is doing.\n\nIt can, however, specialize addition and subtraction between two `float` values!\nReplacing `32` with `32.0` results in successful specializations (confirmed by\nre-running `specialist`):\n\n<div align=center>\n\n\n\n</div>\n\nWe can see that something similar is happening with `float` and `int`\nmultiplication as well. One option could be to continue converting constant\nvalues to `float`:\n\n<div align=center>\n\n\n\n</div>\n\nHowever, there's a better option! Notice that CPython doesn't attempt to\nspecialize division at all (it's left white in the visualization). We can take\nadvantage of CPython's constant folding optimizations by slightly changing the\norder of operations, which allows our scaling factors (`5 / 9` and `9 / 5`) to\nbe computed at compile-time. When we do this, CPython is able to implement our\nconverters *entirely* using native floating-point operations:\n\n<div align=center>\n\n\n\n</div>\n\n\nModes\n-----\n\nLike `python` itself, `specialist` can run code a few different ways. It can be\ngiven a file path:\n\n```sh\n$ specialist spam/eggs.py foo bar baz\n```\n\nOr a module name:\n\n```sh\n$ specialist -m spam.eggs foo bar baz\n```\n\nOr a command:\n\n```sh\n$ specialist -c 'import spam; spam.eggs()' foo bar baz\n```\n\nIt also has a `-t`/`--targets` option to support discovery of arbitrary \"target\"\nfiles to analyze after the script completes. This is useful if the script being\nrun is different from the code you want to visualize:\n\n```sh\n$ specialist --targets spam/eggs.py -c 'import uses_eggs; uses_eggs.run()'\n```\n\nMultiple files can be provided using \"glob\" style patterns:\n\n```sh\n$ specialist --targets 'spam/**/*.py' -m pytest\n```\n\nSpecialist can also write the generated HTML files to the filesystem instead of\nopening them in a browser. To do so, just provide an output directory path using\nthe `-o`/`--output` option:\n\n```sh\n$ specialist --output ../report --targets 'spam/**/*.py' -m pytest\n/home/brandtbucher/sketch/spam/__init__.py -> /home/brandtbucher/report/__init__.html\n/home/brandtbucher/sketch/spam/_spammy.py -> /home/brandtbucher/report/_spammy.html\n/home/brandtbucher/sketch/spam/eggs/__init__.py -> /home/brandtbucher/report/eggs/__init__.html\n/home/brandtbucher/sketch/spam/eggs/_eggy.py -> /home/brandtbucher/report/eggs/_eggy.html\n```\n\nOptions\n-------\n\n### `-b`/`--blue`\n\nUse blue (rather than green) to indicate specialized code. Some users may find\na blue-violet-magenta-red gradient easier to read than the default\ngreen-yellow-orange-red one.\n\n### `-d`/`--dark`\n\nUse light text on a dark background. Some users may find a dark scheme makes\nthem feel cooler than the default light one.\n\n</div>\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Visualize CPython's specializing, adaptive interpreter.",
"version": "0.7.0",
"project_urls": {
"Homepage": "https://github.com/brandtbucher/specialist"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b513e55fcee1af939a0c0274e5e21a1f7c48b4521c945886f7a63017d968b020",
"md5": "a36d058ff43a12675a280d141749bfd1",
"sha256": "e9e7c8b071af90fee9b3c27f5a063f15c3a1950a9696da3e703878f8d5efa6db"
},
"downloads": -1,
"filename": "specialist-0.7.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a36d058ff43a12675a280d141749bfd1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 13016,
"upload_time": "2024-05-19T04:58:28",
"upload_time_iso_8601": "2024-05-19T04:58:28.296312Z",
"url": "https://files.pythonhosted.org/packages/b5/13/e55fcee1af939a0c0274e5e21a1f7c48b4521c945886f7a63017d968b020/specialist-0.7.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bbdcb3f60b02f4356432790d52bb7db9e64d2e5ff799d246b757f3400cfcbd6e",
"md5": "66e046f28d1add03328db2af2bac5d06",
"sha256": "bad93e2cdbc1f123c2161b14163ef66cef4113ea708f437751288a4ab221115d"
},
"downloads": -1,
"filename": "specialist-0.7.0.tar.gz",
"has_sig": false,
"md5_digest": "66e046f28d1add03328db2af2bac5d06",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 12514,
"upload_time": "2024-05-19T04:58:30",
"upload_time_iso_8601": "2024-05-19T04:58:30.568896Z",
"url": "https://files.pythonhosted.org/packages/bb/dc/b3f60b02f4356432790d52bb7db9e64d2e5ff799d246b757f3400cfcbd6e/specialist-0.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-19 04:58:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "brandtbucher",
"github_project": "specialist",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "specialist"
}
JSON
