| Name | ipython-utils JSON |
| Version |
1.0.0
JSON |
| download |
| home_page | None |
| Summary | Utilities to make developing Python responsive and interactive using fully powered shells directly embedded at the target location or exception site, or dynamically reloading updated code |
| upload_time | 2024-12-22 02:55:16 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ipython_utils
Utilities to make developing Python responsive and interactive using fully powered shells directly embedded at the target location or exception site, or dynamically reloading updated code
## Introduction
Sometimes you develop a complex Python software with multiple modules and functions. Jupyter notebooks are messy and hinder refactoring into functions, encouraging bad programming practices, so you decide against using them. But there is nothing as convenient as having a shell to directly inspect the objects and having additions to your existing code applied on live objects immediately! **Now, you can!**
## Installation
Ever since I've packaged this and published it on PyPI (https://pypi.org/project/ipython-utils/), you can just run `pip install ipython_utils`. After installation, the following should work:
```python3
from ipython_utils import embed
def outer_function():
x = 0
def target_function():
print(x)
y1 = 1
y2 = 2
y3 = 3
y4 = 4
def g():
print(y1)
embed() # no local variable persistence
embed(target_function) # x can be updated, as target_function was closed over x
print(x, y1, y2, y3, y4)
embed([target_function, g]) # y1 can also be updated, as g was closed over y1
print(x, y1, y2, y3, y4)
# we can easily force a closure over y2 and y3, and pass it to embed,
# allowing it to update y2 and y3
embed([target_function, lambda: (y2, y3)])
return target_function
```
## Limited functionality in default IPython embedded shell
An existing method is to call `IPython.embed()` at the end of your partially developed code. But the embedded IPython shell is quite limited, because executing dynamically compiled Python code must always have a specific `locals()` dictionary, and updates to `locals()` rarely causes an update of local variables (except perhaps in old Python versions). The default shell just has a copy of the local variables, and existing closures over them will not update the value of the copy. More importantly, all child functions and lambdas do not close over the local variables, instead leaving them as unresolved global names, which means even list comprehensions which use these [do not work](https://github.com/cknoll/ipydex/issues/3). This is shown in the small snippet below:
```python3
def test_ipython():
x = 0
y = 1
def f(z, w):
nonlocal y
def g():
nonlocal w
w += 10
y += 20
print(x, y, z, w)
IPython.embed()
f(2, 3)
test_ipython()
```
```
0 21 2 3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: x, y, z, w
Out[1]: (0, 21, 2, 3)
In [2]: g()
In [3]: x, y, z, w
Out[3]: (0, 21, 2, 3)
In [4]: (lambda: x)()
...
NameError: name 'x' is not defined
In [5]: [x for i in range(10)]
...
NameError: name 'x' is not defined
In [6]: t = 1; [t for i in range(10)]
...
NameError: name 't' is not defined
```
There are a few workarounds for this, when embedding into the scope of a function `f`:
1. Use a mocked-up `globals()` including what was in `locals()`
1. Use the real `globals()` and copy over `locals()` into the real `globals()`
Both of the above require us to patch all the `nonlocal` statements referring to locals in the scope of `f` (and parent functions) to become `global`. In the first workaround, we use IPython with a `globals()` namespace that have been updated with both the original frame's `globals()` and `locals()`, either by running `IPython.start_ipython` with `user_ns` as the mocked-up `globals()` or patching `IPython.terminal.embed.InteractiveShellEmbed` (instead of this, one could possibly keep calling `globals().update(locals())` for every statement executed but that is really inefficient and only works if the created child functions do not modify the referenced variables). In this workaround, references to real globals in created functions will no longer access/modify the real global variable, but a copy, hence diverging from functions created by other means which use the real globals. In the second workaround, the real global namespace gets polluted and globals might have their values incorrectly overwritten by the locals, causing some existing functions to behave incorrectly.
## An almost perfect embedded shell
Recognising the need for code to "just work" when pasting it unedited into the shell, we developed a novel way to wrap the code such that we are able to edit the closure of each wrapper such that they all use the same [variable cells](https://docs.python.org/3/c-api/cell.html), hence they would access and modify the same variable. We even allow embedded shells to make permanent modifications to variables under certain conditions. More details are presented in the docs for the API. The following showcases some features:
```python3
def test_embed():
x0 = 0 # not closed over
x1 = 1 # used only in `f`
x2 = 2 # used in `f` and `g`
def f(y0, y1):
y0: int # not closed over
y1: int = y0 + y1 # used in `g`
nonlocal x1
nonlocal x2
def g():
nonlocal x2, y1
x2 += 10
y1 += 10
x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]
print(x1, x2, y0, y1)
# passing the enclosing function allows variables from the parent scopes
# which were closed over to be accessed and modified (x1 and x2)
# note that none of the shell will see `x0`
embed(funcs=[f])
# run: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()
x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]
print(x1, x2, y0, y1)
# passing a closure over local variables allow the specified variables
# to be accessed and modified (y0 and y1)
embed(funcs=[f, lambda: (y0, y1)])
# run: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()
x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]
print(x1, x2, y0, y1)
f(3, 1)
print(x0, x1, x2)
```
```
101 102 103 104
...
In [1]: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()
...
301 312 203 214
...
In [1]: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()
...
501 522 403 424
0 501 522
```
## Exception hook
In order to inspect the live objects at the point of an exception, we can add a exception handler in `sys.excepthook` using our utility `add_except_hook`, and embedding a shell using the right frame's locals and globals will allow you to quickly figure out what went wrong. However, in general uncaught exceptions are problematic because it is hard to resume the execution. It is surprising that IPython manages to handle exceptions raised in the dynamic code gracefully, and we can make use of this feature, as we remarked at the start of the next section.
## Reloading after an exception
If your program does a lot of computation, and you are only modifying/developing a small piece, you would not want to keep restarting it just because many bugs with this small piece keep causing exceptions, which are generally unrecoverable in Python. However, if you have a perfect embedded shell, there is a workflow that can save you much time when editing a function. Every time you make an edit to a line (say line `i`), it may or may not cause an exception in lines `i` and onwards. You position an `embed()` to before line `i`, run the program, and when the shell appears, paste in all of the code from line `i` onwards. If it causes an exception on line `j`, you modify the code and paste in all the code from line `j` onwards. Repeat this process until there is no exception.
As this copying and pasting process is still tedious, we made it even easier, just decorate a function with `try_all_statements`, and the function will be split into statements to be run one-by-one. If any of them raises an exception, one can either drop into a shell (e.g. by entering 0 in place of the line number; see docs) to inspect the variables, or simply edit the original source code of the function and rerun starting from a certain statement onwards.
```python3
def test_try():
@try_all_statements
def f(x):
print(x)
# editing the below statement to `x = 1 / (x + 1)` after the exception
# is raised will allow it to continue
x = 1 / x # (x + 1)
print(x)
f(1)
# the below raises an exception
f(0)
# subsequent calls use the modified function
f(1)
f(0)
```
```
1
1.0
0
2024-07-03 13:02:56;runner;ipython_utils;759;INFO: exception raised
Traceback (most recent call last):
File "/working/ipython_utils/ipython_utils.py", line 757, in runner
ret = patched(i)
File "/working/ipython_utils/ipython_utils.py", line 1026, in f
x = 1 / x # (x + 1)
ZeroDivisionError: division by zero
filename [/working/ipython_utils/ipython_utils.py]: # after editing
function line num [1022]:
next statement line num [1026]:
1.0
1
0.5
0
1.0
```
We provide a magic variable `_ipy_magic_inner` to access the inner function which has the closure for all the local variables. This allows you to always be able to embed a shell within a `try_all_statements`-decorated function to modify any of the local variables as follows:
```python3
@try_all_statements
def f():
# ...
embed(_ipy_magic_inner)
# changes to local variables will be persistent
```
Raw data
{
"_id": null,
"home_page": null,
"name": "ipython-utils",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "limwz01 <117669574+limwz01@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/c0/db/dd13bd04f945c911e1c398063733510f656e342aa35626f726d882987e63/ipython_utils-1.0.0.tar.gz",
"platform": null,
"description": "# ipython_utils\n\nUtilities to make developing Python responsive and interactive using fully powered shells directly embedded at the target location or exception site, or dynamically reloading updated code\n\n## Introduction\n\nSometimes you develop a complex Python software with multiple modules and functions. Jupyter notebooks are messy and hinder refactoring into functions, encouraging bad programming practices, so you decide against using them. But there is nothing as convenient as having a shell to directly inspect the objects and having additions to your existing code applied on live objects immediately! **Now, you can!**\n\n## Installation\n\nEver since I've packaged this and published it on PyPI (https://pypi.org/project/ipython-utils/), you can just run `pip install ipython_utils`. After installation, the following should work:\n\n```python3\nfrom ipython_utils import embed\ndef outer_function():\n x = 0\n def target_function():\n print(x)\n y1 = 1\n y2 = 2\n y3 = 3\n y4 = 4\n def g():\n print(y1)\n embed() # no local variable persistence\n embed(target_function) # x can be updated, as target_function was closed over x\n print(x, y1, y2, y3, y4)\n embed([target_function, g]) # y1 can also be updated, as g was closed over y1\n print(x, y1, y2, y3, y4)\n # we can easily force a closure over y2 and y3, and pass it to embed,\n # allowing it to update y2 and y3\n embed([target_function, lambda: (y2, y3)])\n return target_function\n```\n\n## Limited functionality in default IPython embedded shell\n\nAn existing method is to call `IPython.embed()` at the end of your partially developed code. But the embedded IPython shell is quite limited, because executing dynamically compiled Python code must always have a specific `locals()` dictionary, and updates to `locals()` rarely causes an update of local variables (except perhaps in old Python versions). The default shell just has a copy of the local variables, and existing closures over them will not update the value of the copy. More importantly, all child functions and lambdas do not close over the local variables, instead leaving them as unresolved global names, which means even list comprehensions which use these [do not work](https://github.com/cknoll/ipydex/issues/3). This is shown in the small snippet below:\n\n```python3\ndef test_ipython():\n x = 0\n y = 1\n def f(z, w):\n nonlocal y\n def g():\n nonlocal w\n w += 10\n y += 20\n print(x, y, z, w)\n IPython.embed()\n f(2, 3)\ntest_ipython()\n```\n\n```\n0 21 2 3\nPython 3.8.10 (default, Nov 22 2023, 10:22:35) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.\n\nIn [1]: x, y, z, w\nOut[1]: (0, 21, 2, 3)\n\nIn [2]: g()\n\nIn [3]: x, y, z, w\nOut[3]: (0, 21, 2, 3)\n\nIn [4]: (lambda: x)()\n...\nNameError: name 'x' is not defined\n\nIn [5]: [x for i in range(10)]\n...\nNameError: name 'x' is not defined\n\nIn [6]: t = 1; [t for i in range(10)]\n...\nNameError: name 't' is not defined\n```\n\nThere are a few workarounds for this, when embedding into the scope of a function `f`:\n1. Use a mocked-up `globals()` including what was in `locals()`\n1. Use the real `globals()` and copy over `locals()` into the real `globals()`\n\nBoth of the above require us to patch all the `nonlocal` statements referring to locals in the scope of `f` (and parent functions) to become `global`. In the first workaround, we use IPython with a `globals()` namespace that have been updated with both the original frame's `globals()` and `locals()`, either by running `IPython.start_ipython` with `user_ns` as the mocked-up `globals()` or patching `IPython.terminal.embed.InteractiveShellEmbed` (instead of this, one could possibly keep calling `globals().update(locals())` for every statement executed but that is really inefficient and only works if the created child functions do not modify the referenced variables). In this workaround, references to real globals in created functions will no longer access/modify the real global variable, but a copy, hence diverging from functions created by other means which use the real globals. In the second workaround, the real global namespace gets polluted and globals might have their values incorrectly overwritten by the locals, causing some existing functions to behave incorrectly.\n\n## An almost perfect embedded shell\n\nRecognising the need for code to \"just work\" when pasting it unedited into the shell, we developed a novel way to wrap the code such that we are able to edit the closure of each wrapper such that they all use the same [variable cells](https://docs.python.org/3/c-api/cell.html), hence they would access and modify the same variable. We even allow embedded shells to make permanent modifications to variables under certain conditions. More details are presented in the docs for the API. The following showcases some features:\n\n```python3\ndef test_embed():\n\n x0 = 0 # not closed over\n x1 = 1 # used only in `f`\n x2 = 2 # used in `f` and `g`\n\n def f(y0, y1):\n y0: int # not closed over\n y1: int = y0 + y1 # used in `g`\n nonlocal x1\n nonlocal x2\n\n def g():\n nonlocal x2, y1\n x2 += 10\n y1 += 10\n\n x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]\n print(x1, x2, y0, y1)\n # passing the enclosing function allows variables from the parent scopes\n # which were closed over to be accessed and modified (x1 and x2)\n # note that none of the shell will see `x0`\n embed(funcs=[f])\n # run: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()\n x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]\n print(x1, x2, y0, y1)\n # passing a closure over local variables allow the specified variables\n # to be accessed and modified (y0 and y1)\n embed(funcs=[f, lambda: (y0, y1)])\n # run: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()\n x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]\n print(x1, x2, y0, y1)\n\n f(3, 1)\n print(x0, x1, x2)\n```\n\n```\n101 102 103 104\n...\nIn [1]: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()\n...\n301 312 203 214\n...\nIn [1]: x1, x2, y0, y1 = [x + 100 for x in [x1, x2, y0, y1]]; g()\n...\n501 522 403 424\n0 501 522\n```\n\n## Exception hook\n\nIn order to inspect the live objects at the point of an exception, we can add a exception handler in `sys.excepthook` using our utility `add_except_hook`, and embedding a shell using the right frame's locals and globals will allow you to quickly figure out what went wrong. However, in general uncaught exceptions are problematic because it is hard to resume the execution. It is surprising that IPython manages to handle exceptions raised in the dynamic code gracefully, and we can make use of this feature, as we remarked at the start of the next section.\n\n## Reloading after an exception\n\nIf your program does a lot of computation, and you are only modifying/developing a small piece, you would not want to keep restarting it just because many bugs with this small piece keep causing exceptions, which are generally unrecoverable in Python. However, if you have a perfect embedded shell, there is a workflow that can save you much time when editing a function. Every time you make an edit to a line (say line `i`), it may or may not cause an exception in lines `i` and onwards. You position an `embed()` to before line `i`, run the program, and when the shell appears, paste in all of the code from line `i` onwards. If it causes an exception on line `j`, you modify the code and paste in all the code from line `j` onwards. Repeat this process until there is no exception.\n\nAs this copying and pasting process is still tedious, we made it even easier, just decorate a function with `try_all_statements`, and the function will be split into statements to be run one-by-one. If any of them raises an exception, one can either drop into a shell (e.g. by entering 0 in place of the line number; see docs) to inspect the variables, or simply edit the original source code of the function and rerun starting from a certain statement onwards.\n\n```python3\ndef test_try():\n\n @try_all_statements\n def f(x):\n print(x)\n # editing the below statement to `x = 1 / (x + 1)` after the exception\n # is raised will allow it to continue\n x = 1 / x # (x + 1)\n print(x)\n\n f(1)\n # the below raises an exception\n f(0)\n # subsequent calls use the modified function\n f(1)\n f(0)\n```\n\n```\n1\n1.0\n0\n2024-07-03 13:02:56;runner;ipython_utils;759;INFO: exception raised\nTraceback (most recent call last):\n File \"/working/ipython_utils/ipython_utils.py\", line 757, in runner\n ret = patched(i)\n File \"/working/ipython_utils/ipython_utils.py\", line 1026, in f\n x = 1 / x # (x + 1)\nZeroDivisionError: division by zero\nfilename [/working/ipython_utils/ipython_utils.py]: # after editing\nfunction line num [1022]:\nnext statement line num [1026]:\n1.0\n1\n0.5\n0\n1.0\n```\n\nWe provide a magic variable `_ipy_magic_inner` to access the inner function which has the closure for all the local variables. This allows you to always be able to embed a shell within a `try_all_statements`-decorated function to modify any of the local variables as follows:\n\n```python3\n@try_all_statements\ndef f():\n # ...\n embed(_ipy_magic_inner)\n # changes to local variables will be persistent\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Utilities to make developing Python responsive and interactive using fully powered shells directly embedded at the target location or exception site, or dynamically reloading updated code",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/limwz01/ipython_utils",
"Issues": "https://github.com/limwz01/ipython_utils/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8f0cb05227c63c4946d4c31075e4453a70e8e985c5cbb406145bb567e23e7775",
"md5": "9fac76389db527766bdb4d0340ec8fb3",
"sha256": "47d27ddb4478a77344fd8aeeba3c019bc8742ba327dafa3a23d0c9b2e76924ea"
},
"downloads": -1,
"filename": "ipython_utils-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9fac76389db527766bdb4d0340ec8fb3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 24534,
"upload_time": "2024-12-22T02:55:14",
"upload_time_iso_8601": "2024-12-22T02:55:14.908896Z",
"url": "https://files.pythonhosted.org/packages/8f/0c/b05227c63c4946d4c31075e4453a70e8e985c5cbb406145bb567e23e7775/ipython_utils-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c0dbdd13bd04f945c911e1c398063733510f656e342aa35626f726d882987e63",
"md5": "8eec75b0ef27ab26b9bb25bd14368499",
"sha256": "1b73380067e440675009f7b65bcec25e755e348474a085062319feb51a21c083"
},
"downloads": -1,
"filename": "ipython_utils-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "8eec75b0ef27ab26b9bb25bd14368499",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 25079,
"upload_time": "2024-12-22T02:55:16",
"upload_time_iso_8601": "2024-12-22T02:55:16.403552Z",
"url": "https://files.pythonhosted.org/packages/c0/db/dd13bd04f945c911e1c398063733510f656e342aa35626f726d882987e63/ipython_utils-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-22 02:55:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "limwz01",
"github_project": "ipython_utils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ipython-utils"
}
