# Dynamic Python (DyPy): a toolset for dynamic python code manipulations
Have you ever found yourself coding up boilerplates to handle different scenarios? It's most likely that you have thought about using python's `eval` or `exec` in order to decouple some part of your code to be specified and evaluated later on at run-time. But, you have probably also come accross people who discourage you from using `eval`/`exec`. This is because programs with `eval`/`exec` are considered vulnarable and can be used to execute malicious code.
While this is true in general, but in many cases, you do not care about security concerns, and your highest priority is implementing a quick solution to a general problem which could be solved by using `eval` or `exec`. This is where dypy comes in. It allows you to use `eval` and `exec` effectively by providing utilities to dynamically compile code, lookup variables and lazily evaluate them.
**Table of Contents**
- [Dynamic Python (DyPy): a toolset for dynamic python code manipulations](#dynamic-python-dypy-a-toolset-for-dynamic-python-code-manipulations)
- [Installation](#installation)
- [Usage](#usage)
- [Dynamic Evaluation](#dynamic-evaluation)
- [Variable Lookup](#variable-lookup)
- [Dynamic Function Evaluation](#dynamic-function-evaluation)
- [Variable Assignment](#variable-assignment)
- [License](#license)
- [Citation](#citation)
## Installation
```bash
pip install dypy
```
## Usage
### Dynamic Evaluation
You can use `dypy.eval` to combine the functionality of `dypy.eval_function` (see [here](#dynamic-function-evaluation)) and `dypy.get_value` (see [here](#variable-lookup)). You can do as follows:
```python
import dypy
dypy.eval("math.cos") # <function <lambda> at MEM_ADDRESS> (math is imported through get_value)
dypy.eval("math.cos", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored (and math is imported through get_value)
dypy.eval("def my_function(x): return x + 1", function_of_interest="my_function") # <function my_function at MEM_ADDRESS>
dypy.eval("def my_function(x): return x + y", function_of_interest="my_function", context={"y": 2})(2) # 4
```
#### Variable Lookup
You can use `dypy.get_value` to lookup a variable as a string and then evaluate it. This is useful when you want to use a variable that is not defined in the current scope. You can do as follows:
```python
from dypy import get_value
get_value("math.pi") # 3.141592653589793
get_value("math.cos(0)") # won't evaluate, this is not a variable but a call to a variable
get_value("math.cos")(0) # 1.0
import math
get_value("cos", context=math) # math.cos
get_value("something_that_does_not_exist") # raises NameError
get_value("something_that_does_not_exist", strict=False) # None
```
`get_value` supports looking up variables in a module or package in your current working directory as well (as opposed to python's `import` which only supports looking up variables in the python standard library and installed packages). This is useful when you want to create a script that can be run from anywhere and still be able to access variables in the current working directory.
For example, imagine you create your own python package with a runnable script that sorts files in a directory. You can use `get_value` to lookup a `config.sort` function in the current working directory. This way, you can create a `config.py` file in the current working directory and define your own sorting function. Or use a default sorting function if the `config.py` file does not exist.
Your code would look like this:
```python
from dypy import get_value
def sort_files():
sort_function = get_value("config.sort", strict=False) or default_sort
# do something with sort
```
Although this example is somewhat contrived, such a use case is very common in data science and machine learning. Imagine writing a package for training a Deep Learning model. You can use `get_value` to lookup custom Dataset classes and model implementations and this way, people can use your package without the need to modifying your code, because they can simply inject their own implementations in places where you have used `get_value`.
Another potential use case is defining Neural Network layers. You can use `get_value` to lookup custom layers and use them in your model. Such as:
```python
from dypy import get_value
import torch
class MyLinearBlock(torch.nn.Module):
def __init__(self, in_features, out_features, activation="torch.nn.ReLU"):
super().__init__()
self.linear = torch.nn.Linear(in_features, out_features)
self.activation = get_value(activation, strict=False)
def forward(self, x):
x = self.linear(x)
x = self.activation(x) if self.activation else x
return x
```
This way, you can change the activation function by simply changing the `activation` argument. For example, you can use `torch.nn.Sigmoid` or `torch.nn.Tanh` or even a custom activation function that you have defined in the current working directory or use one from a 3rd party package.
#### Dynamic Function Evaluation
What if you want to generate python programs dynamically? Meaning that you have a string that contains python code and you want to inject it into your program. You can use `dypy.eval_function` to evaluate a piece of code and retrieve a function. You can do as follows:
```python
from dypy import eval_function
eval_function("lambda x: x + 1") # <function <lambda> at MEM_ADDRESS>
eval_function("def my_function(x): return x + 1") # wont work, this is not a function,
# but a code block and you need to mention your function_of_interest in that code block
eval_function("def my_function(x): return x + 1", function_of_interest="my_function") # <function my_function at MEM_ADDRESS>
```
`eval_function` accepts three types of function descriptors:
1. A lambda function, e.g. `lambda x: x + 1`, or a code which evaluates to a callable object, `math.cos` (in this case, the values being looked up should be present in the evaluation context, more on this later).
2. A code block, which can contain multiple lines and functions, in which case you need to specify the name of the function of interest using the `function_of_interest` argument. `eval_function` will evaluate the code block and retrieve your function of interest.
3. A dictionary of "code", ["context", "function_of_interest"] pairs. Useful when you are using `eval_function` on top of a configuration file, in which case you can specify the code and the context for each function of interest.
When evaluating a function descriptor, you can specify a context in which the code will be evaluated. This is useful when you want to use variables that are not defined in the current scope. `dypy` has a context registry that you can use as a global context for all your function evaluations.
```python
from dypy import eval_function, register_context
import math
register_context(math, "math") # register math package as a context
# you can also use register(math), which will use the name of the package as the context name
eval_function("math.cos") # <function <lambda> at MEM_ADDRESS> (math is looked up through the context registry)
```
You can also specify a context for each function evaluation using the `context` argument. The context is a dictionary of variable names and their values. You can do as follows:
```python
eval_function("def my_function(x): return x + y", function_of_interest="my_function", context={"y": 1})(2) # 3
```
You can also optionally set `dynamic_args=True`, when evaluating a function. This will create a function that intelligently evaluates its arguments, by wrapping it using `dypy.dynamic_args_wrapper`. Functions wrapped by `dypy.dynamic_args_wrapper` preprocess arguments passed to them, and ignore arguments that are not defined in the function signature. For instance:
```python
eval_function("lambda x: x + 1", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored
```
### Variable Assignment
There are times when you want to assign a variable in a dynamic manner. Meaning that you want to change a variable's value that is not necessarily defined in the current scope. You can use `dypy.set_value` to do so. You can do as follows:
```python
from dypy import set_value
set_value("some_package.my_function", lambda x: x + 1)
# changing the value of pi in math package
set_value("math.pi", 3.14)
# now if you import math, math.pi will be 3.14
import math
math.pi # 3.14
```
## License
dypy is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text.
## Citation
If you use dypy in your research, please cite this repository as described in [CITATION.cff](CITATION.cff).
Raw data
{
"_id": null,
"home_page": "https://github.com/vahidzee/dypy",
"name": "dypy",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "dynamic coding,dynamic functions,lazy evaluation",
"author": "Vahid Zehtab, Hamid Kamkari",
"author_email": "vahid@zehtab.me, hamidrezakamkari@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/8b/0e/1b7bfffd5f917b9b1d59d2ab12efd5c001542b0c1aa8f3f64c7a6b7bd7a6/dypy-0.0.7.tar.gz",
"platform": null,
"description": "# Dynamic Python (DyPy): a toolset for dynamic python code manipulations\n\nHave you ever found yourself coding up boilerplates to handle different scenarios? It's most likely that you have thought about using python's `eval` or `exec` in order to decouple some part of your code to be specified and evaluated later on at run-time. But, you have probably also come accross people who discourage you from using `eval`/`exec`. This is because programs with `eval`/`exec` are considered vulnarable and can be used to execute malicious code.\n\nWhile this is true in general, but in many cases, you do not care about security concerns, and your highest priority is implementing a quick solution to a general problem which could be solved by using `eval` or `exec`. This is where dypy comes in. It allows you to use `eval` and `exec` effectively by providing utilities to dynamically compile code, lookup variables and lazily evaluate them.\n\n**Table of Contents**\n- [Dynamic Python (DyPy): a toolset for dynamic python code manipulations](#dynamic-python-dypy-a-toolset-for-dynamic-python-code-manipulations)\n - [Installation](#installation)\n - [Usage](#usage)\n - [Dynamic Evaluation](#dynamic-evaluation)\n - [Variable Lookup](#variable-lookup)\n - [Dynamic Function Evaluation](#dynamic-function-evaluation)\n - [Variable Assignment](#variable-assignment)\n - [License](#license)\n - [Citation](#citation)\n\n## Installation\n\n```bash\npip install dypy\n```\n\n## Usage\n\n### Dynamic Evaluation\n\nYou can use `dypy.eval` to combine the functionality of `dypy.eval_function` (see [here](#dynamic-function-evaluation)) and `dypy.get_value` (see [here](#variable-lookup)). You can do as follows:\n\n```python\nimport dypy\n\ndypy.eval(\"math.cos\") # <function <lambda> at MEM_ADDRESS> (math is imported through get_value)\ndypy.eval(\"math.cos\", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored (and math is imported through get_value)\ndypy.eval(\"def my_function(x): return x + 1\", function_of_interest=\"my_function\") # <function my_function at MEM_ADDRESS>\ndypy.eval(\"def my_function(x): return x + y\", function_of_interest=\"my_function\", context={\"y\": 2})(2) # 4\n```\n\n\n#### Variable Lookup\n\nYou can use `dypy.get_value` to lookup a variable as a string and then evaluate it. This is useful when you want to use a variable that is not defined in the current scope. You can do as follows:\n\n```python\nfrom dypy import get_value\n\nget_value(\"math.pi\") # 3.141592653589793\nget_value(\"math.cos(0)\") # won't evaluate, this is not a variable but a call to a variable\nget_value(\"math.cos\")(0) # 1.0\n\nimport math\nget_value(\"cos\", context=math) # math.cos\n\nget_value(\"something_that_does_not_exist\") # raises NameError\nget_value(\"something_that_does_not_exist\", strict=False) # None\n```\n\n`get_value` supports looking up variables in a module or package in your current working directory as well (as opposed to python's `import` which only supports looking up variables in the python standard library and installed packages). This is useful when you want to create a script that can be run from anywhere and still be able to access variables in the current working directory. \n\nFor example, imagine you create your own python package with a runnable script that sorts files in a directory. You can use `get_value` to lookup a `config.sort` function in the current working directory. This way, you can create a `config.py` file in the current working directory and define your own sorting function. Or use a default sorting function if the `config.py` file does not exist. \n\nYour code would look like this:\n\n```python\nfrom dypy import get_value\n\ndef sort_files():\n sort_function = get_value(\"config.sort\", strict=False) or default_sort\n # do something with sort\n```\n\nAlthough this example is somewhat contrived, such a use case is very common in data science and machine learning. Imagine writing a package for training a Deep Learning model. You can use `get_value` to lookup custom Dataset classes and model implementations and this way, people can use your package without the need to modifying your code, because they can simply inject their own implementations in places where you have used `get_value`.\n\nAnother potential use case is defining Neural Network layers. You can use `get_value` to lookup custom layers and use them in your model. Such as:\n\n```python\nfrom dypy import get_value\nimport torch\n\nclass MyLinearBlock(torch.nn.Module):\n def __init__(self, in_features, out_features, activation=\"torch.nn.ReLU\"):\n super().__init__()\n self.linear = torch.nn.Linear(in_features, out_features)\n self.activation = get_value(activation, strict=False) \n\n def forward(self, x):\n x = self.linear(x)\n x = self.activation(x) if self.activation else x\n return x\n```\n\nThis way, you can change the activation function by simply changing the `activation` argument. For example, you can use `torch.nn.Sigmoid` or `torch.nn.Tanh` or even a custom activation function that you have defined in the current working directory or use one from a 3rd party package.\n\n#### Dynamic Function Evaluation\n\nWhat if you want to generate python programs dynamically? Meaning that you have a string that contains python code and you want to inject it into your program. You can use `dypy.eval_function` to evaluate a piece of code and retrieve a function. You can do as follows:\n\n```python\nfrom dypy import eval_function\n\neval_function(\"lambda x: x + 1\") # <function <lambda> at MEM_ADDRESS>\n\neval_function(\"def my_function(x): return x + 1\") # wont work, this is not a function, \n# but a code block and you need to mention your function_of_interest in that code block\n\neval_function(\"def my_function(x): return x + 1\", function_of_interest=\"my_function\") # <function my_function at MEM_ADDRESS>\n```\n\n`eval_function` accepts three types of function descriptors:\n\n1. A lambda function, e.g. `lambda x: x + 1`, or a code which evaluates to a callable object, `math.cos` (in this case, the values being looked up should be present in the evaluation context, more on this later).\n2. A code block, which can contain multiple lines and functions, in which case you need to specify the name of the function of interest using the `function_of_interest` argument. `eval_function` will evaluate the code block and retrieve your function of interest.\n3. A dictionary of \"code\", [\"context\", \"function_of_interest\"] pairs. Useful when you are using `eval_function` on top of a configuration file, in which case you can specify the code and the context for each function of interest.\n\nWhen evaluating a function descriptor, you can specify a context in which the code will be evaluated. This is useful when you want to use variables that are not defined in the current scope. `dypy` has a context registry that you can use as a global context for all your function evaluations.\n\n```python\nfrom dypy import eval_function, register_context\nimport math\n\nregister_context(math, \"math\") # register math package as a context\n# you can also use register(math), which will use the name of the package as the context name\n\neval_function(\"math.cos\") # <function <lambda> at MEM_ADDRESS> (math is looked up through the context registry)\n```\n\nYou can also specify a context for each function evaluation using the `context` argument. The context is a dictionary of variable names and their values. You can do as follows:\n\n```python\neval_function(\"def my_function(x): return x + y\", function_of_interest=\"my_function\", context={\"y\": 1})(2) # 3\n```\n\nYou can also optionally set `dynamic_args=True`, when evaluating a function. This will create a function that intelligently evaluates its arguments, by wrapping it using `dypy.dynamic_args_wrapper`. Functions wrapped by `dypy.dynamic_args_wrapper` preprocess arguments passed to them, and ignore arguments that are not defined in the function signature. For instance:\n\n```python\neval_function(\"lambda x: x + 1\", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored\n```\n\n### Variable Assignment\n\nThere are times when you want to assign a variable in a dynamic manner. Meaning that you want to change a variable's value that is not necessarily defined in the current scope. You can use `dypy.set_value` to do so. You can do as follows:\n\n```python\nfrom dypy import set_value\n\nset_value(\"some_package.my_function\", lambda x: x + 1)\n\n# changing the value of pi in math package\nset_value(\"math.pi\", 3.14) \n\n# now if you import math, math.pi will be 3.14\nimport math\nmath.pi # 3.14\n```\n\n## License\n\ndypy is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text.\n\n## Citation\n\nIf you use dypy in your research, please cite this repository as described in [CITATION.cff](CITATION.cff).\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A toolset for dynamic python code manipulations",
"version": "0.0.7",
"split_keywords": [
"dynamic coding",
"dynamic functions",
"lazy evaluation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7699d3d709a086a2fcf9e35f9ecdac0570fb1ece04f7a682c87ffca70542063d",
"md5": "74fefb54651fac54d1d09d792c3a1aaf",
"sha256": "128ca47882324d3c6e9538e1290c2ea2029f10ae85188a0b517e6cce01b645f5"
},
"downloads": -1,
"filename": "dypy-0.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "74fefb54651fac54d1d09d792c3a1aaf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 21075,
"upload_time": "2023-03-22T23:42:58",
"upload_time_iso_8601": "2023-03-22T23:42:58.659762Z",
"url": "https://files.pythonhosted.org/packages/76/99/d3d709a086a2fcf9e35f9ecdac0570fb1ece04f7a682c87ffca70542063d/dypy-0.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8b0e1b7bfffd5f917b9b1d59d2ab12efd5c001542b0c1aa8f3f64c7a6b7bd7a6",
"md5": "4d4608ed6b0458ddb6c9e3eb6d84a9bd",
"sha256": "ba9ad5238d5c9d031ae645449178c7523df79fe8fce8fa7f34fd5da6a0a2fd12"
},
"downloads": -1,
"filename": "dypy-0.0.7.tar.gz",
"has_sig": false,
"md5_digest": "4d4608ed6b0458ddb6c9e3eb6d84a9bd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 20245,
"upload_time": "2023-03-22T23:43:01",
"upload_time_iso_8601": "2023-03-22T23:43:01.007642Z",
"url": "https://files.pythonhosted.org/packages/8b/0e/1b7bfffd5f917b9b1d59d2ab12efd5c001542b0c1aa8f3f64c7a6b7bd7a6/dypy-0.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-22 23:43:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "vahidzee",
"github_project": "dypy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dypy"
}
JSON
