expr-codegen


Nameexpr-codegen JSON
Version 0.10.9 PyPI version JSON
download
home_pageNone
Summarysymbol expression to polars expression tool
upload_time2024-12-30 06:22:57
maintainerNone
docs_urlNone
authorNone
requires_python>=3.9
licenseBSD 3-Clause License Copyright (c) 2023, wukan Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
keywords polars expression talib
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # expr_codegen 表达式转译器

## 项目背景

在本人新推出[polars_ta](https://github.com/wukan1986/polars_ta)这个库后，再回头反思`expr_codegen`是什么。

> `expr_codegen`本质是`DSL`，领域特定语⾔(Domain Specific Language)。但它没有定义新的语法

它解决了两个问题:

1. `polars_ta`已经能很方便的写出特征计算表达式，但遇到`混用时序与截面`的表达式，利用`expr_codegen`能自动分组大大节省工作
2. `expr_codegen`利用了`Common Subexpression Elimination`公共子表达式消除，大量减少重复计算，提高效率

就算在量化领域，初级研究员局限于时序指标，仅用`polars_ta`即可，中高级研究员使用截面指标，推荐用`expr_codegen`

虽然现在此项目与`polars_ta`依赖非常紧密，但也是支持翻译成其它库,如`pandas / cudf.pandas`，只是目前缺乏一个比较简易的库

## 在线演示

https://exprcodegen.streamlit.app

初级用户可以直接访问此链接进行表达式转译，不需要另外安装软件。(此工具免费部署在国外，打开可能有些慢)

更完整示例访问[alpha_examples](https://github.com/wukan1986/alpha_examples)

## 使用示例

```python
import sys
from io import StringIO

from expr_codegen import codegen_exec


def _code_block_1():
    # 因子编辑区，可利用IDE的智能提示在此区域编辑因子
    LOG_MC_ZS = cs_mad_zscore(log1p(market_cap))


def _code_block_2():
    # 模板中已经默认导入了from polars_ta.prefix下大量的算子，但
    # talib在模板中没有默认导入。这种写法可实现在生成的代码中导入
    from polars_ta.prefix.talib import ts_LINEARREG_SLOPE  # noqa

    # 1. 下划线开头的变量只是中间变量,会被自动更名，最终输出时会被剔除
    # 2. 下划线开头的变量可以重复使用。多个复杂因子多行书写时有重复中间变时不再冲突
    _avg = ts_mean(corr, 20)
    _std = ts_std_dev(corr, 20)
    _beta = ts_LINEARREG_SLOPE(corr, 20)

    # 3. 下划线开头的变量有环循环赋值。在调试时可快速用注释进行切换
    _avg = cs_mad_zscore_resid(_avg, LOG_MC_ZS, ONE)
    _std = cs_mad_zscore_resid(_std, LOG_MC_ZS, ONE)
    # _beta = cs_mad_zscore_resid(_beta, LOG_MC_ZS, ONE)

    _corr = cs_zscore(_avg) + cs_zscore(_std)
    CPV = cs_zscore(_corr) + cs_zscore(_beta)


code = StringIO()

df = None  # 替换成真实的polars数据
df = codegen_exec(df, _code_block_1, _code_block_2, output_file=sys.stdout)  # 打印代码
df = codegen_exec(df, _code_block_1, _code_block_2, output_file="output.py")  # 保存到文件
df = codegen_exec(df, _code_block_1, _code_block_2)  # 只执行，不保存代码
df = codegen_exec(df, _code_block_1, _code_block_2, output_file=code)  # 保存到字符串
code.seek(0)
code.read()  # 读取代码

df = codegen_exec(df.lazy(), _code_block_1, _code_block_2).collect()  # Lazy CPU
df = codegen_exec(df.lazy(), _code_block_1, _code_block_2).collect(engine="gpu")  # Lazy GPU
```

## 目录结构

```commandline
│  requirements.txt # 通过`pip install -r requirements.txt`安装依赖
├─data
│      prepare_date.py # 准备数据
├─examples
│      demo_express.py # 速成示例。演示如何将表达式转换成代码
│      demo_exec_pl.py # 演示调用转换后代码并绘图
│      demo_transformer.py # 演示将第三方表达式转成内部表达式
│      output.py # 结果输出。可不修改代码，直接被其它项目导入
│      show_tree.py # 画表达式树形图。可用于分析对比优化结果
│      sympy_define.py # 符号定义，由于太多地方重复使用到，所以统一提取到此处
├─expr_codegen
│   │  expr.py # 表达式处理基本函数
│   │  tool.py # 核心工具代码
│   ├─polars
│   │  │  code.py # 针对polars语法的代码生成功能
│   │  │  template.py.j2 # `Jinja2`模板。用于生成对应py文件，一般不需修改
│   │  │  printer.py # 继承于`Sympy`中的`StrPrinter`，添加新函数时可能需修改此文件
```

## 工作原理

本项目依赖于`sympy`项目。所用到的主要函数如下：

1. `simplify`: 对复杂表达式进行化简
2. `cse`: `Common Subexpression Elimination`公共子表达式消除
3. `StrPrinter`: 根据不同的函数输出不同字符串。定制此代码可以支持其它语种或库

因为`groupby`,`sort`都比较占用时间。如果提前将公式分类，不同的类别使用不同的`groupby`，可以减少计算时间。

1. `ts_xxx(ts_xxx)`: 可在同一`groupby`中进行计算
2. `cs_xxx(cs_xxx)`: 可在同一`groupby`中进行计算
3. `ts_xxx(cs_xxx)`: 需在不同`groupby`中进行计算
4. `cs_xxx(ts_xxx(cs_xxx))`: 需三不同`groupby`中进行计算
5. `gp_xxx(aa, )+gp_xxx(bb, )`: 因`aa`,`bb`不同，需在两不同`groupby`中进行计算

所以

1. 需要有一个函数能获取当前表达式的类别(`get_current`)和子表达式的类别(`get_children`)
2. 如果当前类别与子类别不同就可以提取出短公式(`extract`)。不同层的同类别表达式有先后关系，不能放同一`groupby`
3. 利用`cse`的特点，将长表达式替换成前期提取出来的短表达式。然后输入到有向无环图(`DAG`)
4. 利用有向无环图的流转，进行分层。同一层的`ts`,`cs`,`gp`不区分先后
5. 同一层对`ts`,`cs`,`gp`分组，然后生成代码(`codegen`)即可

隐含信息

1. `ts`: sort(by=[ASSET, DATE]).groupby(by=[ASSET], maintain_order=True)
2. `cs`: sort(by=[DATE]).groupby(by=[DATE], maintain_order=False)
3. `gp`: sort(by=[DATE, GROUP]).groupby(by=[DATE, GROUP], maintain_order=False)

即

1. 时序函数隐藏了两个字段`ASSET, DATE`，横截面函数了隐藏了一个字段`DATE`
2. 分组函数转入了一个字段`GROUP`，同时隐藏了一个字段`DATE`

两种分类方法

1. 根据算子前缀分类(`get_current_by_prefix`)，限制算子必需以`ts_`、`cs_`、`gp_`开头
2. 根据算子全名分类(`get_current_by_name`), 不再限制算子名。比如`cs_rank`可以叫`rank`

## Null处理

`null`是如何产生的？

1. 停牌导致。在计算前就直接过滤掉了，不会对后续计算产生影响。
2. 不同品种交易时段不同
3. 计算产生。`null`在数列两端不影响后续时序算子结果，但中间出现`null`会影响。例如： `if_else(close<2, None, close)`

https://github.com/pola-rs/polars/issues/12925#issuecomment-2552764629

非常棒的点子，总结下来有两种实现方式：

1. 将`null`分成一组，`not_null`分成另一组。要调用两次
2. 仅一组，但复合排序，将`null`排在前面，`not_null`排后面。只调用一次，略快一些

```python
X1 = (ts_returns(CLOSE, 3)).over(CLOSE.is_not_null(), _ASSET_, order_by=_DATE_),
X2 = (ts_returns(CLOSE, 3)).over(_ASSET_, order_by=[CLOSE.is_not_null(), _DATE_]),
X3 = (ts_returns(CLOSE, 3)).over(_ASSET_, order_by=_DATE_),
```

第2种开头的`null`区域，是否影响结果由算子所决定，特别时是多列输入时`null`区域可能有数据

1. `over_null='partition_by'`。分到两个区域
2. `over_null='order_by'`。分到一个区域，`null`排在前面
3. `over_null=None`。不处理，直接调用，速度更快。如果确信不会中段产生`null`建议使用此参数

## `expr_codegen`局限性

1. `DAG`只能增加列无法删除。增加列时，遇到同名列会覆盖
2. 不支持`删除行`，但可以添加删除标记列，然后在外进行删除行。删除行影响了所有列，不满足`DAG`
3. 不支持`重采样`，原理同不支持删除行。需在外进行
4. 可以将`删除行`与`重采样`做为分割线，一大块代码分成多个`DAG`串联。复杂不易理解，所以最终没有实现

## 特别语法

1. 支持`C?T:F`三元表达式（仅可字符串中使用），底层会先转成`C or True if( T )else F`，然后修正成`T if C else F`，最后转成`if_else(C,T,F)`。支持与`if else`混用
2. `(A<B)*-1`,底层将转换成`int_(A<B)*-1`
3. 为防止`A==B`被`sympy`替换成`False`，底层会换成`Eq(A,B)`
4. `A^B`的含义与`convert_xor`参数有关，`convert_xor=True`底层会转换成`Pow(A,B)`，反之为`Xor(A,B)`。默认为`False`，用`**`表示乘方
5. 支持`A&B&C`，但不支持`A==B==C`。如果C是布尔，AB是数值，可手工替换成`(A==B)==C`。如果ABC是数值需手工替换成`(A==B)&(B==C)`
6. 不支持`A<=B<=C`，需手工替换成`(A<=B)&(B<=C)`
7. 支持`A[0]+B[1]+C[2]`，底层会转成`A+ts_delay(B,1)+ts_delay(C,2)`
8. 支持`~A`,底层会转换成`Not(A)`
9. `gp_`开头的函数都会返回对应的`cs_`函数。如`gp_func(A,B,C)`会替换成`cs_func(B,C)`,其中`A`用在了`groupby([date, A])`
10. 支持`A,B,C=MACD()`元组解包，在底层会替换成

```python
_x_0 = MACD()
A = unpack(_x_0, 0)
B = unpack(_x_0, 1)
C = unpack(_x_0, 2)
```

## 下划线开头的变量

1. 输出的数据，所有以`_`开头的列，最后会被自动删除。所以需要保留的变量一定不要以`_`开头
2. 为减少重复计算，自动添加了了中间变量，以`_x_`开头，如`_x_0`，`_x_1`等。最后会被自动删除
3. 单行表达式过长，或有重复计算，可以通过中间变量，将单行表达式改成多行。如果中间变量使用`_`开头，将会自动添加数字后缀，形成不同的变量，如`_A`会替换成`_A_0_`、`_A_1_`等。使用场景如下：
    1. 同一变量名，重复使用。本质是不同的变量
    2. 循环赋值，但`DAG`不支持有环。`=`号左右的同名变量其实是不同变量

## 转译结果示例

转译后的代码片段，详细代码请参考[Polars版](examples/output_polars.py)

```python
def func_0_ts__asset(df: pl.DataFrame) -> pl.DataFrame:
    df = df.sort(by=[_DATE_])
    # ========================================
    df = df.with_columns(
        _x_0=1 / ts_delay(OPEN, -1),
        LABEL_CC_1=(-CLOSE + ts_delay(CLOSE, -1)) / CLOSE,
    )
    # ========================================
    df = df.with_columns(
        LABEL_OO_1=_x_0 * ts_delay(OPEN, -2) - 1,
        LABEL_OO_2=_x_0 * ts_delay(OPEN, -3) - 1,
    )
    return df
```

转译后的代码片段，详细代码请参考[Pandas版](examples/output_pandas.py)

```python
def func_2_cs__date(df: pd.DataFrame) -> pd.DataFrame:
    # expr_4 = cs_rank(x_7)
    df["expr_4"] = (df["x_7"]).rank(pct=True)
    return df


def func_3_ts__asset__date(df: pd.DataFrame) -> pd.DataFrame:
    # expr_5 = -ts_corr(OPEN, CLOSE, 10)
    df["expr_5"] = -(df["OPEN"]).rolling(10).corr(df["CLOSE"])
    # expr_6 = ts_delta(OPEN, 10)
    df["expr_6"] = df["OPEN"].diff(10)
    return df

```

## 本地部署交互网页

只需运行`streamlit run streamlit_app.py`

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "expr-codegen",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.9",
    "maintainer_email": null,
    "keywords": "polars, expression, talib",
    "author": null,
    "author_email": "wukan <wu-kan@163.com>",
    "download_url": "https://files.pythonhosted.org/packages/63/85/43dc1128984d7bdec4b2a623efeec2dc1982423e9fcce1c51eef66eb1616/expr_codegen-0.10.9.tar.gz",
    "platform": null,
    "description": "# expr_codegen \u8868\u8fbe\u5f0f\u8f6c\u8bd1\u5668\n\n## \u9879\u76ee\u80cc\u666f\n\n\u5728\u672c\u4eba\u65b0\u63a8\u51fa[polars_ta](https://github.com/wukan1986/polars_ta)\u8fd9\u4e2a\u5e93\u540e\uff0c\u518d\u56de\u5934\u53cd\u601d`expr_codegen`\u662f\u4ec0\u4e48\u3002\n\n> `expr_codegen`\u672c\u8d28\u662f`DSL`\uff0c\u9886\u57df\u7279\u5b9a\u8bed\u2f94(Domain Specific Language)\u3002\u4f46\u5b83\u6ca1\u6709\u5b9a\u4e49\u65b0\u7684\u8bed\u6cd5\n\n\u5b83\u89e3\u51b3\u4e86\u4e24\u4e2a\u95ee\u9898:\n\n1. `polars_ta`\u5df2\u7ecf\u80fd\u5f88\u65b9\u4fbf\u7684\u5199\u51fa\u7279\u5f81\u8ba1\u7b97\u8868\u8fbe\u5f0f\uff0c\u4f46\u9047\u5230`\u6df7\u7528\u65f6\u5e8f\u4e0e\u622a\u9762`\u7684\u8868\u8fbe\u5f0f\uff0c\u5229\u7528`expr_codegen`\u80fd\u81ea\u52a8\u5206\u7ec4\u5927\u5927\u8282\u7701\u5de5\u4f5c\n2. `expr_codegen`\u5229\u7528\u4e86`Common Subexpression Elimination`\u516c\u5171\u5b50\u8868\u8fbe\u5f0f\u6d88\u9664\uff0c\u5927\u91cf\u51cf\u5c11\u91cd\u590d\u8ba1\u7b97\uff0c\u63d0\u9ad8\u6548\u7387\n\n\u5c31\u7b97\u5728\u91cf\u5316\u9886\u57df\uff0c\u521d\u7ea7\u7814\u7a76\u5458\u5c40\u9650\u4e8e\u65f6\u5e8f\u6307\u6807\uff0c\u4ec5\u7528`polars_ta`\u5373\u53ef\uff0c\u4e2d\u9ad8\u7ea7\u7814\u7a76\u5458\u4f7f\u7528\u622a\u9762\u6307\u6807\uff0c\u63a8\u8350\u7528`expr_codegen`\n\n\u867d\u7136\u73b0\u5728\u6b64\u9879\u76ee\u4e0e`polars_ta`\u4f9d\u8d56\u975e\u5e38\u7d27\u5bc6\uff0c\u4f46\u4e5f\u662f\u652f\u6301\u7ffb\u8bd1\u6210\u5176\u5b83\u5e93,\u5982`pandas / cudf.pandas`\uff0c\u53ea\u662f\u76ee\u524d\u7f3a\u4e4f\u4e00\u4e2a\u6bd4\u8f83\u7b80\u6613\u7684\u5e93\n\n## \u5728\u7ebf\u6f14\u793a\n\nhttps://exprcodegen.streamlit.app\n\n\u521d\u7ea7\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u8bbf\u95ee\u6b64\u94fe\u63a5\u8fdb\u884c\u8868\u8fbe\u5f0f\u8f6c\u8bd1\uff0c\u4e0d\u9700\u8981\u53e6\u5916\u5b89\u88c5\u8f6f\u4ef6\u3002(\u6b64\u5de5\u5177\u514d\u8d39\u90e8\u7f72\u5728\u56fd\u5916\uff0c\u6253\u5f00\u53ef\u80fd\u6709\u4e9b\u6162)\n\n\u66f4\u5b8c\u6574\u793a\u4f8b\u8bbf\u95ee[alpha_examples](https://github.com/wukan1986/alpha_examples)\n\n## \u4f7f\u7528\u793a\u4f8b\n\n```python\nimport sys\nfrom io import StringIO\n\nfrom expr_codegen import codegen_exec\n\n\ndef _code_block_1():\n    # \u56e0\u5b50\u7f16\u8f91\u533a\uff0c\u53ef\u5229\u7528IDE\u7684\u667a\u80fd\u63d0\u793a\u5728\u6b64\u533a\u57df\u7f16\u8f91\u56e0\u5b50\n    LOG_MC_ZS = cs_mad_zscore(log1p(market_cap))\n\n\ndef _code_block_2():\n    # \u6a21\u677f\u4e2d\u5df2\u7ecf\u9ed8\u8ba4\u5bfc\u5165\u4e86from polars_ta.prefix\u4e0b\u5927\u91cf\u7684\u7b97\u5b50\uff0c\u4f46\n    # talib\u5728\u6a21\u677f\u4e2d\u6ca1\u6709\u9ed8\u8ba4\u5bfc\u5165\u3002\u8fd9\u79cd\u5199\u6cd5\u53ef\u5b9e\u73b0\u5728\u751f\u6210\u7684\u4ee3\u7801\u4e2d\u5bfc\u5165\n    from polars_ta.prefix.talib import ts_LINEARREG_SLOPE  # noqa\n\n    # 1. \u4e0b\u5212\u7ebf\u5f00\u5934\u7684\u53d8\u91cf\u53ea\u662f\u4e2d\u95f4\u53d8\u91cf,\u4f1a\u88ab\u81ea\u52a8\u66f4\u540d\uff0c\u6700\u7ec8\u8f93\u51fa\u65f6\u4f1a\u88ab\u5254\u9664\n    # 2. \u4e0b\u5212\u7ebf\u5f00\u5934\u7684\u53d8\u91cf\u53ef\u4ee5\u91cd\u590d\u4f7f\u7528\u3002\u591a\u4e2a\u590d\u6742\u56e0\u5b50\u591a\u884c\u4e66\u5199\u65f6\u6709\u91cd\u590d\u4e2d\u95f4\u53d8\u65f6\u4e0d\u518d\u51b2\u7a81\n    _avg = ts_mean(corr, 20)\n    _std = ts_std_dev(corr, 20)\n    _beta = ts_LINEARREG_SLOPE(corr, 20)\n\n    # 3. \u4e0b\u5212\u7ebf\u5f00\u5934\u7684\u53d8\u91cf\u6709\u73af\u5faa\u73af\u8d4b\u503c\u3002\u5728\u8c03\u8bd5\u65f6\u53ef\u5feb\u901f\u7528\u6ce8\u91ca\u8fdb\u884c\u5207\u6362\n    _avg = cs_mad_zscore_resid(_avg, LOG_MC_ZS, ONE)\n    _std = cs_mad_zscore_resid(_std, LOG_MC_ZS, ONE)\n    # _beta = cs_mad_zscore_resid(_beta, LOG_MC_ZS, ONE)\n\n    _corr = cs_zscore(_avg) + cs_zscore(_std)\n    CPV = cs_zscore(_corr) + cs_zscore(_beta)\n\n\ncode = StringIO()\n\ndf = None  # \u66ff\u6362\u6210\u771f\u5b9e\u7684polars\u6570\u636e\ndf = codegen_exec(df, _code_block_1, _code_block_2, output_file=sys.stdout)  # \u6253\u5370\u4ee3\u7801\ndf = codegen_exec(df, _code_block_1, _code_block_2, output_file=\"output.py\")  # \u4fdd\u5b58\u5230\u6587\u4ef6\ndf = codegen_exec(df, _code_block_1, _code_block_2)  # \u53ea\u6267\u884c\uff0c\u4e0d\u4fdd\u5b58\u4ee3\u7801\ndf = codegen_exec(df, _code_block_1, _code_block_2, output_file=code)  # \u4fdd\u5b58\u5230\u5b57\u7b26\u4e32\ncode.seek(0)\ncode.read()  # \u8bfb\u53d6\u4ee3\u7801\n\ndf = codegen_exec(df.lazy(), _code_block_1, _code_block_2).collect()  # Lazy CPU\ndf = codegen_exec(df.lazy(), _code_block_1, _code_block_2).collect(engine=\"gpu\")  # Lazy GPU\n```\n\n## \u76ee\u5f55\u7ed3\u6784\n\n```commandline\n\u2502  requirements.txt # \u901a\u8fc7`pip install -r requirements.txt`\u5b89\u88c5\u4f9d\u8d56\n\u251c\u2500data\n\u2502      prepare_date.py # \u51c6\u5907\u6570\u636e\n\u251c\u2500examples\n\u2502      demo_express.py # \u901f\u6210\u793a\u4f8b\u3002\u6f14\u793a\u5982\u4f55\u5c06\u8868\u8fbe\u5f0f\u8f6c\u6362\u6210\u4ee3\u7801\n\u2502      demo_exec_pl.py # \u6f14\u793a\u8c03\u7528\u8f6c\u6362\u540e\u4ee3\u7801\u5e76\u7ed8\u56fe\n\u2502      demo_transformer.py # \u6f14\u793a\u5c06\u7b2c\u4e09\u65b9\u8868\u8fbe\u5f0f\u8f6c\u6210\u5185\u90e8\u8868\u8fbe\u5f0f\n\u2502      output.py # \u7ed3\u679c\u8f93\u51fa\u3002\u53ef\u4e0d\u4fee\u6539\u4ee3\u7801\uff0c\u76f4\u63a5\u88ab\u5176\u5b83\u9879\u76ee\u5bfc\u5165\n\u2502      show_tree.py # \u753b\u8868\u8fbe\u5f0f\u6811\u5f62\u56fe\u3002\u53ef\u7528\u4e8e\u5206\u6790\u5bf9\u6bd4\u4f18\u5316\u7ed3\u679c\n\u2502      sympy_define.py # \u7b26\u53f7\u5b9a\u4e49\uff0c\u7531\u4e8e\u592a\u591a\u5730\u65b9\u91cd\u590d\u4f7f\u7528\u5230\uff0c\u6240\u4ee5\u7edf\u4e00\u63d0\u53d6\u5230\u6b64\u5904\n\u251c\u2500expr_codegen\n\u2502   \u2502  expr.py # \u8868\u8fbe\u5f0f\u5904\u7406\u57fa\u672c\u51fd\u6570\n\u2502   \u2502  tool.py # \u6838\u5fc3\u5de5\u5177\u4ee3\u7801\n\u2502   \u251c\u2500polars\n\u2502   \u2502  \u2502  code.py # \u9488\u5bf9polars\u8bed\u6cd5\u7684\u4ee3\u7801\u751f\u6210\u529f\u80fd\n\u2502   \u2502  \u2502  template.py.j2 # `Jinja2`\u6a21\u677f\u3002\u7528\u4e8e\u751f\u6210\u5bf9\u5e94py\u6587\u4ef6\uff0c\u4e00\u822c\u4e0d\u9700\u4fee\u6539\n\u2502   \u2502  \u2502  printer.py # \u7ee7\u627f\u4e8e`Sympy`\u4e2d\u7684`StrPrinter`\uff0c\u6dfb\u52a0\u65b0\u51fd\u6570\u65f6\u53ef\u80fd\u9700\u4fee\u6539\u6b64\u6587\u4ef6\n```\n\n## \u5de5\u4f5c\u539f\u7406\n\n\u672c\u9879\u76ee\u4f9d\u8d56\u4e8e`sympy`\u9879\u76ee\u3002\u6240\u7528\u5230\u7684\u4e3b\u8981\u51fd\u6570\u5982\u4e0b\uff1a\n\n1. `simplify`: \u5bf9\u590d\u6742\u8868\u8fbe\u5f0f\u8fdb\u884c\u5316\u7b80\n2. `cse`: `Common Subexpression Elimination`\u516c\u5171\u5b50\u8868\u8fbe\u5f0f\u6d88\u9664\n3. `StrPrinter`: \u6839\u636e\u4e0d\u540c\u7684\u51fd\u6570\u8f93\u51fa\u4e0d\u540c\u5b57\u7b26\u4e32\u3002\u5b9a\u5236\u6b64\u4ee3\u7801\u53ef\u4ee5\u652f\u6301\u5176\u5b83\u8bed\u79cd\u6216\u5e93\n\n\u56e0\u4e3a`groupby`,`sort`\u90fd\u6bd4\u8f83\u5360\u7528\u65f6\u95f4\u3002\u5982\u679c\u63d0\u524d\u5c06\u516c\u5f0f\u5206\u7c7b\uff0c\u4e0d\u540c\u7684\u7c7b\u522b\u4f7f\u7528\u4e0d\u540c\u7684`groupby`\uff0c\u53ef\u4ee5\u51cf\u5c11\u8ba1\u7b97\u65f6\u95f4\u3002\n\n1. `ts_xxx(ts_xxx)`: \u53ef\u5728\u540c\u4e00`groupby`\u4e2d\u8fdb\u884c\u8ba1\u7b97\n2. `cs_xxx(cs_xxx)`: \u53ef\u5728\u540c\u4e00`groupby`\u4e2d\u8fdb\u884c\u8ba1\u7b97\n3. `ts_xxx(cs_xxx)`: \u9700\u5728\u4e0d\u540c`groupby`\u4e2d\u8fdb\u884c\u8ba1\u7b97\n4. `cs_xxx(ts_xxx(cs_xxx))`: \u9700\u4e09\u4e0d\u540c`groupby`\u4e2d\u8fdb\u884c\u8ba1\u7b97\n5. `gp_xxx(aa, )+gp_xxx(bb, )`: \u56e0`aa`,`bb`\u4e0d\u540c\uff0c\u9700\u5728\u4e24\u4e0d\u540c`groupby`\u4e2d\u8fdb\u884c\u8ba1\u7b97\n\n\u6240\u4ee5\n\n1. \u9700\u8981\u6709\u4e00\u4e2a\u51fd\u6570\u80fd\u83b7\u53d6\u5f53\u524d\u8868\u8fbe\u5f0f\u7684\u7c7b\u522b(`get_current`)\u548c\u5b50\u8868\u8fbe\u5f0f\u7684\u7c7b\u522b(`get_children`)\n2. \u5982\u679c\u5f53\u524d\u7c7b\u522b\u4e0e\u5b50\u7c7b\u522b\u4e0d\u540c\u5c31\u53ef\u4ee5\u63d0\u53d6\u51fa\u77ed\u516c\u5f0f(`extract`)\u3002\u4e0d\u540c\u5c42\u7684\u540c\u7c7b\u522b\u8868\u8fbe\u5f0f\u6709\u5148\u540e\u5173\u7cfb\uff0c\u4e0d\u80fd\u653e\u540c\u4e00`groupby`\n3. \u5229\u7528`cse`\u7684\u7279\u70b9\uff0c\u5c06\u957f\u8868\u8fbe\u5f0f\u66ff\u6362\u6210\u524d\u671f\u63d0\u53d6\u51fa\u6765\u7684\u77ed\u8868\u8fbe\u5f0f\u3002\u7136\u540e\u8f93\u5165\u5230\u6709\u5411\u65e0\u73af\u56fe(`DAG`)\n4. \u5229\u7528\u6709\u5411\u65e0\u73af\u56fe\u7684\u6d41\u8f6c\uff0c\u8fdb\u884c\u5206\u5c42\u3002\u540c\u4e00\u5c42\u7684`ts`,`cs`,`gp`\u4e0d\u533a\u5206\u5148\u540e\n5. \u540c\u4e00\u5c42\u5bf9`ts`,`cs`,`gp`\u5206\u7ec4\uff0c\u7136\u540e\u751f\u6210\u4ee3\u7801(`codegen`)\u5373\u53ef\n\n\u9690\u542b\u4fe1\u606f\n\n1. `ts`: sort(by=[ASSET, DATE]).groupby(by=[ASSET], maintain_order=True)\n2. `cs`: sort(by=[DATE]).groupby(by=[DATE], maintain_order=False)\n3. `gp`: sort(by=[DATE, GROUP]).groupby(by=[DATE, GROUP], maintain_order=False)\n\n\u5373\n\n1. \u65f6\u5e8f\u51fd\u6570\u9690\u85cf\u4e86\u4e24\u4e2a\u5b57\u6bb5`ASSET, DATE`\uff0c\u6a2a\u622a\u9762\u51fd\u6570\u4e86\u9690\u85cf\u4e86\u4e00\u4e2a\u5b57\u6bb5`DATE`\n2. \u5206\u7ec4\u51fd\u6570\u8f6c\u5165\u4e86\u4e00\u4e2a\u5b57\u6bb5`GROUP`\uff0c\u540c\u65f6\u9690\u85cf\u4e86\u4e00\u4e2a\u5b57\u6bb5`DATE`\n\n\u4e24\u79cd\u5206\u7c7b\u65b9\u6cd5\n\n1. \u6839\u636e\u7b97\u5b50\u524d\u7f00\u5206\u7c7b(`get_current_by_prefix`)\uff0c\u9650\u5236\u7b97\u5b50\u5fc5\u9700\u4ee5`ts_`\u3001`cs_`\u3001`gp_`\u5f00\u5934\n2. \u6839\u636e\u7b97\u5b50\u5168\u540d\u5206\u7c7b(`get_current_by_name`), \u4e0d\u518d\u9650\u5236\u7b97\u5b50\u540d\u3002\u6bd4\u5982`cs_rank`\u53ef\u4ee5\u53eb`rank`\n\n## Null\u5904\u7406\n\n`null`\u662f\u5982\u4f55\u4ea7\u751f\u7684\uff1f\n\n1. \u505c\u724c\u5bfc\u81f4\u3002\u5728\u8ba1\u7b97\u524d\u5c31\u76f4\u63a5\u8fc7\u6ee4\u6389\u4e86\uff0c\u4e0d\u4f1a\u5bf9\u540e\u7eed\u8ba1\u7b97\u4ea7\u751f\u5f71\u54cd\u3002\n2. \u4e0d\u540c\u54c1\u79cd\u4ea4\u6613\u65f6\u6bb5\u4e0d\u540c\n3. \u8ba1\u7b97\u4ea7\u751f\u3002`null`\u5728\u6570\u5217\u4e24\u7aef\u4e0d\u5f71\u54cd\u540e\u7eed\u65f6\u5e8f\u7b97\u5b50\u7ed3\u679c\uff0c\u4f46\u4e2d\u95f4\u51fa\u73b0`null`\u4f1a\u5f71\u54cd\u3002\u4f8b\u5982\uff1a `if_else(close<2, None, close)`\n\nhttps://github.com/pola-rs/polars/issues/12925#issuecomment-2552764629\n\n\u975e\u5e38\u68d2\u7684\u70b9\u5b50\uff0c\u603b\u7ed3\u4e0b\u6765\u6709\u4e24\u79cd\u5b9e\u73b0\u65b9\u5f0f\uff1a\n\n1. \u5c06`null`\u5206\u6210\u4e00\u7ec4\uff0c`not_null`\u5206\u6210\u53e6\u4e00\u7ec4\u3002\u8981\u8c03\u7528\u4e24\u6b21\n2. \u4ec5\u4e00\u7ec4\uff0c\u4f46\u590d\u5408\u6392\u5e8f\uff0c\u5c06`null`\u6392\u5728\u524d\u9762\uff0c`not_null`\u6392\u540e\u9762\u3002\u53ea\u8c03\u7528\u4e00\u6b21\uff0c\u7565\u5feb\u4e00\u4e9b\n\n```python\nX1 = (ts_returns(CLOSE, 3)).over(CLOSE.is_not_null(), _ASSET_, order_by=_DATE_),\nX2 = (ts_returns(CLOSE, 3)).over(_ASSET_, order_by=[CLOSE.is_not_null(), _DATE_]),\nX3 = (ts_returns(CLOSE, 3)).over(_ASSET_, order_by=_DATE_),\n```\n\n\u7b2c2\u79cd\u5f00\u5934\u7684`null`\u533a\u57df\uff0c\u662f\u5426\u5f71\u54cd\u7ed3\u679c\u7531\u7b97\u5b50\u6240\u51b3\u5b9a\uff0c\u7279\u522b\u65f6\u662f\u591a\u5217\u8f93\u5165\u65f6`null`\u533a\u57df\u53ef\u80fd\u6709\u6570\u636e\n\n1. `over_null='partition_by'`\u3002\u5206\u5230\u4e24\u4e2a\u533a\u57df\n2. `over_null='order_by'`\u3002\u5206\u5230\u4e00\u4e2a\u533a\u57df\uff0c`null`\u6392\u5728\u524d\u9762\n3. `over_null=None`\u3002\u4e0d\u5904\u7406\uff0c\u76f4\u63a5\u8c03\u7528\uff0c\u901f\u5ea6\u66f4\u5feb\u3002\u5982\u679c\u786e\u4fe1\u4e0d\u4f1a\u4e2d\u6bb5\u4ea7\u751f`null`\u5efa\u8bae\u4f7f\u7528\u6b64\u53c2\u6570\n\n## `expr_codegen`\u5c40\u9650\u6027\n\n1. `DAG`\u53ea\u80fd\u589e\u52a0\u5217\u65e0\u6cd5\u5220\u9664\u3002\u589e\u52a0\u5217\u65f6\uff0c\u9047\u5230\u540c\u540d\u5217\u4f1a\u8986\u76d6\n2. \u4e0d\u652f\u6301`\u5220\u9664\u884c`\uff0c\u4f46\u53ef\u4ee5\u6dfb\u52a0\u5220\u9664\u6807\u8bb0\u5217\uff0c\u7136\u540e\u5728\u5916\u8fdb\u884c\u5220\u9664\u884c\u3002\u5220\u9664\u884c\u5f71\u54cd\u4e86\u6240\u6709\u5217\uff0c\u4e0d\u6ee1\u8db3`DAG`\n3. \u4e0d\u652f\u6301`\u91cd\u91c7\u6837`\uff0c\u539f\u7406\u540c\u4e0d\u652f\u6301\u5220\u9664\u884c\u3002\u9700\u5728\u5916\u8fdb\u884c\n4. \u53ef\u4ee5\u5c06`\u5220\u9664\u884c`\u4e0e`\u91cd\u91c7\u6837`\u505a\u4e3a\u5206\u5272\u7ebf\uff0c\u4e00\u5927\u5757\u4ee3\u7801\u5206\u6210\u591a\u4e2a`DAG`\u4e32\u8054\u3002\u590d\u6742\u4e0d\u6613\u7406\u89e3\uff0c\u6240\u4ee5\u6700\u7ec8\u6ca1\u6709\u5b9e\u73b0\n\n## \u7279\u522b\u8bed\u6cd5\n\n1. \u652f\u6301`C?T:F`\u4e09\u5143\u8868\u8fbe\u5f0f\uff08\u4ec5\u53ef\u5b57\u7b26\u4e32\u4e2d\u4f7f\u7528\uff09\uff0c\u5e95\u5c42\u4f1a\u5148\u8f6c\u6210`C or True if( T )else F`\uff0c\u7136\u540e\u4fee\u6b63\u6210`T if C else F`\uff0c\u6700\u540e\u8f6c\u6210`if_else(C,T,F)`\u3002\u652f\u6301\u4e0e`if else`\u6df7\u7528\n2. `(A<B)*-1`,\u5e95\u5c42\u5c06\u8f6c\u6362\u6210`int_(A<B)*-1`\n3. \u4e3a\u9632\u6b62`A==B`\u88ab`sympy`\u66ff\u6362\u6210`False`\uff0c\u5e95\u5c42\u4f1a\u6362\u6210`Eq(A,B)`\n4. `A^B`\u7684\u542b\u4e49\u4e0e`convert_xor`\u53c2\u6570\u6709\u5173\uff0c`convert_xor=True`\u5e95\u5c42\u4f1a\u8f6c\u6362\u6210`Pow(A,B)`\uff0c\u53cd\u4e4b\u4e3a`Xor(A,B)`\u3002\u9ed8\u8ba4\u4e3a`False`\uff0c\u7528`**`\u8868\u793a\u4e58\u65b9\n5. \u652f\u6301`A&B&C`\uff0c\u4f46\u4e0d\u652f\u6301`A==B==C`\u3002\u5982\u679cC\u662f\u5e03\u5c14\uff0cAB\u662f\u6570\u503c\uff0c\u53ef\u624b\u5de5\u66ff\u6362\u6210`(A==B)==C`\u3002\u5982\u679cABC\u662f\u6570\u503c\u9700\u624b\u5de5\u66ff\u6362\u6210`(A==B)&(B==C)`\n6. \u4e0d\u652f\u6301`A<=B<=C`\uff0c\u9700\u624b\u5de5\u66ff\u6362\u6210`(A<=B)&(B<=C)`\n7. \u652f\u6301`A[0]+B[1]+C[2]`\uff0c\u5e95\u5c42\u4f1a\u8f6c\u6210`A+ts_delay(B,1)+ts_delay(C,2)`\n8. \u652f\u6301`~A`,\u5e95\u5c42\u4f1a\u8f6c\u6362\u6210`Not(A)`\n9. `gp_`\u5f00\u5934\u7684\u51fd\u6570\u90fd\u4f1a\u8fd4\u56de\u5bf9\u5e94\u7684`cs_`\u51fd\u6570\u3002\u5982`gp_func(A,B,C)`\u4f1a\u66ff\u6362\u6210`cs_func(B,C)`,\u5176\u4e2d`A`\u7528\u5728\u4e86`groupby([date, A])`\n10. \u652f\u6301`A,B,C=MACD()`\u5143\u7ec4\u89e3\u5305\uff0c\u5728\u5e95\u5c42\u4f1a\u66ff\u6362\u6210\n\n```python\n_x_0 = MACD()\nA = unpack(_x_0, 0)\nB = unpack(_x_0, 1)\nC = unpack(_x_0, 2)\n```\n\n## \u4e0b\u5212\u7ebf\u5f00\u5934\u7684\u53d8\u91cf\n\n1. \u8f93\u51fa\u7684\u6570\u636e\uff0c\u6240\u6709\u4ee5`_`\u5f00\u5934\u7684\u5217\uff0c\u6700\u540e\u4f1a\u88ab\u81ea\u52a8\u5220\u9664\u3002\u6240\u4ee5\u9700\u8981\u4fdd\u7559\u7684\u53d8\u91cf\u4e00\u5b9a\u4e0d\u8981\u4ee5`_`\u5f00\u5934\n2. \u4e3a\u51cf\u5c11\u91cd\u590d\u8ba1\u7b97\uff0c\u81ea\u52a8\u6dfb\u52a0\u4e86\u4e86\u4e2d\u95f4\u53d8\u91cf\uff0c\u4ee5`_x_`\u5f00\u5934\uff0c\u5982`_x_0`\uff0c`_x_1`\u7b49\u3002\u6700\u540e\u4f1a\u88ab\u81ea\u52a8\u5220\u9664\n3. \u5355\u884c\u8868\u8fbe\u5f0f\u8fc7\u957f\uff0c\u6216\u6709\u91cd\u590d\u8ba1\u7b97\uff0c\u53ef\u4ee5\u901a\u8fc7\u4e2d\u95f4\u53d8\u91cf\uff0c\u5c06\u5355\u884c\u8868\u8fbe\u5f0f\u6539\u6210\u591a\u884c\u3002\u5982\u679c\u4e2d\u95f4\u53d8\u91cf\u4f7f\u7528`_`\u5f00\u5934\uff0c\u5c06\u4f1a\u81ea\u52a8\u6dfb\u52a0\u6570\u5b57\u540e\u7f00\uff0c\u5f62\u6210\u4e0d\u540c\u7684\u53d8\u91cf\uff0c\u5982`_A`\u4f1a\u66ff\u6362\u6210`_A_0_`\u3001`_A_1_`\u7b49\u3002\u4f7f\u7528\u573a\u666f\u5982\u4e0b\uff1a\n    1. \u540c\u4e00\u53d8\u91cf\u540d\uff0c\u91cd\u590d\u4f7f\u7528\u3002\u672c\u8d28\u662f\u4e0d\u540c\u7684\u53d8\u91cf\n    2. \u5faa\u73af\u8d4b\u503c\uff0c\u4f46`DAG`\u4e0d\u652f\u6301\u6709\u73af\u3002`=`\u53f7\u5de6\u53f3\u7684\u540c\u540d\u53d8\u91cf\u5176\u5b9e\u662f\u4e0d\u540c\u53d8\u91cf\n\n## \u8f6c\u8bd1\u7ed3\u679c\u793a\u4f8b\n\n\u8f6c\u8bd1\u540e\u7684\u4ee3\u7801\u7247\u6bb5\uff0c\u8be6\u7ec6\u4ee3\u7801\u8bf7\u53c2\u8003[Polars\u7248](examples/output_polars.py)\n\n```python\ndef func_0_ts__asset(df: pl.DataFrame) -> pl.DataFrame:\n    df = df.sort(by=[_DATE_])\n    # ========================================\n    df = df.with_columns(\n        _x_0=1 / ts_delay(OPEN, -1),\n        LABEL_CC_1=(-CLOSE + ts_delay(CLOSE, -1)) / CLOSE,\n    )\n    # ========================================\n    df = df.with_columns(\n        LABEL_OO_1=_x_0 * ts_delay(OPEN, -2) - 1,\n        LABEL_OO_2=_x_0 * ts_delay(OPEN, -3) - 1,\n    )\n    return df\n```\n\n\u8f6c\u8bd1\u540e\u7684\u4ee3\u7801\u7247\u6bb5\uff0c\u8be6\u7ec6\u4ee3\u7801\u8bf7\u53c2\u8003[Pandas\u7248](examples/output_pandas.py)\n\n```python\ndef func_2_cs__date(df: pd.DataFrame) -> pd.DataFrame:\n    # expr_4 = cs_rank(x_7)\n    df[\"expr_4\"] = (df[\"x_7\"]).rank(pct=True)\n    return df\n\n\ndef func_3_ts__asset__date(df: pd.DataFrame) -> pd.DataFrame:\n    # expr_5 = -ts_corr(OPEN, CLOSE, 10)\n    df[\"expr_5\"] = -(df[\"OPEN\"]).rolling(10).corr(df[\"CLOSE\"])\n    # expr_6 = ts_delta(OPEN, 10)\n    df[\"expr_6\"] = df[\"OPEN\"].diff(10)\n    return df\n\n```\n\n## \u672c\u5730\u90e8\u7f72\u4ea4\u4e92\u7f51\u9875\n\n\u53ea\u9700\u8fd0\u884c`streamlit run streamlit_app.py`\n",
    "bugtrack_url": null,
    "license": "BSD 3-Clause License  Copyright (c) 2023, wukan  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ",
    "summary": "symbol expression to polars expression tool",
    "version": "0.10.9",
    "project_urls": null,
    "split_keywords": [
        "polars",
        " expression",
        " talib"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "837c1dcf25f29ebc3d9ffd5e04fbc5318e37cf6af8867f5ba9f59f936ef253d1",
                "md5": "4eae4482f6254319e1c9840203924a0a",
                "sha256": "60aef631cd95eed89f64fc7f497369ace9c60f9a74af9408bd6f43b60af5ec19"
            },
            "downloads": -1,
            "filename": "expr_codegen-0.10.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4eae4482f6254319e1c9840203924a0a",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.9",
            "size": 45072,
            "upload_time": "2024-12-30T06:22:55",
            "upload_time_iso_8601": "2024-12-30T06:22:55.301297Z",
            "url": "https://files.pythonhosted.org/packages/83/7c/1dcf25f29ebc3d9ffd5e04fbc5318e37cf6af8867f5ba9f59f936ef253d1/expr_codegen-0.10.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "638543dc1128984d7bdec4b2a623efeec2dc1982423e9fcce1c51eef66eb1616",
                "md5": "16dba76192c4fe7c25f3b6214ae19ff6",
                "sha256": "7c5c8a8ac4941bb83aa630f364e6f731ca612a419a88814e6c676b69f4670b01"
            },
            "downloads": -1,
            "filename": "expr_codegen-0.10.9.tar.gz",
            "has_sig": false,
            "md5_digest": "16dba76192c4fe7c25f3b6214ae19ff6",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.9",
            "size": 37595,
            "upload_time": "2024-12-30T06:22:57",
            "upload_time_iso_8601": "2024-12-30T06:22:57.604692Z",
            "url": "https://files.pythonhosted.org/packages/63/85/43dc1128984d7bdec4b2a623efeec2dc1982423e9fcce1c51eef66eb1616/expr_codegen-0.10.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-12-30 06:22:57",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "expr-codegen"
}
None
Elapsed time: 4.35395s