本仓库旨在方便快速地为深度学习项目提供便捷功能，例如钉钉通知、快捷配置、批量实验、GPU排队等等
# 安装
```bash
pip install lazydl -i https://pypi.org/simple
```
# 使用
```python
import lazydl as l
l.hi()
```
# 功能支持
### 帮助信息
- hi()：打印 Banner
- help()：打印帮助信息，待完善
- print_error_info(e: Exception)：打印错误信息
### 通知
- notice(msg, warning=False, access_token="", secret="")：钉钉通知，默认读取 init_env() 设置的环境变量中的凭证
```
Args:
msg (str): 通知内容
warning (bool, optional): 是否为警告. Defaults to False.
access_token (str, optional): 钉钉机器人 access_token. Defaults to "".
secret (str, optional): 钉钉机器人 secret. Defaults to "".
```
### 配置
- init_env(config: Union[DictConfig, dict])：初始化环境
```
包括随机种子、可见GPU、Comet.ml环境变量、进程名
Args:
config (DictConfig): 配置文件
seed (str, optional): 种子. Defaults to '3407'.
use_deterministic (bool, optional): 是否使用确定性算法，使用了训练会变慢. Defaults to False.
visibale_cuda (str, optional): 设置可见 GPU，如果不使用设置为 ''. Defaults to 'all'.
comet_exp (dict, optional): Comet.ml相关环境变量设置. Defaults to {}.
proctitle (str, optional): 进程名. Defaults to 'python'.
proctitle_prefix_id (bool, optional): 是否在进程名前边添加进程 ID. Defaults to True.
```
### GPU 自动选择、排队
- gpu_ready())：检查 GPU 是否可用
- set_processing_units(config: Union[DictConfig, dict] = {})：设置处理器
```
支持自动选择和手动选择、排队和不排队
Args:
config (dict, optional): Defaults to {}.
processing_unit: 处理器类型，可选值为：cpu、mps, 1, ^2 等，其中 ^2 表示自动选择两块 GPU，默认为 cpu
processing_unit_type: 选择处理器，可选值为：cpu、mps、gpu-a-s、gpu-a-m、gpu-m-s、gpu-m-m
processing_unit_min_free_memory: 最小空闲内存，单位为 GiB，默认为 10
processing_unit_min_free_memory_ratio: 最小空闲内存比例，默认为 0.5
queuing: 是否排队，依赖 Redis，默认为 False
visible_devices: 可见的 GPU 序号，用逗号分隔，默认为 None
Returns:
Union[DictConfig, dict]: config
```
### 包装类
- Result：包装类，用于作为模块间通信的媒介，可使用字典进行初始化
### 文件读取和保存
- load_in(path, data_name="")：读取文件
```
根据文件后缀名自动选择读取方法
目前支持保存类型有：‘pkl’、‘txt’、‘pt’、‘json’, 'jsonl'、'csv'
Args:
data_name: str, 打印提示时需要，便于控制台查看保存的文件是什么文件, 默认为空
Returns:
data：Object
```
- save_as(data, save_path, file_format="pt", data_name="", protocol=4)：保存文件
```
将参数中的文件对象保存为指定格式格式文件
目前支持保存类型有：‘pkl’、‘txt’、‘pt’、‘json’, 'jsonl'
默认为‘pt’
Args:
data: obj, 要保存的文件对象
save_path: str, 文件的保存路径，应当包含文件名和后缀名
data_name: str, 打印提示时需要，便于控制台查看保存的文件是什么文件, 默认为空
protocol: int, 当文件特别大的时候，需要将此参数调到4以上, 默认为4
file_format: str, 要保存的文件类型，支持‘pkl’、‘txt’、‘pt’、‘json’、‘jsonl’
Returns:
None
```
Raw data
{
"_id": null,
"home_page": "https://github.com/deng1fan",
"name": "lazydl",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "lazydl,deep learning,pytorch",
"author": "deng1fan",
"author_email": "dengyifan@iie.ac.cn",
"download_url": "https://files.pythonhosted.org/packages/4a/80/0d145265feda623fa10dac2aaebfcc4cfc21937412678fbdaea787134f99/lazydl-0.2.83.tar.gz",
"platform": null,
"description": "\u672c\u4ed3\u5e93\u65e8\u5728\u65b9\u4fbf\u5feb\u901f\u5730\u4e3a\u6df1\u5ea6\u5b66\u4e60\u9879\u76ee\u63d0\u4f9b\u4fbf\u6377\u529f\u80fd\uff0c\u4f8b\u5982\u9489\u9489\u901a\u77e5\u3001\u5feb\u6377\u914d\u7f6e\u3001\u6279\u91cf\u5b9e\u9a8c\u3001GPU\u6392\u961f\u7b49\u7b49\n\n# \u5b89\u88c5\n```bash\npip install lazydl -i https://pypi.org/simple \n```\n\n\n# \u4f7f\u7528\n```python\nimport lazydl as l\n\nl.hi()\n```\n\n\n# \u529f\u80fd\u652f\u6301\n\n### \u5e2e\u52a9\u4fe1\u606f\n- hi()\uff1a\u6253\u5370 Banner\n- help()\uff1a\u6253\u5370\u5e2e\u52a9\u4fe1\u606f\uff0c\u5f85\u5b8c\u5584\n- print_error_info(e: Exception)\uff1a\u6253\u5370\u9519\u8bef\u4fe1\u606f\n\n\n### \u901a\u77e5\n- notice(msg, warning=False, access_token=\"\", secret=\"\")\uff1a\u9489\u9489\u901a\u77e5\uff0c\u9ed8\u8ba4\u8bfb\u53d6 init_env() \u8bbe\u7f6e\u7684\u73af\u5883\u53d8\u91cf\u4e2d\u7684\u51ed\u8bc1\n ```\n Args:\n msg (str): \u901a\u77e5\u5185\u5bb9\n warning (bool, optional): \u662f\u5426\u4e3a\u8b66\u544a. Defaults to False.\n access_token (str, optional): \u9489\u9489\u673a\u5668\u4eba access_token. Defaults to \"\".\n secret (str, optional): \u9489\u9489\u673a\u5668\u4eba secret. Defaults to \"\".\n ```\n\n### \u914d\u7f6e\n- init_env(config: Union[DictConfig, dict])\uff1a\u521d\u59cb\u5316\u73af\u5883\n ```\n \u5305\u62ec\u968f\u673a\u79cd\u5b50\u3001\u53ef\u89c1GPU\u3001Comet.ml\u73af\u5883\u53d8\u91cf\u3001\u8fdb\u7a0b\u540d\n\n Args:\n config (DictConfig): \u914d\u7f6e\u6587\u4ef6\n seed (str, optional): \u79cd\u5b50. Defaults to '3407'.\n use_deterministic (bool, optional): \u662f\u5426\u4f7f\u7528\u786e\u5b9a\u6027\u7b97\u6cd5\uff0c\u4f7f\u7528\u4e86\u8bad\u7ec3\u4f1a\u53d8\u6162. Defaults to False.\n visibale_cuda (str, optional): \u8bbe\u7f6e\u53ef\u89c1 GPU\uff0c\u5982\u679c\u4e0d\u4f7f\u7528\u8bbe\u7f6e\u4e3a ''. Defaults to 'all'.\n comet_exp (dict, optional): Comet.ml\u76f8\u5173\u73af\u5883\u53d8\u91cf\u8bbe\u7f6e. Defaults to {}.\n proctitle (str, optional): \u8fdb\u7a0b\u540d. Defaults to 'python'.\n proctitle_prefix_id (bool, optional): \u662f\u5426\u5728\u8fdb\u7a0b\u540d\u524d\u8fb9\u6dfb\u52a0\u8fdb\u7a0b ID. Defaults to True.\n ```\n\n### GPU \u81ea\u52a8\u9009\u62e9\u3001\u6392\u961f\n- gpu_ready())\uff1a\u68c0\u67e5 GPU \u662f\u5426\u53ef\u7528\n- set_processing_units(config: Union[DictConfig, dict] = {})\uff1a\u8bbe\u7f6e\u5904\u7406\u5668\n ```\n \u652f\u6301\u81ea\u52a8\u9009\u62e9\u548c\u624b\u52a8\u9009\u62e9\u3001\u6392\u961f\u548c\u4e0d\u6392\u961f\n\n Args:\n config (dict, optional): Defaults to {}.\n processing_unit: \u5904\u7406\u5668\u7c7b\u578b\uff0c\u53ef\u9009\u503c\u4e3a\uff1acpu\u3001mps, 1, ^2 \u7b49\uff0c\u5176\u4e2d ^2 \u8868\u793a\u81ea\u52a8\u9009\u62e9\u4e24\u5757 GPU\uff0c\u9ed8\u8ba4\u4e3a cpu\n processing_unit_type: \u9009\u62e9\u5904\u7406\u5668\uff0c\u53ef\u9009\u503c\u4e3a\uff1acpu\u3001mps\u3001gpu-a-s\u3001gpu-a-m\u3001gpu-m-s\u3001gpu-m-m\n processing_unit_min_free_memory: \u6700\u5c0f\u7a7a\u95f2\u5185\u5b58\uff0c\u5355\u4f4d\u4e3a GiB\uff0c\u9ed8\u8ba4\u4e3a 10\n processing_unit_min_free_memory_ratio: \u6700\u5c0f\u7a7a\u95f2\u5185\u5b58\u6bd4\u4f8b\uff0c\u9ed8\u8ba4\u4e3a 0.5\n queuing: \u662f\u5426\u6392\u961f\uff0c\u4f9d\u8d56 Redis\uff0c\u9ed8\u8ba4\u4e3a False\n visible_devices: \u53ef\u89c1\u7684 GPU \u5e8f\u53f7\uff0c\u7528\u9017\u53f7\u5206\u9694\uff0c\u9ed8\u8ba4\u4e3a None\n\n Returns:\n Union[DictConfig, dict]: config\n ```\n\n### \u5305\u88c5\u7c7b\n- Result\uff1a\u5305\u88c5\u7c7b\uff0c\u7528\u4e8e\u4f5c\u4e3a\u6a21\u5757\u95f4\u901a\u4fe1\u7684\u5a92\u4ecb\uff0c\u53ef\u4f7f\u7528\u5b57\u5178\u8fdb\u884c\u521d\u59cb\u5316\n\n\n### \u6587\u4ef6\u8bfb\u53d6\u548c\u4fdd\u5b58\n- load_in(path, data_name=\"\")\uff1a\u8bfb\u53d6\u6587\u4ef6\n ```\n \u6839\u636e\u6587\u4ef6\u540e\u7f00\u540d\u81ea\u52a8\u9009\u62e9\u8bfb\u53d6\u65b9\u6cd5\n \u76ee\u524d\u652f\u6301\u4fdd\u5b58\u7c7b\u578b\u6709\uff1a\u2018pkl\u2019\u3001\u2018txt\u2019\u3001\u2018pt\u2019\u3001\u2018json\u2019, 'jsonl'\u3001'csv'\n\n Args:\n data_name: str, \u6253\u5370\u63d0\u793a\u65f6\u9700\u8981\uff0c\u4fbf\u4e8e\u63a7\u5236\u53f0\u67e5\u770b\u4fdd\u5b58\u7684\u6587\u4ef6\u662f\u4ec0\u4e48\u6587\u4ef6, \u9ed8\u8ba4\u4e3a\u7a7a\n\n Returns:\n data\uff1aObject\n ```\n\n- save_as(data, save_path, file_format=\"pt\", data_name=\"\", protocol=4)\uff1a\u4fdd\u5b58\u6587\u4ef6\n ```\n \u5c06\u53c2\u6570\u4e2d\u7684\u6587\u4ef6\u5bf9\u8c61\u4fdd\u5b58\u4e3a\u6307\u5b9a\u683c\u5f0f\u683c\u5f0f\u6587\u4ef6\n \u76ee\u524d\u652f\u6301\u4fdd\u5b58\u7c7b\u578b\u6709\uff1a\u2018pkl\u2019\u3001\u2018txt\u2019\u3001\u2018pt\u2019\u3001\u2018json\u2019, 'jsonl'\n \u9ed8\u8ba4\u4e3a\u2018pt\u2019\n\n Args:\n data: obj, \u8981\u4fdd\u5b58\u7684\u6587\u4ef6\u5bf9\u8c61\n save_path: str, \u6587\u4ef6\u7684\u4fdd\u5b58\u8def\u5f84\uff0c\u5e94\u5f53\u5305\u542b\u6587\u4ef6\u540d\u548c\u540e\u7f00\u540d\n data_name: str, \u6253\u5370\u63d0\u793a\u65f6\u9700\u8981\uff0c\u4fbf\u4e8e\u63a7\u5236\u53f0\u67e5\u770b\u4fdd\u5b58\u7684\u6587\u4ef6\u662f\u4ec0\u4e48\u6587\u4ef6, \u9ed8\u8ba4\u4e3a\u7a7a\n protocol: int, \u5f53\u6587\u4ef6\u7279\u522b\u5927\u7684\u65f6\u5019\uff0c\u9700\u8981\u5c06\u6b64\u53c2\u6570\u8c03\u52304\u4ee5\u4e0a, \u9ed8\u8ba4\u4e3a4\n file_format: str, \u8981\u4fdd\u5b58\u7684\u6587\u4ef6\u7c7b\u578b\uff0c\u652f\u6301\u2018pkl\u2019\u3001\u2018txt\u2019\u3001\u2018pt\u2019\u3001\u2018json\u2019\u3001\u2018jsonl\u2019\n\n Returns:\n None\n ```\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Deep learning tools",
"version": "0.2.83",
"project_urls": {
"Homepage": "https://github.com/deng1fan"
},
"split_keywords": [
"lazydl",
"deep learning",
"pytorch"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4a800d145265feda623fa10dac2aaebfcc4cfc21937412678fbdaea787134f99",
"md5": "c1a71eddf6107b8fc16883fa7cb56990",
"sha256": "54db82e06d80548aa2262aa0348cf215566528c386978f5e09451d934ffcb4ab"
},
"downloads": -1,
"filename": "lazydl-0.2.83.tar.gz",
"has_sig": false,
"md5_digest": "c1a71eddf6107b8fc16883fa7cb56990",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 31706,
"upload_time": "2024-02-15T10:25:31",
"upload_time_iso_8601": "2024-02-15T10:25:31.582287Z",
"url": "https://files.pythonhosted.org/packages/4a/80/0d145265feda623fa10dac2aaebfcc4cfc21937412678fbdaea787134f99/lazydl-0.2.83.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-15 10:25:31",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "lazydl"
}
JSON