# Manytasks
A lightweight tool for deploying many tasks automatically, without any modification to your code.
- [Manytasks](#manytasks)
- [Installation](#installation)
- [Quick Example](#quick-example)
- [Sample Configuration](#sample-configuration)
- [Syntax Sugar](#syntax-sugar)
- [Design Philosophy](#design-philosophy)
- [Analysis](#analysis)
- [History](#history)
## Installation
I **recommend** you install from github to get the newest features:
`pip install git+https://github.com/dugu9sword/manytasks.git`
You can also install the package (**maybe outdated**) from pypi:
`pip install manytasks [not recommended]`
## Quick Example
```
cd examples/python
# configuration is stored in tasks.json
manytasks run tasks
```
All running logs are stored in `tasks.logs`.
- The running log of the manytasks is written into `task.logs/status.txt`
- The IO stream of tasks (e.g. `print()`, `Exception`, `Error`, ...) are redirected to `task.logs/task-[index].txt`.
## Sample Configuration
For more complex cases, see `ADVANCED_CASES.md`.
```python
{
"executor": "python some.py",
"cuda": [4, 5, 6, 7],
"concurrency": 4, # num of tasks to run in parallel
"cuda_per_task": 1,
"configs": {
# basic configurations
"==base==": [
"arg0",
"--a", [50, 100], # `--a` takes value from [50, 100]
"--name", "a_${a}" # "${a}" refers to the value of `--a`
],
# more disjoint configurations
"==more==": [
[ "--c1", [1, 2] ], # [1, 2]
[ "--c2", "$<x|y>.$<1:4>" ], # [x.1, x.2, x.3, y.1, y.2, y.3]
]
}
}
```
which yields:
```bash
--- ---- --- ------ ---- ----
idx __1 --a --name --c1 --c2
0 arg0 50 a_50 1 -
1 arg0 50 a_50 2 -
2 arg0 100 a_100 1 -
3 arg0 100 a_100 2 -
4 arg0 50 a_50 - x.1
5 arg0 50 a_50 - x.2
6 arg0 50 a_50 - x.3
7 arg0 50 a_50 - y.1
8 arg0 50 a_50 - y.2
9 arg0 50 a_50 - y.3
10 arg0 100 a_100 - x.1
11 arg0 100 a_100 - x.2
12 arg0 100 a_100 - x.3
13 arg0 100 a_100 - y.1
14 arg0 100 a_100 - y.2
15 arg0 100 a_100 - y.3
--- ---- --- ------ ---- ----
```
### Syntax Sugar
The syntax sugar makes the enumeration of arguments more easier.
| Type | Example Input | Example Output |
| ------------------------ | ---------------------------- | ------------------------------------------ |
| **list** | `$<1:6>` | `[1, 2, 3, 4, 5]` |
| **list** (with step) | `$<1:6:2>` | `[1, 3, 5]` |
| **list** (with zero-pad) | `$<1:6:2;3>` | `[001, 003, 005]` |
| **files** | `$<files:/home/*.py>` | `[/home/foo.py, /home/bar.py, ...]` |
| **files** (without path) | `$<files:/home/*.py;nameonly>` | `[foo.py, bar.py, ...]` |
| **lines** | `$<lines:urls.txt>` | `[baidu.com, google.com, ...]` |
| **set** | `$<a\|b\|c>` | `[a, b, c]` |
| **composition** | `x_$<1:3;3>.txt` | `[x_001.txt, x_002.txt]` |
| **composition** (more) | `logs/$<a\|b>.$<1:3>` | `[logs/a.1, logs/a.2, logs/b.1, logs/b.2]` |
## Design Philosophy
**Q**: Why not use other open-source tools for hyper-parameter tuning, such as `optuna`, `hydra`, `wandb`?
**A**: The first time I developed this tool is 2019-1-1, when most of those tools (except `optuna`) were **not born** yet. What's more,
- These tools are **heavy**, which means:
- You need to modify your code (add several lines of `import xxx; xxx.foo(); xxx.bar();`) to integrate their functionalities.
- You may have to replace your `print(...)` or `log(...)` function with theirs, sometimes your results may be logged to their server.
- These tools are highly binded with `python`, which means they may fail if your code is written in `perl` (such as `mosetokenizer` in machine translation), `bash` (such as your code for preprocessing), etc.
- `manytasks` is a lightweight tool which helps you deploy many tasks **without any modification** to your code, all logs will be saved in your machine.
**Q**:When should you use other open-source tools for hyper-parameter tuning, such as `optuna`, `hydra`, `wandb`?
**A**: For complex cases when you would like to enjoy their power of hyper-parameter tuning.
## Analysis
See `Analysis.md`.
## History
See `History.md`.
Raw data
{
"_id": null,
"home_page": "https://github.com/dugu9sword/manytasks",
"name": "manytasks",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "manytasks",
"author": "dugu9sword",
"author_email": "dugu9sword@163.com",
"download_url": "https://files.pythonhosted.org/packages/76/f4/68b02de96feb56d5ca6187c605d64078d7bf76a4cf1190e2d775c550fc84/manytasks-2.3.tar.gz",
"platform": "any",
"description": "# Manytasks\n\nA lightweight tool for deploying many tasks automatically, without any modification to your code.\n\n- [Manytasks](#manytasks)\n - [Installation](#installation)\n - [Quick Example](#quick-example)\n - [Sample Configuration](#sample-configuration)\n - [Syntax Sugar](#syntax-sugar)\n - [Design Philosophy](#design-philosophy)\n - [Analysis](#analysis)\n - [History](#history)\n\n\n## Installation\n\nI **recommend** you install from github to get the newest features:\n\n`pip install git+https://github.com/dugu9sword/manytasks.git` \n\nYou can also install the package (**maybe outdated**) from pypi:\n\n`pip install manytasks [not recommended]` \n\n\n## Quick Example\n\n```\ncd examples/python\n\n# configuration is stored in tasks.json\nmanytasks run tasks\n```\n\nAll running logs are stored in `tasks.logs`. \n\n- The running log of the manytasks is written into `task.logs/status.txt` \n\n- The IO stream of tasks (e.g. `print()`, `Exception`, `Error`, ...) are redirected to `task.logs/task-[index].txt`.\n\n\n## Sample Configuration\n\nFor more complex cases, see `ADVANCED_CASES.md`.\n\n```python\n{\n \"executor\": \"python some.py\", \n \"cuda\": [4, 5, 6, 7],\n \"concurrency\": 4, # num of tasks to run in parallel\n \"cuda_per_task\": 1,\n \"configs\": {\n # basic configurations\n \"==base==\": [ \n \"arg0\",\n \"--a\", [50, 100], # `--a` takes value from [50, 100]\n \"--name\", \"a_${a}\" # \"${a}\" refers to the value of `--a`\n ],\n # more disjoint configurations\n \"==more==\": [\n [ \"--c1\", [1, 2] ], # [1, 2]\n [ \"--c2\", \"$<x|y>.$<1:4>\" ], # [x.1, x.2, x.3, y.1, y.2, y.3]\n ]\n }\n}\n```\n\nwhich yields:\n```bash\n--- ---- --- ------ ---- ----\nidx __1 --a --name --c1 --c2\n0 arg0 50 a_50 1 -\n1 arg0 50 a_50 2 -\n2 arg0 100 a_100 1 -\n3 arg0 100 a_100 2 -\n4 arg0 50 a_50 - x.1\n5 arg0 50 a_50 - x.2\n6 arg0 50 a_50 - x.3\n7 arg0 50 a_50 - y.1\n8 arg0 50 a_50 - y.2\n9 arg0 50 a_50 - y.3\n10 arg0 100 a_100 - x.1\n11 arg0 100 a_100 - x.2\n12 arg0 100 a_100 - x.3\n13 arg0 100 a_100 - y.1\n14 arg0 100 a_100 - y.2\n15 arg0 100 a_100 - y.3\n--- ---- --- ------ ---- ----\n```\n\n### Syntax Sugar\n\nThe syntax sugar makes the enumeration of arguments more easier.\n\n| Type | Example Input | Example Output |\n| ------------------------ | ---------------------------- | ------------------------------------------ |\n| **list** | `$<1:6>` | `[1, 2, 3, 4, 5]` |\n| **list** (with step) | `$<1:6:2>` | `[1, 3, 5]` |\n| **list** (with zero-pad) | `$<1:6:2;3>` | `[001, 003, 005]` |\n| **files** | `$<files:/home/*.py>` | `[/home/foo.py, /home/bar.py, ...]` |\n| **files** (without path) | `$<files:/home/*.py;nameonly>` | `[foo.py, bar.py, ...]` |\n| **lines** | `$<lines:urls.txt>` | `[baidu.com, google.com, ...]` |\n| **set** | `$<a\\|b\\|c>` | `[a, b, c]` |\n| **composition** | `x_$<1:3;3>.txt` | `[x_001.txt, x_002.txt]` |\n| **composition** (more) | `logs/$<a\\|b>.$<1:3>` | `[logs/a.1, logs/a.2, logs/b.1, logs/b.2]` |\n\n\n\n\n\n## Design Philosophy\n\n**Q**: Why not use other open-source tools for hyper-parameter tuning, such as `optuna`, `hydra`, `wandb`?\n\n**A**: The first time I developed this tool is 2019-1-1, when most of those tools (except `optuna`) were **not born** yet. What's more, \n\n- These tools are **heavy**, which means:\n - You need to modify your code (add several lines of `import xxx; xxx.foo(); xxx.bar();`) to integrate their functionalities.\n - You may have to replace your `print(...)` or `log(...)` function with theirs, sometimes your results may be logged to their server. \n - These tools are highly binded with `python`, which means they may fail if your code is written in `perl` (such as `mosetokenizer` in machine translation), `bash` (such as your code for preprocessing), etc.\n- `manytasks` is a lightweight tool which helps you deploy many tasks **without any modification** to your code, all logs will be saved in your machine.\n\n**Q**:When should you use other open-source tools for hyper-parameter tuning, such as `optuna`, `hydra`, `wandb`?\n\n**A**: For complex cases when you would like to enjoy their power of hyper-parameter tuning.\n\n## Analysis\n\nSee `Analysis.md`.\n\n## History\n\nSee `History.md`.\n\n\n",
"bugtrack_url": null,
"license": "WTFPL Licence",
"summary": "A tool for deploying many tasks automatically.",
"version": "2.3",
"project_urls": {
"Homepage": "https://github.com/dugu9sword/manytasks"
},
"split_keywords": [
"manytasks"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "057b04da896edf72231006ec1ebac2cbd0e3612c97f8b899ccee37780cf4e5d9",
"md5": "070139695576c45d478cfb7dc1f721b1",
"sha256": "119aff9735636b9728bd66bdd8f50d1b7679f709f0d615ecf913944b780cf352"
},
"downloads": -1,
"filename": "manytasks-2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "070139695576c45d478cfb7dc1f721b1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 15521,
"upload_time": "2024-09-09T12:18:22",
"upload_time_iso_8601": "2024-09-09T12:18:22.056136Z",
"url": "https://files.pythonhosted.org/packages/05/7b/04da896edf72231006ec1ebac2cbd0e3612c97f8b899ccee37780cf4e5d9/manytasks-2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "76f468b02de96feb56d5ca6187c605d64078d7bf76a4cf1190e2d775c550fc84",
"md5": "aada5de0ae7e430bac13681729cdafca",
"sha256": "bd4f2fb03e8f682efb79b1118d92f557e704d70b14554d788f1a4ff91b255b55"
},
"downloads": -1,
"filename": "manytasks-2.3.tar.gz",
"has_sig": false,
"md5_digest": "aada5de0ae7e430bac13681729cdafca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 15238,
"upload_time": "2024-09-09T12:18:25",
"upload_time_iso_8601": "2024-09-09T12:18:25.309490Z",
"url": "https://files.pythonhosted.org/packages/76/f4/68b02de96feb56d5ca6187c605d64078d7bf76a4cf1190e2d775c550fc84/manytasks-2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-09 12:18:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dugu9sword",
"github_project": "manytasks",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "jstyleson",
"specs": []
},
{
"name": "tabulate",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "psutil",
"specs": []
},
{
"name": "pyyaml",
"specs": []
}
],
"lcname": "manytasks"
}
JSON
