# Paralleltask
Paralleltask is a simple and lightweight parallel task engine. It can launch a given number of tasks from a batch of independent tasks, and keep this number of running tasks until all tasks are completed.
## Why Paralleltask?
Suppose you have dozens or hundreds of independent tasks that can run in parallel (non-independent tasks can be put together to form an independent task group). Due to the limitation of computing resources, you cannot run all tasks at the same time. Of course, it is not realistic to run one by one, so you want to run a specific number of tasks at the same time, and keep this number of running tasks (once a task is completed, start a new task) until all tasks are completed.
* zero configuration, no dependencies, no prior knowledge required, easy to install and use.
* support breakpoint resume, automatically re-execute failed tasks and ignore successful tasks.
* automatically kill submitted tasks once the main program receives a termination signal (`Ctrl+C`).
* support multiple task scheduling systems, such as `LOCAL`, `SGE`, `PBS`, `SLURM` and `LSF`.
* automatically convert relative path to absolute path in shell scripts.
* support python 2 and 3.
## Installation
```
pip install paralleltask
```
If you prefer to use the [drmaa](https://github.com/pygridtools/drmaa-python) library, instead of using commands (such as `qsub`) to submit and control tasks, see [here](https://github.com/moold/ParallelTask/blob/master/DRMAA.md) to install `drmaa`.
## Usage
```
paralleltask work.sh
```
The only required input file `work.sh` is a command collection file, which contains all the task commands that need to be run. A task command can be one or more lines, and an independent task group can be set by the parameter `--lines`. Using `paralleltask -h` for details.
## Testing
```bash
# produce a work file: test.sh
for i in {1..50};do echo "sleep ${i} && echo \"I am subtask${i}\"" >> test.sh;done;
# run
paralleltask test.sh
```
A demo file `test.sh` is [here](https://github.com/moold/ParallelTask/blob/master/test/test.sh)
## Configuration
If you want to change some of the default settings, you can pass parameters (use `paralleltask -h` for details) or directly edit the configure template file `cluster.cfg`.
***Note***: Paralleltask will replace `{mem}`, `{cpu}`, `{bash}`, `{out}`, `{err}`, `{script}` and `{job_id}` with specific values needed for each jobs, see the configure template [file](https://github.com/moold/ParallelTask/blob/master/src/paralleltask/cluster.cfg) for details.
## Getting Help
Feel free to raise an issue at the [issue page](https://github.com/moold/ParallelTask/issues), and welcome to [pull request](https://github.com/moold/ParallelTask/pulls) if you find a bug or have an idea for this project.
## Star
You can track updates by tab the `Star` button on the upper-right corner at the [github page](https://github.com/moold/ParallelTask).
Raw data
{
"_id": null,
"home_page": "https://github.com/moold/ParallelTask",
"name": "Paralleltask",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Hu Jiang",
"author_email": "mooldhu@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c1/14/7384ac9eac759d286e94f5456c82335dd268495294cbcc1a0e5d647ecffa/Paralleltask-0.2.3.tar.gz",
"platform": null,
"description": "# Paralleltask\nParalleltask is a simple and lightweight parallel task engine. It can launch a given number of tasks from a batch of independent tasks, and keep this number of running tasks until all tasks are completed.\n\n## Why Paralleltask?\nSuppose you have dozens or hundreds of independent tasks that can run in parallel (non-independent tasks can be put together to form an independent task group). Due to the limitation of computing resources, you cannot run all tasks at the same time. Of course, it is not realistic to run one by one, so you want to run a specific number of tasks at the same time, and keep this number of running tasks (once a task is completed, start a new task) until all tasks are completed.\n\n* zero configuration, no dependencies, no prior knowledge required, easy to install and use.\n\n* support breakpoint resume, automatically re-execute failed tasks and ignore successful tasks.\n\n* automatically kill submitted tasks once the main program receives a termination signal (`Ctrl+C`).\n\n* support multiple task scheduling systems, such as `LOCAL`, `SGE`, `PBS`, `SLURM` and `LSF`.\n\n* automatically convert relative path to absolute path in shell scripts.\n\n* support python 2 and 3.\n\n## Installation\n```\npip install paralleltask\n```\n\nIf you prefer to use the [drmaa](https://github.com/pygridtools/drmaa-python) library, instead of using commands (such as `qsub`) to submit and control tasks, see [here](https://github.com/moold/ParallelTask/blob/master/DRMAA.md) to install `drmaa`.\n\n## Usage\n```\nparalleltask work.sh\n```\n\nThe only required input file `work.sh` is a command collection file, which contains all the task commands that need to be run. A task command can be one or more lines, and an independent task group can be set by the parameter `--lines`. Using `paralleltask -h` for details.\n\n## Testing\n```bash\n# produce a work file: test.sh\nfor i in {1..50};do echo \"sleep ${i} && echo \\\"I am subtask${i}\\\"\" >> test.sh;done;\n# run\nparalleltask test.sh\n```\nA demo file `test.sh` is [here](https://github.com/moold/ParallelTask/blob/master/test/test.sh)\n\n\n## Configuration \nIf you want to change some of the default settings, you can pass parameters (use `paralleltask -h` for details) or directly edit the configure template file `cluster.cfg`.\n\n***Note***: Paralleltask will replace `{mem}`, `{cpu}`, `{bash}`, `{out}`, `{err}`, `{script}` and `{job_id}` with specific values needed for each jobs, see the configure template [file](https://github.com/moold/ParallelTask/blob/master/src/paralleltask/cluster.cfg) for details.\n\n## Getting Help\n\nFeel free to raise an issue at the [issue page](https://github.com/moold/ParallelTask/issues), and welcome to [pull request](https://github.com/moold/ParallelTask/pulls) if you find a bug or have an idea for this project.\n\n## Star\nYou can track updates by tab the `Star` button on the upper-right corner at the [github page](https://github.com/moold/ParallelTask).\n\n\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "A simple and lightweight parallel task engine",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/moold/ParallelTask"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2f7bf2c87acaba0aeac327fdd89d5a289f027bafe254a7c5320b0112026c7318",
"md5": "daf0036850eedcda96ae1ee3a19d70dc",
"sha256": "1f6b2945d7fd34cfdef16d3481f6bd6b64c68d3ea978cb5fb6fc292397c49e53"
},
"downloads": -1,
"filename": "Paralleltask-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "daf0036850eedcda96ae1ee3a19d70dc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 24209,
"upload_time": "2024-08-12T04:41:30",
"upload_time_iso_8601": "2024-08-12T04:41:30.378953Z",
"url": "https://files.pythonhosted.org/packages/2f/7b/f2c87acaba0aeac327fdd89d5a289f027bafe254a7c5320b0112026c7318/Paralleltask-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c1147384ac9eac759d286e94f5456c82335dd268495294cbcc1a0e5d647ecffa",
"md5": "a3fe3d8cb3d9ce5eec118be4018a6667",
"sha256": "8015a8311d5021bc44edbfbf45ff2557a529999e235d25190bac62993fdf7b66"
},
"downloads": -1,
"filename": "Paralleltask-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "a3fe3d8cb3d9ce5eec118be4018a6667",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 22364,
"upload_time": "2024-08-12T04:41:32",
"upload_time_iso_8601": "2024-08-12T04:41:32.160299Z",
"url": "https://files.pythonhosted.org/packages/c1/14/7384ac9eac759d286e94f5456c82335dd268495294cbcc1a0e5d647ecffa/Paralleltask-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-12 04:41:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "moold",
"github_project": "ParallelTask",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "paralleltask"
}
JSON
