# batch_queue2
**batch_queue2** is a simple task queuing system built using Python's
`asyncio`. It allows you to submit, manage, and track tasks using a
lightweight, XML-RPC-based interface. It's main purpose is to allow
queueing up more tasks to run on the local machine than we can support
running simultaneously; either because the number of tasks is more
than the number of CPUs or because running all those tasks
simultaneously will use too much memory.
## Features
- Submit, list, suspend, resume, and kill tasks using command-line commands.
- Tasks can be queued, paused, or resumed based on available CPUs.
- XML-RPC server for managing task requests.
- Ability to run tasks using multiple CPUs.
- Tasks can be paused and resumed.
## Requirements
- Python 3.7+
- `aiohttp`
- `requests`
## Installation
You can install **batch_queue2** from PyPI:
```sh
pip install batch_queue2
```
Or clone the repository from [Sourcehut](https://hg.sr.ht/~ndbecker2/batch_queue2/):
```sh
hg clone https://hg.sr.ht/~ndbecker2/batch_queue2
cd batch_queue2
python -m pip install .
```
## Usage
After installing, you can use the `batch_queue` command to manage tasks. Below are the available options:
### Starting the Server
To start the server:
```sh
batch_queue start --max-cpus 4
```
- `--max-cpus`: (Optional) Specify the maximum number of CPUs to use. Defaults to the number of CPUs available on your system.
The server will start in daemon mode by default.
### Submitting a Task
To submit a task, use the `submit` command:
```sh
batch_queue submit <command>
```
For example:
```sh
batch_queue submit sleep 10
```
You can also optionally specify:
- `--log-stdout <file>`: Redirect the standard output of the task to a file.
- `--log-stderr <file>`: Redirect the standard error of the task to a file.
### Listing Tasks
To list all tasks:
```sh
batch_queue list
```
This will display:
- Max CPUs available.
- Active tasks.
- Queued tasks.
- Paused tasks.
- Runnable paused tasks.
### Suspending and Resuming Tasks
To suspend task(s) (running or queued):
```sh
batch_queue suspend <task_ids>
```
To resume a paused task(s):
```sh
batch_queue resume <task_ids>
```
### Killing Task(s)
To kill specific task(s):
```sh
batch_queue kill <task_ids> <signal>
```
You can also optionally specify the signal to use, default is `SIGTERM`.
### Getting Task Information
To get detailed information about a specific task:
```sh
batch_queue id <task_id>
```
This command provides detailed information about the task including command, user, working directory, environment variables, and logs.
### Change #cpus
You can reset the max #cpus. When you do this, if the new maximum is
greater than the number of active tasks the scheduler is recalled and
queued tasks can be started. If the new maximum is smaller than the
number of active tasks tasks will be suspended, but marked runnable to
be continued later.
```sh
batch_queue setcpus #cpus
```
### Stopping the Server
To stop the server:
```sh
batch_queue stop
```
This command gracefully stops the server, ensuring no tasks are left in a zombie state.
## Example Workflow
1. Start the server using:
```sh
batch_queue start --max-cpus 4
```
2. Submit a couple of tasks:
```sh
batch_queue submit sleep 10
batch_queue submit echo "Hello World"
```
3. List the tasks to see the active, queued, and paused tasks:
```sh
batch_queue list
```
4. Suspend running tasks:
```sh
batch_queue suspend 0
```
5. Resume paused tasks:
```sh
batch_queue resume 0
```
6. Stop the server:
```sh
batch_queue stop
```
## Logging
The server logs all activity to `~/batch_queue.log`. You can view the log to monitor task submissions, task status changes, server starts and stops, etc.
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
## Security
The server listens on a unix domain socket normally in the user's home
directory who starts it. Only users with access to this socket can
use it.
Raw data
{
"_id": null,
"home_page": "https://sr.ht/~ndbecker2/batch_queue2/",
"name": "batch-queue2",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "queue, asyncio, batch",
"author": "Neal Becker",
"author_email": "Neal Becker <ndbecker2@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/ff/78/5d29fa91695167cfbd42b186da11a1bc68572c1080313f6f9ebd1b5af4d6/batch_queue2-1.0.12.tar.gz",
"platform": null,
"description": "# batch_queue2\n\n**batch_queue2** is a simple task queuing system built using Python's\n`asyncio`. It allows you to submit, manage, and track tasks using a\nlightweight, XML-RPC-based interface. It's main purpose is to allow\nqueueing up more tasks to run on the local machine than we can support\nrunning simultaneously; either because the number of tasks is more\nthan the number of CPUs or because running all those tasks\nsimultaneously will use too much memory.\n\n## Features\n- Submit, list, suspend, resume, and kill tasks using command-line commands.\n- Tasks can be queued, paused, or resumed based on available CPUs.\n- XML-RPC server for managing task requests.\n- Ability to run tasks using multiple CPUs.\n- Tasks can be paused and resumed.\n\n## Requirements\n- Python 3.7+\n- `aiohttp`\n- `requests`\n\n## Installation\nYou can install **batch_queue2** from PyPI:\n\n```sh\npip install batch_queue2\n```\n\nOr clone the repository from [Sourcehut](https://hg.sr.ht/~ndbecker2/batch_queue2/):\n\n```sh\nhg clone https://hg.sr.ht/~ndbecker2/batch_queue2\ncd batch_queue2\npython -m pip install .\n```\n\n## Usage\n\nAfter installing, you can use the `batch_queue` command to manage tasks. Below are the available options:\n\n### Starting the Server\nTo start the server:\n\n```sh\nbatch_queue start --max-cpus 4\n```\n\n- `--max-cpus`: (Optional) Specify the maximum number of CPUs to use. Defaults to the number of CPUs available on your system.\n\nThe server will start in daemon mode by default.\n\n### Submitting a Task\nTo submit a task, use the `submit` command:\n\n```sh\nbatch_queue submit <command>\n```\nFor example:\n\n```sh\nbatch_queue submit sleep 10\n```\n\nYou can also optionally specify:\n- `--log-stdout <file>`: Redirect the standard output of the task to a file.\n- `--log-stderr <file>`: Redirect the standard error of the task to a file.\n\n### Listing Tasks\nTo list all tasks:\n\n```sh\nbatch_queue list\n```\nThis will display:\n- Max CPUs available.\n- Active tasks.\n- Queued tasks.\n- Paused tasks.\n- Runnable paused tasks.\n\n### Suspending and Resuming Tasks\nTo suspend task(s) (running or queued):\n\n```sh\nbatch_queue suspend <task_ids>\n```\n\nTo resume a paused task(s):\n\n```sh\nbatch_queue resume <task_ids>\n```\n\n### Killing Task(s)\nTo kill specific task(s):\n\n```sh\nbatch_queue kill <task_ids> <signal>\n```\nYou can also optionally specify the signal to use, default is `SIGTERM`.\n\n### Getting Task Information\nTo get detailed information about a specific task:\n\n```sh\nbatch_queue id <task_id>\n```\nThis command provides detailed information about the task including command, user, working directory, environment variables, and logs.\n\n### Change #cpus\nYou can reset the max #cpus. When you do this, if the new maximum is\ngreater than the number of active tasks the scheduler is recalled and\nqueued tasks can be started. If the new maximum is smaller than the\nnumber of active tasks tasks will be suspended, but marked runnable to\nbe continued later.\n\n```sh\nbatch_queue setcpus #cpus\n```\n\n### Stopping the Server\nTo stop the server:\n\n```sh\nbatch_queue stop\n```\nThis command gracefully stops the server, ensuring no tasks are left in a zombie state.\n\n## Example Workflow\n1. Start the server using:\n ```sh\n batch_queue start --max-cpus 4\n ```\n\n2. Submit a couple of tasks:\n ```sh\n batch_queue submit sleep 10\n batch_queue submit echo \"Hello World\"\n ```\n\n3. List the tasks to see the active, queued, and paused tasks:\n ```sh\n batch_queue list\n ```\n\n4. Suspend running tasks:\n ```sh\n batch_queue suspend 0\n ```\n\n5. Resume paused tasks:\n ```sh\n batch_queue resume 0\n ```\n\n6. Stop the server:\n ```sh\n batch_queue stop\n ```\n\n## Logging\nThe server logs all activity to `~/batch_queue.log`. You can view the log to monitor task submissions, task status changes, server starts and stops, etc.\n\n## License\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\n\n## Security\nThe server listens on a unix domain socket normally in the user's home\ndirectory who starts it. Only users with access to this socket can\nuse it.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A batch queue system implemented with asyncio",
"version": "1.0.12",
"project_urls": {
"Homepage": "https://sr.ht/~ndbecker2/batch_queue2/",
"Repository": "https://hg.sr.ht/~ndbecker2/batch_queue2"
},
"split_keywords": [
"queue",
" asyncio",
" batch"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "467f9e0bb57aef5d4772b4589a7b46b70d58c327e3f720c973a4c80f2e1c137e",
"md5": "5fb104733c38361aeca384b20e64e8d6",
"sha256": "28ef5f8c4051c0f2f36b1a44ff660e6c844509b5f4390017f76c88c51cfd7cb8"
},
"downloads": -1,
"filename": "batch_queue2-1.0.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5fb104733c38361aeca384b20e64e8d6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 9976,
"upload_time": "2024-12-03T22:10:59",
"upload_time_iso_8601": "2024-12-03T22:10:59.949834Z",
"url": "https://files.pythonhosted.org/packages/46/7f/9e0bb57aef5d4772b4589a7b46b70d58c327e3f720c973a4c80f2e1c137e/batch_queue2-1.0.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ff785d29fa91695167cfbd42b186da11a1bc68572c1080313f6f9ebd1b5af4d6",
"md5": "1d692ac22b7de1b869ef7c4484a89e33",
"sha256": "ba05e3e7d74b17be3ee197723368d4e5eb4e5900d5fbe0d574933d1e7ab1bbde"
},
"downloads": -1,
"filename": "batch_queue2-1.0.12.tar.gz",
"has_sig": false,
"md5_digest": "1d692ac22b7de1b869ef7c4484a89e33",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 12056,
"upload_time": "2024-12-03T22:11:01",
"upload_time_iso_8601": "2024-12-03T22:11:01.537555Z",
"url": "https://files.pythonhosted.org/packages/ff/78/5d29fa91695167cfbd42b186da11a1bc68572c1080313f6f9ebd1b5af4d6/batch_queue2-1.0.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-03 22:11:01",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "batch-queue2"
}
JSON