| Name | sball JSON |
| Version |
1.0.5
JSON |
| download |
| home_page | |
| Summary | Submit scripts in job arrays using Yale's dSQ. |
| upload_time | 2023-05-17 18:25:46 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3 |
| license | |
| keywords |
slurm
job array
sbatch
dsq
dsq
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# sball
`sball` facilitates submitting multiple scripts in Slurm job arrays using Yale's dSQ module. "sball" is short for "sbatch all".
## Usage
After installing using `pip`, you can run `sball` as a command. It takes a few arguments, which are the following. Any additional arguments not provided here are passed along to `sbatch` when the job array is submitted. Arguments must be specified as `key=value` pairs, with the exception of the `pattern` argument, which should always be passed as the final argument. Additional arguments should be specified in the usual way for `sbatch`; i.e., `--$argument_name $argument_value`.
If you need to include spaces in arguments, make sure to escape them by using single quotes. If you do not escape `pattern` using single quotes, make sure to escape any `*` in the glob expression with a backslash, since otherwise they will be expanded by the shell before sball gets them.
- `name`: (required) the name to give the job array.
- `regex`: (optional) only scripts with filenames matching `regex` will be included in any arrays.
- `log_dir`: (optional) the directory where log files for the job array should be stored. Default is `joblogs`.
- (`pattern`): (required, not named) a glob expression that matches scripts to include in job arrays. Only files ending in `.sh` that match the glob will be included. If a glob expression is insufficient to filter to just the scripts you want, you should use the `regex` argument.
- Additional arguments are passed to the underlying calls to `sbatch`. This allows you to, e.g., set up job dependencies. They should be inserted _before_ the final, pattern argument.
## Description
`sball` finds all `.sh` scripts matching `pattern` (and `regex`, if provided). If scripts are found, it sorts them into bins with unique sets of SBATCH options (since job arrays must all be run with the same SBATCH options). For each bin, the scripts are added to a job file, where the contents of each script takes up a single line. These files are named `$name.txt`, and are saved in the deepest directory common to all scripts in a bin. In case there is more than one bin, the file is suffixed with their bin number. Then, `dsq` called to create a job script from this file with the SBATCH options for that bin. Finally, the created job script is submitted to the queue with `sbatch`, and the (now unnecessary) job script is removed. The `$name.txt` files must be present when the job actually procs in the queue, so they are not removed (and should not be removed until after the jobs finish).
In case a bin contains only one script, that script is just submitted in the usual way with `sbatch`, as a fallback.
## Examples
Sample directory structure:
```
main-project-directory/
├── ...
│
└── scripts/
├── script1.sh
├── script2.sh
├── script3.sh
└── ...
```
To submit all the scripts in `main-project-directory/scripts` in a job array named `my-job-array`, from `main-project-directory` you would run:
```bash
module load dSQ # if not already loaded
sball name=my-job-array scripts/\*
```
Note that the `*` in the glob expression in `pattern` is escaped. After running this, your job array will be created, and you will have a new file in `main-project-directory/scripts` that Slurm references to find the jobs.
```
main-project-directory/
├── ...
│
└── scripts/
├── script1.sh
├── script2.sh
├── script3.sh
├── ...
└── my-job-array.txt
```
After your job array finishes running, you can safely remove `my-job-array.txt`. You can make further refinements of which scripts to run by changing the glob expression, or by using the `regex=` argument.
Raw data
{
"_id": null,
"home_page": "",
"name": "sball",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": "",
"keywords": "slurm,job array,sbatch,dSQ,dsq",
"author": "",
"author_email": "Michael Wilson <michael.a.wilson@yale.edu>",
"download_url": "https://files.pythonhosted.org/packages/1f/9e/e84cee5242a50b03695b371c6fca5b38b9316c9b09ac1c5129b558c9b134/sball-1.0.5.tar.gz",
"platform": null,
"description": "# sball\r\n\r\n`sball` facilitates submitting multiple scripts in Slurm job arrays using Yale's dSQ module. \"sball\" is short for \"sbatch all\".\r\n\r\n## Usage\r\n\r\nAfter installing using `pip`, you can run `sball` as a command. It takes a few arguments, which are the following. Any additional arguments not provided here are passed along to `sbatch` when the job array is submitted. Arguments must be specified as `key=value` pairs, with the exception of the `pattern` argument, which should always be passed as the final argument. Additional arguments should be specified in the usual way for `sbatch`; i.e., `--$argument_name $argument_value`.\r\n\r\nIf you need to include spaces in arguments, make sure to escape them by using single quotes. If you do not escape `pattern` using single quotes, make sure to escape any `*` in the glob expression with a backslash, since otherwise they will be expanded by the shell before sball gets them.\r\n\r\n- `name`: (required) the name to give the job array.\r\n- `regex`: (optional) only scripts with filenames matching `regex` will be included in any arrays.\r\n- `log_dir`: (optional) the directory where log files for the job array should be stored. Default is `joblogs`.\r\n- (`pattern`): (required, not named) a glob expression that matches scripts to include in job arrays. Only files ending in `.sh` that match the glob will be included. If a glob expression is insufficient to filter to just the scripts you want, you should use the `regex` argument.\r\n- Additional arguments are passed to the underlying calls to `sbatch`. This allows you to, e.g., set up job dependencies. They should be inserted _before_ the final, pattern argument.\r\n\r\n## Description\r\n\r\n`sball` finds all `.sh` scripts matching `pattern` (and `regex`, if provided). If scripts are found, it sorts them into bins with unique sets of SBATCH options (since job arrays must all be run with the same SBATCH options). For each bin, the scripts are added to a job file, where the contents of each script takes up a single line. These files are named `$name.txt`, and are saved in the deepest directory common to all scripts in a bin. In case there is more than one bin, the file is suffixed with their bin number. Then, `dsq` called to create a job script from this file with the SBATCH options for that bin. Finally, the created job script is submitted to the queue with `sbatch`, and the (now unnecessary) job script is removed. The `$name.txt` files must be present when the job actually procs in the queue, so they are not removed (and should not be removed until after the jobs finish).\r\n\r\nIn case a bin contains only one script, that script is just submitted in the usual way with `sbatch`, as a fallback.\r\n\r\n## Examples\r\n\r\nSample directory structure:\r\n```\r\nmain-project-directory/\r\n\u251c\u2500\u2500 ...\r\n\u2502\r\n\u2514\u2500\u2500 scripts/\r\n\t\u251c\u2500\u2500 script1.sh\r\n\t\u251c\u2500\u2500 script2.sh\r\n\t\u251c\u2500\u2500 script3.sh\r\n\t\u2514\u2500\u2500 ...\r\n```\r\n\r\nTo submit all the scripts in `main-project-directory/scripts` in a job array named `my-job-array`, from `main-project-directory` you would run:\r\n```bash\r\nmodule load dSQ # if not already loaded\r\nsball name=my-job-array scripts/\\*\r\n```\r\n\r\nNote that the `*` in the glob expression in `pattern` is escaped. After running this, your job array will be created, and you will have a new file in `main-project-directory/scripts` that Slurm references to find the jobs.\r\n```\r\nmain-project-directory/\r\n\u251c\u2500\u2500 ...\r\n\u2502\r\n\u2514\u2500\u2500 scripts/\r\n\t\u251c\u2500\u2500 script1.sh\r\n\t\u251c\u2500\u2500 script2.sh\r\n\t\u251c\u2500\u2500 script3.sh\r\n\t\u251c\u2500\u2500 ...\r\n\t\u2514\u2500\u2500 my-job-array.txt\r\n```\r\nAfter your job array finishes running, you can safely remove `my-job-array.txt`. You can make further refinements of which scripts to run by changing the glob expression, or by using the `regex=` argument.\r\n",
"bugtrack_url": null,
"license": "",
"summary": "Submit scripts in job arrays using Yale's dSQ.",
"version": "1.0.5",
"project_urls": {
"Homepage": "https://github.com/mawilson1234/sball"
},
"split_keywords": [
"slurm",
"job array",
"sbatch",
"dsq",
"dsq"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3184d3781bcd63f20608bccf68a6c97ad1002ae25414a6876190047f442647ec",
"md5": "d0e3cdefb34e6f075480cb95ed6a6aad",
"sha256": "78ac2f02f5d39f14dee34719775aabda09458ed49d1f03ffefbedbef27afa6bc"
},
"downloads": -1,
"filename": "sball-1.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d0e3cdefb34e6f075480cb95ed6a6aad",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3",
"size": 6248,
"upload_time": "2023-05-17T18:25:44",
"upload_time_iso_8601": "2023-05-17T18:25:44.276536Z",
"url": "https://files.pythonhosted.org/packages/31/84/d3781bcd63f20608bccf68a6c97ad1002ae25414a6876190047f442647ec/sball-1.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1f9ee84cee5242a50b03695b371c6fca5b38b9316c9b09ac1c5129b558c9b134",
"md5": "df90eef8c3ae812874f5da0ec667fe11",
"sha256": "7194ab494707752ea55df439aef68df89e5d81067755ac5d2e84ac03187cbbef"
},
"downloads": -1,
"filename": "sball-1.0.5.tar.gz",
"has_sig": false,
"md5_digest": "df90eef8c3ae812874f5da0ec667fe11",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3",
"size": 5619,
"upload_time": "2023-05-17T18:25:46",
"upload_time_iso_8601": "2023-05-17T18:25:46.108455Z",
"url": "https://files.pythonhosted.org/packages/1f/9e/e84cee5242a50b03695b371c6fca5b38b9316c9b09ac1c5129b558c9b134/sball-1.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-17 18:25:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mawilson1234",
"github_project": "sball",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "sball"
}
