reportseff


Namereportseff JSON
Version 2.7.6 PyPI version JSON
download
home_pagehttps://github.com/troycomi/reportseff
SummaryTablular seff output
upload_time2023-09-14 15:51:40
maintainer
docs_urlNone
authorTroy Comi
requires_python>=3.7.0,<4.0
licenseMIT
keywords slurm seff
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            [![Tests](https://github.com/troycomi/reportseff/workflows/Tests/badge.svg)](https://github.com/troycomi/reportseff/actions?workflow=Tests)
[![codecov](https://codecov.io/gh/troycomi/reportseff/branch/main/graph/badge.svg)](https://codecov.io/gh/troycomi/reportseff)
[![PyPI](https://img.shields.io/pypi/v/reportseff.svg)](https://pypi.org/project/reportseff/)
[![DOI](https://img.shields.io/badge/doi-10.1145/3569951.3604396-blue.svg?style=flat&labelColor=whitesmoke&logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAAB8AAAAfCAYAAAAfrhY5AAAJsklEQVR42qWXd1DTaRrHf%2BiB2Hdt5zhrAUKz4IKEYu9IGiGFFJJQ0gkJCAKiWFDWBRdFhCQUF3UVdeVcRQEBxUI3yY9iEnQHb3bdW1fPubnyz%2F11M7lvEHfOQee2ZOYzPyDv%2B3yf9%2Fk95YX4fx%2BltfUt08GcFEuPR4U9hDDZ%2FVngIlhb%2FSiI6InkTgLzgDcgfvtnovhH4BzoVlrbwr55QnhCtBW4QHXnFrZbPBaQoBh4%2FSYH2EnpBEtqcDMVzB93wA%2F8AFwa23XFGcc8CkT3mxz%2BfXWtq9T9IQlLIXYEuHojudb%2BCM7Hgdq8ydi%2FAHiBXyY%2BLjwFlAEnS6Jnar%2FvnQVhvdzasad0eKvWZKe8hvDB2ofLZ%2FZEcWsh%2BhyIuyO5Bxs2iZIE4nRv7NWAb0EO8AC%2FWPxjYAWuOEX2MSXZVgPxzmRL3xKz3ScGpx6p6QnOx4mDIFqO0w6Q4fEhO5IzwxlSwyD2FYHzwAW%2BAZ4fEsf74gCumykwNHskLM7taQxLYjjIyy8MUtraGhTWdkfhkFJqtvuVl%2F9l2ZquDfEyrH8B0W06nnpH3JtIyRGpH1iJ6SfxDIHjRXHJmdQjLpfHeN54gnfFx4W9QRnovx%2FN20aXZeTD2J84hn3%2BqoF2Tqr14VqTPUCIcP%2B5%2Fly4qC%2BUL3sYxSvNj1NwsVYPsWdMUfomsdkYm3Tj0nbV0N1wRKwFe1MgKACDIBdMAhPE%2FwicwNWxll8Ag40w%2BFfhibJkGHmutjYeQ8gVlaN%2BjO51nDysa9TwNUFMqaGbKdRJZFfOJSp6mkRKsv0rRIpEVWjAvyFkxNOEpwvcAVPfEe%2Bl8ojeNTx3nXLBcWRrYGxSRjDEk0VlpxYrbe1ZmaQ5xuT0u3r%2B2qe5j0J5uytiZPGsRL2Jm32AldpxPUNJ3jmmsN4x62z1cXrbedXBQf2yvIFCeZrtyicZZG2U2nrrBJzYorI2EXLrvTfCSB43s41PKEvbZDEfQby6L4JTj%2FfIwam%2B4%2BwucBu%2BDgNK05Nle1rSt9HvR%2FKPC4U6LTfvUIaip1mjIa8fPzykii23h2eanT57zQ7fsyYH5QjywwlooAUcAdOh5QumgTHx6aAO7%2FL52eaQNEShrxfhL6albEDmfhGflrsT4tps8gTHNOJbeDeBlt0WJWDHSgxs6cW6lQqyg1FpD5ZVDfhn1HYFF1y4Eiaqa18pQf3zzYMBhcanlBjYfgWNayAf%2FASOgklu8bmgD7hADrk4cRlOL7NSOewEcbqSmaivT33QuFdHXj5sdvjlN5yMDrAECmdgDWG2L8P%2BAKLs9ZLZ7dJda%2BB4Xl84t7QvnKfvpXJv9obz2KgK8dXyqISyV0sXGZ0U47hOA%2FAiigbEMECJxC9aoKp86re5O5prxOlHkcksutSQJzxZRlPZmrOKhsQBF5zEZKybUC0vVjG8PqOnhOq46qyDTDnj5gZBriWCk4DvXrudQnXQmnXblebhAC2cCB6zIbM4PYgGl0elPSgIf3iFEA21aLdHYLHUQuVkpgi02SxFdrG862Y8ymYGMvXDzUmiX8DS5vKZyZlGmsSgQqfLub5RyLNS4zfDiZc9Edzh%2FtCE%2BX8j9k%2FqWB071rcZyMImne1SLkL4GRw4UPHMV3jjwEYpPG5uW5fAEot0aTSJnsGAwHJi2nvF1Y5OIqWziVCQd5NT7t6Q8guOSpgS%2Fa1dSRn8JGGaCD3BPXDyQRG4Bqhu8XrgAp0yy8DMSvvyVXDgJcJTcr1wQ2BvFKf65jqhvmxXUuDpGBlRvV36XvGjQzLi8KAKT2lYOnmxQPGorURSV0NhyTIuIyqOmKTMhQ%2BieEsgOgpc4KBbfDM4B3SIgFljvfHF6cef7qpyLBXAiQcXvg5l3Iunp%2FWv4dH6qFziO%2BL9PbrimQ9RY6MQphEfGUpOmma7KkGzuS8sPUFnCtIYcKCaI9EXo4HlQLgGrBjbiK5EqMj2AKWt9QWcIFMtnVvQVDQV9lXJJqdPVtUQpbh6gCI2Ov1nvZts7yYdsnvRgxiWFOtNJcOMVLn1vgptVi6qrNiFOfEjHCDB3J%2BHDLqUB77YgQGwX%2Fb1eYna3hGKdlqJKIyiE4nSbV8VFgxmxR4b5mVkkeUhMgs5YTi4ja2XZ009xJRHdkfwMi%2BfocaancuO7h%2FMlcLOa0V%2FSw6Dq47CumRQAKhgbOP8t%2BMTjuxjJGhXCY6XpmDDFqWlVYbQ1aDJ5Cptdw4oLbf3Ck%2BdWkVP0LpH7s9XLPXI%2FQX8ws%2Bj2In63IcRvOOo%2BTTjiN%2BlssfRsanW%2B3REVKoavBOAPTXABW4AL7e4NygHdpAKBscmlDh9Jysp4wxbnUNna3L3xBvyE1jyrGIkUHaqQMuxhHElV6oj1picvgL1QEuS5PyZTEaivqh5vUCKJqOuIgPFGESns8kyFk7%2FDxyima3cYxi%2FYOQCj%2F%2B9Ms2Ll%2Bhn4FmKnl7JkGXQGDKDAz9rUGL1TIlBpuJr9Be2JjK6qPzyDg495UxXYF7JY1qKimw9jWjF0iV6DRIqE%2B%2FeWG0J2ofmZTk0mLYVd4GLiFCOoKR0Cg727tWq981InYynvCuKW43aXgEjofVbxIqrm0VL76zlH3gQzWP3R3Bv9oXxclrlO7VVtgBRpSP4hMFWJ8BrUSBCJXC07l40X4jWuvtc42ofNCxtlX2JH6bdeojXgTh5TxOBKEyY5wvBE%2BACh8BtOPNPkApjoxi5h%2B%2FFMQQNpWvZaMH7MKFu5Ax8HoCQdmGkJrtnOiLHwD3uS5y8%2F2xTSDrE%2F4PT1yqtt6vGe8ldMBVMEPd6KwqiYECHDlfbvzphcWP%2BJiZuL5swoWQYlS%2Br7Yu5mNUiGD2retxBi9fl6RDGn4Ti9B1oyYy%2BMP5G87D%2FCpRlvdnuy0PY6RC8BzTA40NXqckQ9TaOUDywkYsudxJzPgyDoAWn%2BB6nEFbaVxxC6UXjJiuDkW9TWq7uRBOJocky9iMfUhGpv%2FdQuVVIuGjYqACbXf8aa%2BPeYNIHZsM7l4s5gAQuUAzRUoT51hnH3EWofXf2vkD5HJJ33vwE%2FaEWp36GHr6GpMaH4AAPuqM5eabH%2FhfG9zcCz4nN6cPinuAw6IHwtvyB%2FdO1toZciBaPh25U0ducR2PI3Zl7mokyLWKkSnEDOg1x5fCsJE9EKhH7HwFNhWMGMS7%2BqxyYsbHHRUDUH4I%2FAheQY7wujJNnFUH4KdCju83riuQeHU9WEqNzjsJFuF%2FdTDAZ%2FK7%2F1WaAU%2BAWymT59pVMT4g2AxcwNa0XEBDdBDpAPvgDIH73R25teeuAF5ime2Ul0OUIiG4GpSAEJeYW9wDTf43wfwHgHLKJoPznkwAAAABJRU5ErkJggg%3D%3D)](https://doi.org/10.1145/3569951.3604396)

# `reportseff`

> A python script for tabular display of slurm efficiency information

![Example](https://github.com/troycomi/reportseff/raw/main/imgs/example.png)

## About

### Motivation

Whether a sys admin or cluster user, knowing how well you are estimating job
resources can help streamline job scheduling and maximize your priority. If you
have ever tried to use `sacct` you probably had some trouble interpreting the
output.  While `seff` or `jobstats` can provide detailed summaries, they don't
scale easily to array jobs or offer a way to see all the jobs from a single
user.  `reportseff` aims to fill this role.  Read more about the [motivation
for reportseff](https://github.com/troycomi/reportseff/blob/main/ABOUT.md).

### Audience

If you are running more than one slurm job at a time, you should try
`reportseff`.  Users of HPC systems can get an idea how well they estimate
resource usage.  By tuning these values, you can get scheduled earlier and not
be penalized for unused allocations.  Since `reportseff` can parse job ids from
slurm output files, it simplifies the task of identifying which jobs have
failed and why. Sys admins can pipe `reportseff` output to identify users with
poor utilization or produce summaries at the end of a billing cycle.

### Implementation

`reportseff` is a wrapper around `sacct` that provides more complex option
parsing, simpler options, and cleaner, colored outputs.  All querying is
performed in a single call to `sacct` and should have similar performance.
Multi-node and GPU utilization is acquired from information contained in the
`AdminComment` field, as generated by `jobstats`.

## Usage

### Installation

`reportseff` runs on python >= 3.6.
The only external dependency is click (>= 6.7).
Calling

```sh
pip install --user reportseff
# OR
pipx install reportseff
```

will create command line bindings and install click.

### Sample Usage

Try `reportseff -u $USER` or just `reportseff` in a directory with some slurm
outputs.  You may be surprised by your results!

#### Single job

Calling `reportseff` with a single jobid will provide equivalent information to
seff for that job. `reportseff 24371789` and `reportseff map_fastq_24371789`
produce the following output:

```txt
   JobID      State          Elapsed   CPUEff   MemEff
24371789    COMPLETED       03:08:03   71.2%    45.7%
```

#### Single array job

Providing either the raw job id or the array job id will get efficiency
information for a single element of the array job. `reportseff 24220929_421`
and `reportseff 24221219` generate:

```txt
       JobID      State          Elapsed    CPUEff   MemEff
24220929_421    COMPLETED       00:09:34    99.0%    34.6%
```

#### Array job group

If the base job id of an array is provided, all elements of the array will
be added to the output. `reportseff 24220929`

```txt
          JobID      State          Elapsed    CPUEff   MemEff
     24220929_1    COMPLETED       00:10:43    99.2%    33.4%
    24220929_11    COMPLETED       00:10:10    99.2%    37.5%
    24220929_21    COMPLETED       00:09:25    98.8%    36.1%
    24220929_31    COMPLETED       00:09:19    98.9%    33.3%
    24220929_41    COMPLETED       00:09:23    98.9%    33.3%
    24220929_51    COMPLETED       00:08:02    98.5%    36.3%
    ...
   24220929_951    COMPLETED       00:25:12    99.5%    33.5%
   24220929_961    COMPLETED       00:39:26    99.7%    34.1%
   24220929_971    COMPLETED       00:24:11    99.5%    34.2%
   24220929_981    COMPLETED       00:24:50    99.5%    44.3%
   24220929_991    COMPLETED       00:13:05    98.7%    33.7%
```

#### Glob expansion of slurm outputs

Because slurm output files can act as job id inputs, the following can
get all seff information for a given job name:

```txt
slurm_out  ❯❯❯ reportseff split_ubam_24*
              JobID      State          Elapsed   CPUEff   MemEff
split_ubam_24342816    COMPLETED       23:30:32   99.9%    4.5%
split_ubam_24342914    COMPLETED       22:40:51   99.9%    4.6%
split_ubam_24393599    COMPLETED       23:43:36   99.4%    4.4%
split_ubam_24393655    COMPLETED       21:36:58   99.3%    4.5%
split_ubam_24418960     RUNNING        02:53:11    ---      ---
split_ubam_24419972     RUNNING        01:26:26    ---      ---
```

#### No arguments

Without arguments, reportseff will try to find slurm output files in the
current directory. Combine with `watch` to monitor job progress:
`watch -cn 300 reportseff --color --modified-sort`

```txt
                JobID           State          Elapsed   CPUEff   MemEff
   split_ubam_24418960          RUNNING        02:56:14    ---      ---
fastq_to_ubam_24419971          RUNNING        01:29:29    ---      ---
   split_ubam_24419972          RUNNING        01:29:29    ---      ---
fastq_to_ubam_24393600         COMPLETED     1-02:00:47   58.3%    41.1%
    map_fastq_24419330          RUNNING        02:14:53    ---      ---
    map_fastq_24419323          RUNNING        02:15:24    ---      ---
    map_fastq_24419324          RUNNING        02:15:24    ---      ---
    map_fastq_24419322          RUNNING        02:15:24    ---      ---
mark_adapters_24418437         COMPLETED       01:29:23   99.8%    48.2%
mark_adapters_24418436         COMPLETED       01:29:03   99.9%    47.4%
```

#### Filtering slurm output files

One useful application of `reportseff` is filtering a directory of slurm output
files based on the state or time since running. Additionally, if only the
`jobid` is specified as a format output, the filenames will be returned in a
pipe-friendly manner:

```txt
old_runs   ❯❯❯ reportseff --since d=4 --state Timeout

                 JobID   State      Elapsed  CPUEff   MemEff
call_variants_31550458  TIMEOUT    20:05:17  99.5%     0.0%
call_variants_31550474  TIMEOUT    20:05:17  99.6%     0.0%
call_variants_31550500  TIMEOUT    20:05:08  99.7%     0.0%
old_runs   ❯❯❯ reportseff --since d=4 --state Timeout --format jobid
call_variants_31550458
call_variants_31550474
call_variants_31550500
```

To find all lines with `output:` in jobs which have timed out or failed
in the last 4 days:

```sh
reportseff --since 'd=4' --state TO,F --format jobid | xargs grep output:
```

### Arguments

Jobs can be passed as arguments in the following ways:

- Job ID such as 1234567.  If the id is part of an array job, only the element
for that ID will be displayed.  If the id is the base part of an array job,
all elements in the array will be displayed.
- Array Job ID such as 1234567\_89.  Will display only the element specified.
- Slurm output file.  Format must be BASE\_%A\_%a.  BASE is optional as is a
'.out' suffix.  Unix glob expansions can also be used to filter which jobs
are displayed.
- From current directory.  If no argument is supplied, `reportseff` will attempt
to find slurm output files in the current directory as described above.
If a user is provided, instead `reportseff` will show recent jobs for that user.
If only `since` is set, all recent jobs for all users will be shown (if allowed).
- Supplying a directory as a single argument will override the current
directory to check for slurm outputs.

### Options

- `--color/--no-color`: Force color output or not.  By default, will force color
  output.  With the no-color flag, click will strip color codes for everything
  besides stdout.
- `--modified-sort`: Instead of sorting by filename/jobid, sort by last
  modification time of the slurm output file.
- `--debug`: Write sacct result to stderr.
- `--user/-u`: Ignore job arguments and instead query sacct with provided user.
  Returns all jobs from the last week.
- `--state/-s`: Output only jobs with states matching one of the provided options.
  Accepts comma separated values of job codes (e.g. 'R') or full names
  (e.g. RUNNING).  Case insensitive.
- `--not-state/-S`: Output only jobs with states not matching any of the provided options.
  Accepts comma separated values of job codes (e.g. 'R') or full names
  (e.g. RUNNING).  Case insensitive.
- `--format`: Provide a comma separated list of columns to produce. Prefixing the
  argument with `+` adds the specified values to the defaults.  Values can
  be any valid column name to sacct and the custom efficiency values: TimeEff,
  cpuEff, MemEff.  Can also optionally set alignment (<, ^, >) and maximum width.
  Default is center-aligned with a width of the maximum column entry.  For
  example, `--format 'jobid%>,state%10,memeff%<5'` produces 3 columns with:
  - JobId aligned right, width set automatically
  - State with width 10 (center aligned by default)
  - MemEff aligned left, width 5
- `--slurm-format`: The filename pattern passed to sbatch during job submission.
  Overrides the default regex for job id parsing from filenames.  E.g. to match
  filenames like `123456.out` set `--slurm-format %j.out`.
- `--since`: Limit results to those occurring after the specified time.  Accepts
  sacct formats and a comma separated list of key/value pairs.  To get jobs in
  the last hour and a half, can pass `h=1,m=30`.
-`--until`: Limit results to those occurring before the specified time. Accepts
  sacct formats and a comma separated list of key/value pairs.
  Useful in combination with the 'since' option to query a specific range.
- `--partition`: Limit results to a specific partition.
- `--node/-n`: Display information for multi-node jobs; requires additional
  sacct fields from jobstats.
- `--node-and-gpu/-g`: Display information for multi-node jobs and GPU information;
  requires additional sacct fields from jobstats.
- `--parsable/-p`: Ignore formatting and output as a `|` delimited table.  Useful
  for piping into more complex analyses.

## Status, Contributions, and Support

`reportseff` is actively maintained but currently feature complete.  If there
is a function missing, please open an issue to discuss its merit!

Bug reports, pull requests, and any feedback are welcome! Prior to submitting
a pull request, be sure any new features have been tested and all unit tests
are passing. In the cloned repo with
[poetry](https://github.com/python-poetry/poetry#installation) installed:

```sh
poetry install
poetry run pytest
poetry run pre-commit install
nox
```

## Troubleshooting

### I can't install, what is pip?

[pip](https://pip.pypa.io/en/stable/) is the package installer for python.  If
you get an error that pip isn't found, look for a python/anaconda/conda module.
[pipx](https://pypa.github.io/pipx/) ensures that each application is installed
in an isolated environment.  This resolves issues of dependency versions and
allows applications to be run from any environment.

### The output has no color with many jobs!

Click should determine if the output supports color display and react automatically
in a way you expect.  Check that your terminal is setup to display colors and
that your pager (probably less) will display color by default.  Some commands,
e.g. `watch` aren't handled properly even when invoked to support color.  Here
are some useful settings for your `.bashrc`:
```
# have less display colors by default.  Will fix `reportseff` not showing colors
export LESS="-R"
# for watch aliases, include the `--color` option
watch -cn 300 reportseff --color --modified-sort
#      ^                 ^^^^^^^
```
You can always for display of color (or suppress it) with the `--color/--no-color`
options

### I get an error about broken pipes when chaining to other commands

Python will report that the consumer of process output has closed the stream
(i.e. the pipe) while still attempting to write.  Newer versions of click
should suppress the warning output, but it seems to not always work.  Besides
some extra printing on stderr, the output is not affected.

### My jobs don't have any information about multiple nodes or GPU efficiency

Because `sacct` doesn't currently record this information, `reportseff`
retrieves it from a custom field from `jobstats`, developed at Princeton
University.  If you are outside a Research Computing cluster, that information
will likely be absent.  Node-level reporting is only shown for jobs which use
multiple nodes or GPUs.  If you need a list of where jobs were run, you can add
`--format +NodeList`.

## Acknowledgments

The code for calling sacct and parsing the returning information was taken
from [Slurmee](https://github.com/PrincetonUniversity/slurmee).

Style and tooling from [hypermodern python](https://cjolowicz.github.io/posts/hypermodern-python-01-setup/)

Code review provided from a [repo-review](https://researchcomputing.princeton.edu/services/repo-review-consultations)
which vastly improved this readme.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/troycomi/reportseff",
    "name": "reportseff",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.7.0,<4.0",
    "maintainer_email": "",
    "keywords": "slurm,seff",
    "author": "Troy Comi",
    "author_email": "troycomi@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/15/4e/3999f38c1021ad7d06a2980b1dea53a50489d481f398dfb3e0e8611b988a/reportseff-2.7.6.tar.gz",
    "platform": null,
    "description": "[![Tests](https://github.com/troycomi/reportseff/workflows/Tests/badge.svg)](https://github.com/troycomi/reportseff/actions?workflow=Tests)\n[![codecov](https://codecov.io/gh/troycomi/reportseff/branch/main/graph/badge.svg)](https://codecov.io/gh/troycomi/reportseff)\n[![PyPI](https://img.shields.io/pypi/v/reportseff.svg)](https://pypi.org/project/reportseff/)\n[![DOI](https://img.shields.io/badge/doi-10.1145/3569951.3604396-blue.svg?style=flat&labelColor=whitesmoke&logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAAB8AAAAfCAYAAAAfrhY5AAAJsklEQVR42qWXd1DTaRrHf%2BiB2Hdt5zhrAUKz4IKEYu9IGiGFFJJQ0gkJCAKiWFDWBRdFhCQUF3UVdeVcRQEBxUI3yY9iEnQHb3bdW1fPubnyz%2F11M7lvEHfOQee2ZOYzPyDv%2B3yf9%2Fk95YX4fx%2BltfUt08GcFEuPR4U9hDDZ%2FVngIlhb%2FSiI6InkTgLzgDcgfvtnovhH4BzoVlrbwr55QnhCtBW4QHXnFrZbPBaQoBh4%2FSYH2EnpBEtqcDMVzB93wA%2F8AFwa23XFGcc8CkT3mxz%2BfXWtq9T9IQlLIXYEuHojudb%2BCM7Hgdq8ydi%2FAHiBXyY%2BLjwFlAEnS6Jnar%2FvnQVhvdzasad0eKvWZKe8hvDB2ofLZ%2FZEcWsh%2BhyIuyO5Bxs2iZIE4nRv7NWAb0EO8AC%2FWPxjYAWuOEX2MSXZVgPxzmRL3xKz3ScGpx6p6QnOx4mDIFqO0w6Q4fEhO5IzwxlSwyD2FYHzwAW%2BAZ4fEsf74gCumykwNHskLM7taQxLYjjIyy8MUtraGhTWdkfhkFJqtvuVl%2F9l2ZquDfEyrH8B0W06nnpH3JtIyRGpH1iJ6SfxDIHjRXHJmdQjLpfHeN54gnfFx4W9QRnovx%2FN20aXZeTD2J84hn3%2BqoF2Tqr14VqTPUCIcP%2B5%2Fly4qC%2BUL3sYxSvNj1NwsVYPsWdMUfomsdkYm3Tj0nbV0N1wRKwFe1MgKACDIBdMAhPE%2FwicwNWxll8Ag40w%2BFfhibJkGHmutjYeQ8gVlaN%2BjO51nDysa9TwNUFMqaGbKdRJZFfOJSp6mkRKsv0rRIpEVWjAvyFkxNOEpwvcAVPfEe%2Bl8ojeNTx3nXLBcWRrYGxSRjDEk0VlpxYrbe1ZmaQ5xuT0u3r%2B2qe5j0J5uytiZPGsRL2Jm32AldpxPUNJ3jmmsN4x62z1cXrbedXBQf2yvIFCeZrtyicZZG2U2nrrBJzYorI2EXLrvTfCSB43s41PKEvbZDEfQby6L4JTj%2FfIwam%2B4%2BwucBu%2BDgNK05Nle1rSt9HvR%2FKPC4U6LTfvUIaip1mjIa8fPzykii23h2eanT57zQ7fsyYH5QjywwlooAUcAdOh5QumgTHx6aAO7%2FL52eaQNEShrxfhL6albEDmfhGflrsT4tps8gTHNOJbeDeBlt0WJWDHSgxs6cW6lQqyg1FpD5ZVDfhn1HYFF1y4Eiaqa18pQf3zzYMBhcanlBjYfgWNayAf%2FASOgklu8bmgD7hADrk4cRlOL7NSOewEcbqSmaivT33QuFdHXj5sdvjlN5yMDrAECmdgDWG2L8P%2BAKLs9ZLZ7dJda%2BB4Xl84t7QvnKfvpXJv9obz2KgK8dXyqISyV0sXGZ0U47hOA%2FAiigbEMECJxC9aoKp86re5O5prxOlHkcksutSQJzxZRlPZmrOKhsQBF5zEZKybUC0vVjG8PqOnhOq46qyDTDnj5gZBriWCk4DvXrudQnXQmnXblebhAC2cCB6zIbM4PYgGl0elPSgIf3iFEA21aLdHYLHUQuVkpgi02SxFdrG862Y8ymYGMvXDzUmiX8DS5vKZyZlGmsSgQqfLub5RyLNS4zfDiZc9Edzh%2FtCE%2BX8j9k%2FqWB071rcZyMImne1SLkL4GRw4UPHMV3jjwEYpPG5uW5fAEot0aTSJnsGAwHJi2nvF1Y5OIqWziVCQd5NT7t6Q8guOSpgS%2Fa1dSRn8JGGaCD3BPXDyQRG4Bqhu8XrgAp0yy8DMSvvyVXDgJcJTcr1wQ2BvFKf65jqhvmxXUuDpGBlRvV36XvGjQzLi8KAKT2lYOnmxQPGorURSV0NhyTIuIyqOmKTMhQ%2BieEsgOgpc4KBbfDM4B3SIgFljvfHF6cef7qpyLBXAiQcXvg5l3Iunp%2FWv4dH6qFziO%2BL9PbrimQ9RY6MQphEfGUpOmma7KkGzuS8sPUFnCtIYcKCaI9EXo4HlQLgGrBjbiK5EqMj2AKWt9QWcIFMtnVvQVDQV9lXJJqdPVtUQpbh6gCI2Ov1nvZts7yYdsnvRgxiWFOtNJcOMVLn1vgptVi6qrNiFOfEjHCDB3J%2BHDLqUB77YgQGwX%2Fb1eYna3hGKdlqJKIyiE4nSbV8VFgxmxR4b5mVkkeUhMgs5YTi4ja2XZ009xJRHdkfwMi%2BfocaancuO7h%2FMlcLOa0V%2FSw6Dq47CumRQAKhgbOP8t%2BMTjuxjJGhXCY6XpmDDFqWlVYbQ1aDJ5Cptdw4oLbf3Ck%2BdWkVP0LpH7s9XLPXI%2FQX8ws%2Bj2In63IcRvOOo%2BTTjiN%2BlssfRsanW%2B3REVKoavBOAPTXABW4AL7e4NygHdpAKBscmlDh9Jysp4wxbnUNna3L3xBvyE1jyrGIkUHaqQMuxhHElV6oj1picvgL1QEuS5PyZTEaivqh5vUCKJqOuIgPFGESns8kyFk7%2FDxyima3cYxi%2FYOQCj%2F%2B9Ms2Ll%2Bhn4FmKnl7JkGXQGDKDAz9rUGL1TIlBpuJr9Be2JjK6qPzyDg495UxXYF7JY1qKimw9jWjF0iV6DRIqE%2B%2FeWG0J2ofmZTk0mLYVd4GLiFCOoKR0Cg727tWq981InYynvCuKW43aXgEjofVbxIqrm0VL76zlH3gQzWP3R3Bv9oXxclrlO7VVtgBRpSP4hMFWJ8BrUSBCJXC07l40X4jWuvtc42ofNCxtlX2JH6bdeojXgTh5TxOBKEyY5wvBE%2BACh8BtOPNPkApjoxi5h%2B%2FFMQQNpWvZaMH7MKFu5Ax8HoCQdmGkJrtnOiLHwD3uS5y8%2F2xTSDrE%2F4PT1yqtt6vGe8ldMBVMEPd6KwqiYECHDlfbvzphcWP%2BJiZuL5swoWQYlS%2Br7Yu5mNUiGD2retxBi9fl6RDGn4Ti9B1oyYy%2BMP5G87D%2FCpRlvdnuy0PY6RC8BzTA40NXqckQ9TaOUDywkYsudxJzPgyDoAWn%2BB6nEFbaVxxC6UXjJiuDkW9TWq7uRBOJocky9iMfUhGpv%2FdQuVVIuGjYqACbXf8aa%2BPeYNIHZsM7l4s5gAQuUAzRUoT51hnH3EWofXf2vkD5HJJ33vwE%2FaEWp36GHr6GpMaH4AAPuqM5eabH%2FhfG9zcCz4nN6cPinuAw6IHwtvyB%2FdO1toZciBaPh25U0ducR2PI3Zl7mokyLWKkSnEDOg1x5fCsJE9EKhH7HwFNhWMGMS7%2BqxyYsbHHRUDUH4I%2FAheQY7wujJNnFUH4KdCju83riuQeHU9WEqNzjsJFuF%2FdTDAZ%2FK7%2F1WaAU%2BAWymT59pVMT4g2AxcwNa0XEBDdBDpAPvgDIH73R25teeuAF5ime2Ul0OUIiG4GpSAEJeYW9wDTf43wfwHgHLKJoPznkwAAAABJRU5ErkJggg%3D%3D)](https://doi.org/10.1145/3569951.3604396)\n\n# `reportseff`\n\n> A python script for tabular display of slurm efficiency information\n\n![Example](https://github.com/troycomi/reportseff/raw/main/imgs/example.png)\n\n## About\n\n### Motivation\n\nWhether a sys admin or cluster user, knowing how well you are estimating job\nresources can help streamline job scheduling and maximize your priority. If you\nhave ever tried to use `sacct` you probably had some trouble interpreting the\noutput.  While `seff` or `jobstats` can provide detailed summaries, they don't\nscale easily to array jobs or offer a way to see all the jobs from a single\nuser.  `reportseff` aims to fill this role.  Read more about the [motivation\nfor reportseff](https://github.com/troycomi/reportseff/blob/main/ABOUT.md).\n\n### Audience\n\nIf you are running more than one slurm job at a time, you should try\n`reportseff`.  Users of HPC systems can get an idea how well they estimate\nresource usage.  By tuning these values, you can get scheduled earlier and not\nbe penalized for unused allocations.  Since `reportseff` can parse job ids from\nslurm output files, it simplifies the task of identifying which jobs have\nfailed and why. Sys admins can pipe `reportseff` output to identify users with\npoor utilization or produce summaries at the end of a billing cycle.\n\n### Implementation\n\n`reportseff` is a wrapper around `sacct` that provides more complex option\nparsing, simpler options, and cleaner, colored outputs.  All querying is\nperformed in a single call to `sacct` and should have similar performance.\nMulti-node and GPU utilization is acquired from information contained in the\n`AdminComment` field, as generated by `jobstats`.\n\n## Usage\n\n### Installation\n\n`reportseff` runs on python >= 3.6.\nThe only external dependency is click (>= 6.7).\nCalling\n\n```sh\npip install --user reportseff\n# OR\npipx install reportseff\n```\n\nwill create command line bindings and install click.\n\n### Sample Usage\n\nTry `reportseff -u $USER` or just `reportseff` in a directory with some slurm\noutputs.  You may be surprised by your results!\n\n#### Single job\n\nCalling `reportseff` with a single jobid will provide equivalent information to\nseff for that job. `reportseff 24371789` and `reportseff map_fastq_24371789`\nproduce the following output:\n\n```txt\n   JobID      State          Elapsed   CPUEff   MemEff\n24371789    COMPLETED       03:08:03   71.2%    45.7%\n```\n\n#### Single array job\n\nProviding either the raw job id or the array job id will get efficiency\ninformation for a single element of the array job. `reportseff 24220929_421`\nand `reportseff 24221219` generate:\n\n```txt\n       JobID      State          Elapsed    CPUEff   MemEff\n24220929_421    COMPLETED       00:09:34    99.0%    34.6%\n```\n\n#### Array job group\n\nIf the base job id of an array is provided, all elements of the array will\nbe added to the output. `reportseff 24220929`\n\n```txt\n          JobID      State          Elapsed    CPUEff   MemEff\n     24220929_1    COMPLETED       00:10:43    99.2%    33.4%\n    24220929_11    COMPLETED       00:10:10    99.2%    37.5%\n    24220929_21    COMPLETED       00:09:25    98.8%    36.1%\n    24220929_31    COMPLETED       00:09:19    98.9%    33.3%\n    24220929_41    COMPLETED       00:09:23    98.9%    33.3%\n    24220929_51    COMPLETED       00:08:02    98.5%    36.3%\n    ...\n   24220929_951    COMPLETED       00:25:12    99.5%    33.5%\n   24220929_961    COMPLETED       00:39:26    99.7%    34.1%\n   24220929_971    COMPLETED       00:24:11    99.5%    34.2%\n   24220929_981    COMPLETED       00:24:50    99.5%    44.3%\n   24220929_991    COMPLETED       00:13:05    98.7%    33.7%\n```\n\n#### Glob expansion of slurm outputs\n\nBecause slurm output files can act as job id inputs, the following can\nget all seff information for a given job name:\n\n```txt\nslurm_out  \u276f\u276f\u276f reportseff split_ubam_24*\n              JobID      State          Elapsed   CPUEff   MemEff\nsplit_ubam_24342816    COMPLETED       23:30:32   99.9%    4.5%\nsplit_ubam_24342914    COMPLETED       22:40:51   99.9%    4.6%\nsplit_ubam_24393599    COMPLETED       23:43:36   99.4%    4.4%\nsplit_ubam_24393655    COMPLETED       21:36:58   99.3%    4.5%\nsplit_ubam_24418960     RUNNING        02:53:11    ---      ---\nsplit_ubam_24419972     RUNNING        01:26:26    ---      ---\n```\n\n#### No arguments\n\nWithout arguments, reportseff will try to find slurm output files in the\ncurrent directory. Combine with `watch` to monitor job progress:\n`watch -cn 300 reportseff --color --modified-sort`\n\n```txt\n                JobID           State          Elapsed   CPUEff   MemEff\n   split_ubam_24418960          RUNNING        02:56:14    ---      ---\nfastq_to_ubam_24419971          RUNNING        01:29:29    ---      ---\n   split_ubam_24419972          RUNNING        01:29:29    ---      ---\nfastq_to_ubam_24393600         COMPLETED     1-02:00:47   58.3%    41.1%\n    map_fastq_24419330          RUNNING        02:14:53    ---      ---\n    map_fastq_24419323          RUNNING        02:15:24    ---      ---\n    map_fastq_24419324          RUNNING        02:15:24    ---      ---\n    map_fastq_24419322          RUNNING        02:15:24    ---      ---\nmark_adapters_24418437         COMPLETED       01:29:23   99.8%    48.2%\nmark_adapters_24418436         COMPLETED       01:29:03   99.9%    47.4%\n```\n\n#### Filtering slurm output files\n\nOne useful application of `reportseff` is filtering a directory of slurm output\nfiles based on the state or time since running. Additionally, if only the\n`jobid` is specified as a format output, the filenames will be returned in a\npipe-friendly manner:\n\n```txt\nold_runs   \u276f\u276f\u276f reportseff --since d=4 --state Timeout\n\n                 JobID   State      Elapsed  CPUEff   MemEff\ncall_variants_31550458  TIMEOUT    20:05:17  99.5%     0.0%\ncall_variants_31550474  TIMEOUT    20:05:17  99.6%     0.0%\ncall_variants_31550500  TIMEOUT    20:05:08  99.7%     0.0%\nold_runs   \u276f\u276f\u276f reportseff --since d=4 --state Timeout --format jobid\ncall_variants_31550458\ncall_variants_31550474\ncall_variants_31550500\n```\n\nTo find all lines with `output:` in jobs which have timed out or failed\nin the last 4 days:\n\n```sh\nreportseff --since 'd=4' --state TO,F --format jobid | xargs grep output:\n```\n\n### Arguments\n\nJobs can be passed as arguments in the following ways:\n\n- Job ID such as 1234567.  If the id is part of an array job, only the element\nfor that ID will be displayed.  If the id is the base part of an array job,\nall elements in the array will be displayed.\n- Array Job ID such as 1234567\\_89.  Will display only the element specified.\n- Slurm output file.  Format must be BASE\\_%A\\_%a.  BASE is optional as is a\n'.out' suffix.  Unix glob expansions can also be used to filter which jobs\nare displayed.\n- From current directory.  If no argument is supplied, `reportseff` will attempt\nto find slurm output files in the current directory as described above.\nIf a user is provided, instead `reportseff` will show recent jobs for that user.\nIf only `since` is set, all recent jobs for all users will be shown (if allowed).\n- Supplying a directory as a single argument will override the current\ndirectory to check for slurm outputs.\n\n### Options\n\n- `--color/--no-color`: Force color output or not.  By default, will force color\n  output.  With the no-color flag, click will strip color codes for everything\n  besides stdout.\n- `--modified-sort`: Instead of sorting by filename/jobid, sort by last\n  modification time of the slurm output file.\n- `--debug`: Write sacct result to stderr.\n- `--user/-u`: Ignore job arguments and instead query sacct with provided user.\n  Returns all jobs from the last week.\n- `--state/-s`: Output only jobs with states matching one of the provided options.\n  Accepts comma separated values of job codes (e.g. 'R') or full names\n  (e.g. RUNNING).  Case insensitive.\n- `--not-state/-S`: Output only jobs with states not matching any of the provided options.\n  Accepts comma separated values of job codes (e.g. 'R') or full names\n  (e.g. RUNNING).  Case insensitive.\n- `--format`: Provide a comma separated list of columns to produce. Prefixing the\n  argument with `+` adds the specified values to the defaults.  Values can\n  be any valid column name to sacct and the custom efficiency values: TimeEff,\n  cpuEff, MemEff.  Can also optionally set alignment (<, ^, >) and maximum width.\n  Default is center-aligned with a width of the maximum column entry.  For\n  example, `--format 'jobid%>,state%10,memeff%<5'` produces 3 columns with:\n  - JobId aligned right, width set automatically\n  - State with width 10 (center aligned by default)\n  - MemEff aligned left, width 5\n- `--slurm-format`: The filename pattern passed to sbatch during job submission.\n  Overrides the default regex for job id parsing from filenames.  E.g. to match\n  filenames like `123456.out` set `--slurm-format %j.out`.\n- `--since`: Limit results to those occurring after the specified time.  Accepts\n  sacct formats and a comma separated list of key/value pairs.  To get jobs in\n  the last hour and a half, can pass `h=1,m=30`.\n-`--until`: Limit results to those occurring before the specified time. Accepts\n  sacct formats and a comma separated list of key/value pairs.\n  Useful in combination with the 'since' option to query a specific range.\n- `--partition`: Limit results to a specific partition.\n- `--node/-n`: Display information for multi-node jobs; requires additional\n  sacct fields from jobstats.\n- `--node-and-gpu/-g`: Display information for multi-node jobs and GPU information;\n  requires additional sacct fields from jobstats.\n- `--parsable/-p`: Ignore formatting and output as a `|` delimited table.  Useful\n  for piping into more complex analyses.\n\n## Status, Contributions, and Support\n\n`reportseff` is actively maintained but currently feature complete.  If there\nis a function missing, please open an issue to discuss its merit!\n\nBug reports, pull requests, and any feedback are welcome! Prior to submitting\na pull request, be sure any new features have been tested and all unit tests\nare passing. In the cloned repo with\n[poetry](https://github.com/python-poetry/poetry#installation) installed:\n\n```sh\npoetry install\npoetry run pytest\npoetry run pre-commit install\nnox\n```\n\n## Troubleshooting\n\n### I can't install, what is pip?\n\n[pip](https://pip.pypa.io/en/stable/) is the package installer for python.  If\nyou get an error that pip isn't found, look for a python/anaconda/conda module.\n[pipx](https://pypa.github.io/pipx/) ensures that each application is installed\nin an isolated environment.  This resolves issues of dependency versions and\nallows applications to be run from any environment.\n\n### The output has no color with many jobs!\n\nClick should determine if the output supports color display and react automatically\nin a way you expect.  Check that your terminal is setup to display colors and\nthat your pager (probably less) will display color by default.  Some commands,\ne.g. `watch` aren't handled properly even when invoked to support color.  Here\nare some useful settings for your `.bashrc`:\n```\n# have less display colors by default.  Will fix `reportseff` not showing colors\nexport LESS=\"-R\"\n# for watch aliases, include the `--color` option\nwatch -cn 300 reportseff --color --modified-sort\n#      ^                 ^^^^^^^\n```\nYou can always for display of color (or suppress it) with the `--color/--no-color`\noptions\n\n### I get an error about broken pipes when chaining to other commands\n\nPython will report that the consumer of process output has closed the stream\n(i.e. the pipe) while still attempting to write.  Newer versions of click\nshould suppress the warning output, but it seems to not always work.  Besides\nsome extra printing on stderr, the output is not affected.\n\n### My jobs don't have any information about multiple nodes or GPU efficiency\n\nBecause `sacct` doesn't currently record this information, `reportseff`\nretrieves it from a custom field from `jobstats`, developed at Princeton\nUniversity.  If you are outside a Research Computing cluster, that information\nwill likely be absent.  Node-level reporting is only shown for jobs which use\nmultiple nodes or GPUs.  If you need a list of where jobs were run, you can add\n`--format +NodeList`.\n\n## Acknowledgments\n\nThe code for calling sacct and parsing the returning information was taken\nfrom [Slurmee](https://github.com/PrincetonUniversity/slurmee).\n\nStyle and tooling from [hypermodern python](https://cjolowicz.github.io/posts/hypermodern-python-01-setup/)\n\nCode review provided from a [repo-review](https://researchcomputing.princeton.edu/services/repo-review-consultations)\nwhich vastly improved this readme.\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Tablular seff output",
    "version": "2.7.6",
    "project_urls": {
        "Homepage": "https://github.com/troycomi/reportseff",
        "Repository": "https://github.com/troycomi/reportseff"
    },
    "split_keywords": [
        "slurm",
        "seff"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "325a390847605abb7a8739c58dbbe14b1d9ecab8453eb3fbbba7e64c5d45a4b2",
                "md5": "9a3d0364390e81f61db0532b1fa37090",
                "sha256": "298804bd5aea18dcfd3266e28b9dac3b18a68c7265b82bb46c017792c6bd7b2a"
            },
            "downloads": -1,
            "filename": "reportseff-2.7.6-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "9a3d0364390e81f61db0532b1fa37090",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.7.0,<4.0",
            "size": 28698,
            "upload_time": "2023-09-14T15:51:38",
            "upload_time_iso_8601": "2023-09-14T15:51:38.700474Z",
            "url": "https://files.pythonhosted.org/packages/32/5a/390847605abb7a8739c58dbbe14b1d9ecab8453eb3fbbba7e64c5d45a4b2/reportseff-2.7.6-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "154e3999f38c1021ad7d06a2980b1dea53a50489d481f398dfb3e0e8611b988a",
                "md5": "6c6c08e03eb5f929eb0fb44cf9caea7a",
                "sha256": "1d62e406059c5ff0f8bb5eb4949de8cc322299418b5bfe75e0a27bd0f8c12197"
            },
            "downloads": -1,
            "filename": "reportseff-2.7.6.tar.gz",
            "has_sig": false,
            "md5_digest": "6c6c08e03eb5f929eb0fb44cf9caea7a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.7.0,<4.0",
            "size": 34976,
            "upload_time": "2023-09-14T15:51:40",
            "upload_time_iso_8601": "2023-09-14T15:51:40.102299Z",
            "url": "https://files.pythonhosted.org/packages/15/4e/3999f38c1021ad7d06a2980b1dea53a50489d481f398dfb3e0e8611b988a/reportseff-2.7.6.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-09-14 15:51:40",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "troycomi",
    "github_project": "reportseff",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "reportseff"
}
        
Elapsed time: 0.13841s