# duct
[](https://codecov.io/gh/con/duct)
[](https://badge.fury.io/py/con-duct)
## Installation
pip install con-duct
## Quickstart
Try it out!
duct --sample-interval 0.5 --report-interval 1 test/data/test_script.py --duration 3 --memory-size=1000
`duct` is most useful when the report-interval is less than the duration of the script.
## Summary:
A process wrapper script that monitors the execution of a command.
<!-- BEGIN HELP -->
```shell
>duct --help
usage: duct [-h] [--version] [-p OUTPUT_PREFIX]
[--summary-format SUMMARY_FORMAT] [--colors] [--clobber]
[-l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}] [-q]
[--sample-interval SAMPLE_INTERVAL]
[--report-interval REPORT_INTERVAL] [--fail-time FAIL_TIME]
[-c {all,none,stdout,stderr}] [-o {all,none,stdout,stderr}]
[-t {all,system-summary,processes-samples}]
command [command_args ...] ...
duct is a lightweight wrapper that collects execution data for an arbitrary
command. Execution data includes execution time, system information, and
resource usage statistics of the command and all its child processes. It is
intended to simplify the problem of recording the resources necessary to
execute a command, particularly in an HPC environment.
Resource usage is determined by polling (at a sample-interval).
During execution, duct produces a JSON lines (see https://jsonlines.org) file
with one data point recorded for each report (at a report-interval).
environment variables:
Many duct options can be configured by environment variables (which are
overridden by command line options).
DUCT_LOG_LEVEL: see --log-level
DUCT_OUTPUT_PREFIX: see --output-prefix
DUCT_SUMMARY_FORMAT: see --summary-format
DUCT_SAMPLE_INTERVAL: see --sample-interval
DUCT_REPORT_INTERVAL: see --report-interval
DUCT_CAPTURE_OUTPUTS: see --capture-outputs
positional arguments:
command [command_args ...]
The command to execute, along with its arguments.
command_args Arguments for the command.
options:
-h, --help show this help message and exit
--version show program's version number and exit
-p OUTPUT_PREFIX, --output-prefix OUTPUT_PREFIX
File string format to be used as a prefix for the
files -- the captured stdout and stderr and the
resource usage logs. The understood variables are
{datetime}, {datetime_filesafe}, and {pid}. Leading
directories will be created if they do not exist. You
can also provide value via DUCT_OUTPUT_PREFIX env
variable. (default:
.duct/logs/{datetime_filesafe}-{pid}_)
--summary-format SUMMARY_FORMAT
Output template to use when printing the summary
following execution. Accepts custom conversion flags:
!S: Converts filesizes to human readable units, green
if measured, red if None. !E: Colors exit code, green
if falsey, red if truthy, and red if None. !X: Colors
green if truthy, red if falsey. !N: Colors green if
not None, red if None (default: Summary: Exit Code:
{exit_code!E} Command: {command} Log files location:
{logs_prefix} Wall Clock Time: {wall_clock_time:.3f}
sec Memory Peak Usage (RSS): {peak_rss!S} Memory
Average Usage (RSS): {average_rss!S} Virtual Memory
Peak Usage (VSZ): {peak_vsz!S} Virtual Memory Average
Usage (VSZ): {average_vsz!S} Memory Peak Percentage:
{peak_pmem:.2f!N}% Memory Average Percentage:
{average_pmem:.2f!N}% CPU Peak Usage:
{peak_pcpu:.2f!N}% Average CPU Usage:
{average_pcpu:.2f!N}% )
--colors Use colors in duct output. (default: False)
--clobber Replace log files if they already exist. (default:
False)
-l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}, --log_level {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}
Level of log output to stderr, use NONE to entirely
disable. (default: INFO)
-q, --quiet [deprecated, use log level NONE] Disable duct logging
output (to stderr) (default: False)
--sample-interval SAMPLE_INTERVAL, --s-i SAMPLE_INTERVAL
Interval in seconds between status checks of the
running process. Sample interval must be less than or
equal to report interval, and it achieves the best
results when sample is significantly less than the
runtime of the process. (default: 1.0)
--report-interval REPORT_INTERVAL, --r-i REPORT_INTERVAL
Interval in seconds at which to report aggregated
data. (default: 60.0)
--fail-time FAIL_TIME, --f-t FAIL_TIME
If command fails in less than this specified time,
duct would remove logs. Set to 0 if you would like to
keep logs for a failing command regardless of its run
time. Set to negative (e.g. -1) if you would like to
not keep logs for any failing command. (default: 3.0)
-c {all,none,stdout,stderr}, --capture-outputs {all,none,stdout,stderr}
Record stdout, stderr, all, or none to log files. You
can also provide value via DUCT_CAPTURE_OUTPUTS env
variable. (default: all)
-o {all,none,stdout,stderr}, --outputs {all,none,stdout,stderr}
Print stdout, stderr, all, or none to stdout/stderr
respectively. (default: all)
-t {all,system-summary,processes-samples}, --record-types {all,system-summary,processes-samples}
Record system-summary, processes-samples, or all
(default: all)
```
<!-- END HELP -->
# con-duct suite
In addition to `duct`, this project also includes a set of optional helpers under the `con-duct` command.
These helpers may use 3rd party python libraries.
## Installation
pip install con-duct[all]
## Extras Helptext
<!-- BEGIN EXTRAS HELP -->
```shell
>con-duct --help
usage: con-duct <command> [options]
A suite of commands to manage or manipulate con-duct logs.
positional arguments:
{pp,plot} Available subcommands
pp Pretty print a JSON log.
plot Plot resource usage for an execution.
options:
-h, --help show this help message and exit
```
<!-- END EXTRAS HELP -->
## FAQs
### git-annex add keeps adding duct logs directly into git
By default, [git-annex](https://git-annex.branchable.com/) treats all dotfiles, and files under directories starting with a `.` as "small" regardless of `annex.largefiles` setting [[ref: an issue describing the logic](https://git-annex.branchable.com/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/?updated#comment-efc1f2aa8f46e88a8be9837a56cfa6f7)].
It is necessary to set `annex.dotfiles` variable to `true` to make git-annex treat them as regular files and thus subject to `annex.largefiles` setting [[ref: git-annex config](https://git-annex.branchable.com/git-annex-config/)].
Could be done the repository (not just specific clone, but any instance since records in `git-annex` branch) wide using `git annex config --set annex.dotfiles true`.
Raw data
{
"_id": null,
"home_page": "https://github.com/con/duct/",
"name": "con-duct",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "command-line, cpu, memory, metrics, output-capture, provenance, time, usage",
"author": "Austin Macdonald",
"author_email": "austin@dartmouth.edu",
"download_url": "https://files.pythonhosted.org/packages/95/8e/5c5887b6bf807989214b3c1158a9c18179b93e16959bc9ffb68610a72fc9/con_duct-0.9.0.tar.gz",
"platform": null,
"description": "# duct\n\n[](https://codecov.io/gh/con/duct)\n[](https://badge.fury.io/py/con-duct)\n\n## Installation\n\n pip install con-duct\n\n## Quickstart\n\nTry it out!\n\n duct --sample-interval 0.5 --report-interval 1 test/data/test_script.py --duration 3 --memory-size=1000\n\n`duct` is most useful when the report-interval is less than the duration of the script.\n\n## Summary:\n\nA process wrapper script that monitors the execution of a command.\n\n<!-- BEGIN HELP -->\n```shell\n>duct --help\n\nusage: duct [-h] [--version] [-p OUTPUT_PREFIX]\n [--summary-format SUMMARY_FORMAT] [--colors] [--clobber]\n [-l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}] [-q]\n [--sample-interval SAMPLE_INTERVAL]\n [--report-interval REPORT_INTERVAL] [--fail-time FAIL_TIME]\n [-c {all,none,stdout,stderr}] [-o {all,none,stdout,stderr}]\n [-t {all,system-summary,processes-samples}]\n command [command_args ...] ...\n\nduct is a lightweight wrapper that collects execution data for an arbitrary\ncommand. Execution data includes execution time, system information, and\nresource usage statistics of the command and all its child processes. It is\nintended to simplify the problem of recording the resources necessary to\nexecute a command, particularly in an HPC environment.\n\nResource usage is determined by polling (at a sample-interval).\nDuring execution, duct produces a JSON lines (see https://jsonlines.org) file\nwith one data point recorded for each report (at a report-interval).\n\nenvironment variables:\n Many duct options can be configured by environment variables (which are\n overridden by command line options).\n\n DUCT_LOG_LEVEL: see --log-level\n DUCT_OUTPUT_PREFIX: see --output-prefix\n DUCT_SUMMARY_FORMAT: see --summary-format\n DUCT_SAMPLE_INTERVAL: see --sample-interval\n DUCT_REPORT_INTERVAL: see --report-interval\n DUCT_CAPTURE_OUTPUTS: see --capture-outputs\n\npositional arguments:\n command [command_args ...]\n The command to execute, along with its arguments.\n command_args Arguments for the command.\n\noptions:\n -h, --help show this help message and exit\n --version show program's version number and exit\n -p OUTPUT_PREFIX, --output-prefix OUTPUT_PREFIX\n File string format to be used as a prefix for the\n files -- the captured stdout and stderr and the\n resource usage logs. The understood variables are\n {datetime}, {datetime_filesafe}, and {pid}. Leading\n directories will be created if they do not exist. You\n can also provide value via DUCT_OUTPUT_PREFIX env\n variable. (default:\n .duct/logs/{datetime_filesafe}-{pid}_)\n --summary-format SUMMARY_FORMAT\n Output template to use when printing the summary\n following execution. Accepts custom conversion flags:\n !S: Converts filesizes to human readable units, green\n if measured, red if None. !E: Colors exit code, green\n if falsey, red if truthy, and red if None. !X: Colors\n green if truthy, red if falsey. !N: Colors green if\n not None, red if None (default: Summary: Exit Code:\n {exit_code!E} Command: {command} Log files location:\n {logs_prefix} Wall Clock Time: {wall_clock_time:.3f}\n sec Memory Peak Usage (RSS): {peak_rss!S} Memory\n Average Usage (RSS): {average_rss!S} Virtual Memory\n Peak Usage (VSZ): {peak_vsz!S} Virtual Memory Average\n Usage (VSZ): {average_vsz!S} Memory Peak Percentage:\n {peak_pmem:.2f!N}% Memory Average Percentage:\n {average_pmem:.2f!N}% CPU Peak Usage:\n {peak_pcpu:.2f!N}% Average CPU Usage:\n {average_pcpu:.2f!N}% )\n --colors Use colors in duct output. (default: False)\n --clobber Replace log files if they already exist. (default:\n False)\n -l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}, --log_level {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}\n Level of log output to stderr, use NONE to entirely\n disable. (default: INFO)\n -q, --quiet [deprecated, use log level NONE] Disable duct logging\n output (to stderr) (default: False)\n --sample-interval SAMPLE_INTERVAL, --s-i SAMPLE_INTERVAL\n Interval in seconds between status checks of the\n running process. Sample interval must be less than or\n equal to report interval, and it achieves the best\n results when sample is significantly less than the\n runtime of the process. (default: 1.0)\n --report-interval REPORT_INTERVAL, --r-i REPORT_INTERVAL\n Interval in seconds at which to report aggregated\n data. (default: 60.0)\n --fail-time FAIL_TIME, --f-t FAIL_TIME\n If command fails in less than this specified time,\n duct would remove logs. Set to 0 if you would like to\n keep logs for a failing command regardless of its run\n time. Set to negative (e.g. -1) if you would like to\n not keep logs for any failing command. (default: 3.0)\n -c {all,none,stdout,stderr}, --capture-outputs {all,none,stdout,stderr}\n Record stdout, stderr, all, or none to log files. You\n can also provide value via DUCT_CAPTURE_OUTPUTS env\n variable. (default: all)\n -o {all,none,stdout,stderr}, --outputs {all,none,stdout,stderr}\n Print stdout, stderr, all, or none to stdout/stderr\n respectively. (default: all)\n -t {all,system-summary,processes-samples}, --record-types {all,system-summary,processes-samples}\n Record system-summary, processes-samples, or all\n (default: all)\n\n```\n<!-- END HELP -->\n\n# con-duct suite\n\nIn addition to `duct`, this project also includes a set of optional helpers under the `con-duct` command.\nThese helpers may use 3rd party python libraries.\n\n## Installation\n\n pip install con-duct[all]\n\n## Extras Helptext\n\n<!-- BEGIN EXTRAS HELP -->\n```shell\n>con-duct --help\n\nusage: con-duct <command> [options]\n\nA suite of commands to manage or manipulate con-duct logs.\n\npositional arguments:\n {pp,plot} Available subcommands\n pp Pretty print a JSON log.\n plot Plot resource usage for an execution.\n\noptions:\n -h, --help show this help message and exit\n\n```\n<!-- END EXTRAS HELP -->\n\n## FAQs\n\n### git-annex add keeps adding duct logs directly into git\n\nBy default, [git-annex](https://git-annex.branchable.com/) treats all dotfiles, and files under directories starting with a `.` as \"small\" regardless of `annex.largefiles` setting [[ref: an issue describing the logic](https://git-annex.branchable.com/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/?updated#comment-efc1f2aa8f46e88a8be9837a56cfa6f7)].\nIt is necessary to set `annex.dotfiles` variable to `true` to make git-annex treat them as regular files and thus subject to `annex.largefiles` setting [[ref: git-annex config](https://git-annex.branchable.com/git-annex-config/)].\nCould be done the repository (not just specific clone, but any instance since records in `git-annex` branch) wide using `git annex config --set annex.dotfiles true`.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Runs a not-so-simple command and collects resource usage metrics",
"version": "0.9.0",
"project_urls": {
"Bug Tracker": "https://github.com/con/duct/issues",
"Homepage": "https://github.com/con/duct/",
"Source Code": "https://github.com/con/duct/"
},
"split_keywords": [
"command-line",
" cpu",
" memory",
" metrics",
" output-capture",
" provenance",
" time",
" usage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ccd7594f5fee5bded72e98001bf4296528529bf702b354c3db019b67f61448dc",
"md5": "577295e3cc2992e65d4fdeafae373db6",
"sha256": "0dcd568acc4ae7f18c876bd42da264cfeec7dcd9d20b1ef6b605250cc73a0009"
},
"downloads": -1,
"filename": "con_duct-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "577295e3cc2992e65d4fdeafae373db6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 18947,
"upload_time": "2024-12-03T21:17:50",
"upload_time_iso_8601": "2024-12-03T21:17:50.142950Z",
"url": "https://files.pythonhosted.org/packages/cc/d7/594f5fee5bded72e98001bf4296528529bf702b354c3db019b67f61448dc/con_duct-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "958e5c5887b6bf807989214b3c1158a9c18179b93e16959bc9ffb68610a72fc9",
"md5": "a052ce7ca96c4a5a861db1320b99b297",
"sha256": "23d9d3f437f19a188ce55027482bdbce1ab76db411c6b7697377764357368fb3"
},
"downloads": -1,
"filename": "con_duct-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "a052ce7ca96c4a5a861db1320b99b297",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 45996,
"upload_time": "2024-12-03T21:17:51",
"upload_time_iso_8601": "2024-12-03T21:17:51.765381Z",
"url": "https://files.pythonhosted.org/packages/95/8e/5c5887b6bf807989214b3c1158a9c18179b93e16959bc9ffb68610a72fc9/con_duct-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-03 21:17:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "con",
"github_project": "duct",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "con-duct"
}
JSON
