SciCommander
============
[](https://github.com/samuell/scicommander/actions/workflows/python-run-tests.yml)
This is a small tool that executes single shell commands in a scientifically
more reproducible and robust way, by doing the following things:
- Auditing: Creating an audit log of most output files
- Caching: Skipping executions where output files already exist
- (Coming soon): Atomic writes - Writes files to a temporary location until
command is finished
## Requirements
- A unix like operating system such as Linux or Mac OS (On Windows you can use
[WSL](https://learn.microsoft.com/en-us/windows/wsl/about) or [MSYS2](https://www.msys2.org/))
- Python 3.6 or higher
- A bash shell
- For graph plotting for the HTML report, you need
[GraphViz](https://graphviz.org/) and its `dot` command.
## Installation
```bash
pip install scicommander
```
This will install the `scicmd` command into your `PATH` variable, so that it
should be executable from your shell.
## Usage
To view the options of the `scicmd` command, execute:
```bash
scicmd -h
```
To get the benefits from SciCommander, do the following:
1. Prepend all your shell commands with the `scicmd -c` command.
2. Wrap the command itself in quotes, either `""` or `''`. This is not strictly
required always, but will be required for example if using redirection using
`>` or piping with `|` (Alternatively one can just add quotes around those).
3. Wrap definitions of input fields in `{i:INPATH}` and output files in
`{o:OUTPATH}` for output paths.
4. You can also just prepend input paths with `i:` and output paths with `o:`,
but this is a slightly less robust method, that might fail to wrap the
correct number of characters in some situations.
5. Then run your script as usual.
Now you will notice that if you run your script again, it will skip all
commands that have already finished and produced output files.
You will also have files with the extension `.au.json` for every output that
you decorated with the syntax above.
To convert such an audit report into a nice HTML-report, run the following:
```bash
scicmd --to-html <audit-file>
```
## Example
To demonstrate how you can use SciCommander, imagine that you want to write the
following little bioinformatics pipeline, that writes some DNA and converts its
reverse complement, as a shell script, `my_pipeline.sh`:
```bash
#!/bin/bash
# Create a fasta file with some DNA
echo AAAGCCCGTGGGGGACCTGTTC > o:dna.fa
# Compute the complement sequence
cat i:dna.fa | tr ACGT TGCA > o:dna.compl.fa
# Reverse the DNA string
cat i:dna.compl.fa | rev > o:dna.compl.rev.fa
```
Now, to make the commands run through SciCommander, change the syntax in the
script like this:
```bash
#!/bin/bash
# Create a fasta file with some DNA
scicmd -c echo AAAGCCCGTGGGGGACCTGTTC '>' o:dna.fa
# Compute the complement sequence
scicmd -c cat i:dna.fa '|' tr ACGT TGCA '>' o:dna.compl.fa
# Reverse the DNA string
scicmd -c cat i:dna.compl.fa '|' rev '>' o:dna.compl.rev.fa
```
Notice how all input paths are prepended with `i:` and output paths with `o:`,
and also that we had to wrap all pipe characters (`|`) and redirection
characters (`>`) in quotes. This is so that they are not grabbed by bash
immediately, but instead passed with the command to SciCommander, and executed
as part of its execution.
Now you can run the script as usual, e.g. with:
```bash
bash my_pipeline.sh
```
Now, the files in your folder will look like this, if you list them with `ls -tr`:
```bash
my_pipeline.sh
dna.fa.au.json
dna.fa
dna.compl.fa.au.json
dna.compl.fa
dna.compl.rev.fa.au.json
dna.compl.rev.fa
```
Now, you see that the last `.au.json` file is `dna.compl.rev.fa.au.json`.
To convert this file to HTML and view it in a browser, you can do:
```bash
scicmd --to-html dna.compl.rev.fa.au.json
```
Then you will see [an HTML page like this](https://htmlpreview.github.io/?https://github.com/samuell/scicommander/blob/main/python/examples/dna.compl.rev.fa.au.html)
## Experimental: Bash integration
There is very early and experimental support for running SciCommander commands
in bash, without needing to run them via the `scicmd -c` command.
To do this, start the SciCommander shell with the following command:
```bash
scishell
```
And then, you can run the example commands above as follows:
```bash
# Create a fasta file with some DNA
echo AAAGCCCGTGGGGGACCTGTTC > o:dna.fa
# Compute the complement sequence
cat i:dna.fa | tr ACGT TGCA > o:dna.compl.fa
# Reverse the DNA string
cat i:dna.compl.fa | rev > o:dna.compl.rev.fa
```
In other words, only the `i:` and `o:` markers are now needed, and no extra
syntax.
## Notes
[1] Although Nextflow and Snakemake already take care of some of the benefits,
such as atomic writes, SciCommander adds additional features such as detailed
per-output audit logs.
Raw data
{
"_id": null,
"home_page": "https://github.com/samuell/scicommander",
"name": "scicommander",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "reproducibility,shell,bash",
"author": "Samuel Lampa",
"author_email": "samuel.lampa@scilifelab.se",
"download_url": "https://files.pythonhosted.org/packages/24/5c/ccb3652a7e9f6643e2564fe60becb3e3b2e396825ce46d521c99a29efdb8/scicommander-0.3.3.tar.gz",
"platform": null,
"description": "SciCommander\n============\n\n[](https://github.com/samuell/scicommander/actions/workflows/python-run-tests.yml)\n\nThis is a small tool that executes single shell commands in a scientifically\nmore reproducible and robust way, by doing the following things:\n\n- Auditing: Creating an audit log of most output files\n- Caching: Skipping executions where output files already exist\n- (Coming soon): Atomic writes - Writes files to a temporary location until\n command is finished\n\n## Requirements\n\n- A unix like operating system such as Linux or Mac OS (On Windows you can use\n [WSL](https://learn.microsoft.com/en-us/windows/wsl/about) or [MSYS2](https://www.msys2.org/))\n- Python 3.6 or higher\n- A bash shell\n- For graph plotting for the HTML report, you need\n [GraphViz](https://graphviz.org/) and its `dot` command.\n\n## Installation\n\n```bash\npip install scicommander\n```\n\nThis will install the `scicmd` command into your `PATH` variable, so that it\nshould be executable from your shell.\n\n## Usage\n\nTo view the options of the `scicmd` command, execute:\n\n```bash\nscicmd -h\n```\n\nTo get the benefits from SciCommander, do the following:\n\n1. Prepend all your shell commands with the `scicmd -c` command.\n2. Wrap the command itself in quotes, either `\"\"` or `''`. This is not strictly\n required always, but will be required for example if using redirection using\n `>` or piping with `|` (Alternatively one can just add quotes around those).\n3. Wrap definitions of input fields in `{i:INPATH}` and output files in\n `{o:OUTPATH}` for output paths.\n4. You can also just prepend input paths with `i:` and output paths with `o:`,\n but this is a slightly less robust method, that might fail to wrap the\n correct number of characters in some situations.\n5. Then run your script as usual.\n\nNow you will notice that if you run your script again, it will skip all\ncommands that have already finished and produced output files.\n\nYou will also have files with the extension `.au.json` for every output that\nyou decorated with the syntax above.\n\nTo convert such an audit report into a nice HTML-report, run the following:\n\n```bash\nscicmd --to-html <audit-file>\n\n```\n\n## Example\n\nTo demonstrate how you can use SciCommander, imagine that you want to write the\nfollowing little bioinformatics pipeline, that writes some DNA and converts its\nreverse complement, as a shell script, `my_pipeline.sh`:\n\n```bash\n#!/bin/bash\n\n# Create a fasta file with some DNA\necho AAAGCCCGTGGGGGACCTGTTC > o:dna.fa\n# Compute the complement sequence\ncat i:dna.fa | tr ACGT TGCA > o:dna.compl.fa\n# Reverse the DNA string\ncat i:dna.compl.fa | rev > o:dna.compl.rev.fa\n```\nNow, to make the commands run through SciCommander, change the syntax in the\nscript like this:\n\n```bash\n#!/bin/bash\n\n# Create a fasta file with some DNA\nscicmd -c echo AAAGCCCGTGGGGGACCTGTTC '>' o:dna.fa\n# Compute the complement sequence\nscicmd -c cat i:dna.fa '|' tr ACGT TGCA '>' o:dna.compl.fa\n# Reverse the DNA string\nscicmd -c cat i:dna.compl.fa '|' rev '>' o:dna.compl.rev.fa\n```\n\nNotice how all input paths are prepended with `i:` and output paths with `o:`,\nand also that we had to wrap all pipe characters (`|`) and redirection\ncharacters (`>`) in quotes. This is so that they are not grabbed by bash\nimmediately, but instead passed with the command to SciCommander, and executed\nas part of its execution.\n\nNow you can run the script as usual, e.g. with:\n\n```bash\nbash my_pipeline.sh\n```\n\nNow, the files in your folder will look like this, if you list them with `ls -tr`:\n\n```bash\nmy_pipeline.sh\ndna.fa.au.json\ndna.fa\ndna.compl.fa.au.json\ndna.compl.fa\ndna.compl.rev.fa.au.json\ndna.compl.rev.fa\n```\n\nNow, you see that the last `.au.json` file is `dna.compl.rev.fa.au.json`.\n\nTo convert this file to HTML and view it in a browser, you can do:\n\n```bash\nscicmd --to-html dna.compl.rev.fa.au.json\n```\n\nThen you will see [an HTML page like this](https://htmlpreview.github.io/?https://github.com/samuell/scicommander/blob/main/python/examples/dna.compl.rev.fa.au.html)\n\n## Experimental: Bash integration\n\nThere is very early and experimental support for running SciCommander commands\nin bash, without needing to run them via the `scicmd -c` command.\n\nTo do this, start the SciCommander shell with the following command:\n\n```bash\nscishell\n```\n\nAnd then, you can run the example commands above as follows:\n\n```bash\n# Create a fasta file with some DNA\necho AAAGCCCGTGGGGGACCTGTTC > o:dna.fa\n# Compute the complement sequence\ncat i:dna.fa | tr ACGT TGCA > o:dna.compl.fa\n# Reverse the DNA string\ncat i:dna.compl.fa | rev > o:dna.compl.rev.fa\n```\n\nIn other words, only the `i:` and `o:` markers are now needed, and no extra\nsyntax.\n\n## Notes\n\n[1] Although Nextflow and Snakemake already take care of some of the benefits,\nsuch as atomic writes, SciCommander adds additional features such as detailed\nper-output audit logs.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A small library for executing shell commands in a reproducible way.",
"version": "0.3.3",
"project_urls": {
"Homepage": "https://github.com/samuell/scicommander"
},
"split_keywords": [
"reproducibility",
"shell",
"bash"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4ae1fc19c071f8a76f099169a582091a97eeb54a540e0e6c629289ca7d0d8ab7",
"md5": "038230edcc91898657dcc70edbcbaddd",
"sha256": "ecaa85a3dbc042b07dbba483f2041beaf3e9f804e59016a80e88cb5d15ed2fd7"
},
"downloads": -1,
"filename": "scicommander-0.3.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "038230edcc91898657dcc70edbcbaddd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 7204,
"upload_time": "2023-11-09T17:19:23",
"upload_time_iso_8601": "2023-11-09T17:19:23.746922Z",
"url": "https://files.pythonhosted.org/packages/4a/e1/fc19c071f8a76f099169a582091a97eeb54a540e0e6c629289ca7d0d8ab7/scicommander-0.3.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "245cccb3652a7e9f6643e2564fe60becb3e3b2e396825ce46d521c99a29efdb8",
"md5": "6140b72e878d0554c46a7ec4d20a25aa",
"sha256": "903d401392ba5fd82270914c825bf2bb09b51559ed23ee915e2609f5b8c06528"
},
"downloads": -1,
"filename": "scicommander-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "6140b72e878d0554c46a7ec4d20a25aa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6930,
"upload_time": "2023-11-09T17:19:25",
"upload_time_iso_8601": "2023-11-09T17:19:25.509911Z",
"url": "https://files.pythonhosted.org/packages/24/5c/ccb3652a7e9f6643e2564fe60becb3e3b2e396825ce46d521c99a29efdb8/scicommander-0.3.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-09 17:19:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "samuell",
"github_project": "scicommander",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "scicommander"
}
JSON
