# FAIR Data Pipeline Command Line Interface
[](https://pypi.org/project/fair-cli/) [](https://pypi.org/project/fair-cli/) [](https://zenodo.org/badge/latestdoi/377398464)  [](https://bestpractices.coreinfrastructure.org/projects/5411)
[](https://github.com/FAIRDataPipeline/FAIR-CLI/actions/workflows/fair-cli.yaml) [](https://codecov.io/gh/FAIRDataPipeline/FAIR-CLI) [](https://sonarcloud.io/summary/new_code?id=FAIRDataPipeline_FAIR-CLI)
FAIR-CLI forms the main interface for synchronising changes between local and shared remote FAIR Data Pipeline registries, it is also used to instantiate model runs/data submissions to the pipeline. Full documentation of the FAIR Data Pipeline can be found on the project [website](https://www.fairdatapipeline.org/).
## Installation
The package is installed using Pip:
```sh
pip install fair-cli
```
To enable tab completion you need to modify your shell:
### Bash
```
_FAIR_COMPLETE=bash_source fair > ~/.config/.fair-complete.bash
echo '. ~/.config/.fair-complete.bash' >> ~/.bashrc
```
### zsh
```
_FAIR_COMPLETE=zsh_source fair > ~/.fair-complete.zsh
echo '. ~/.fair-complete.zsh' >> ~/.bashrc
```
### Fish
```
_FAIR_COMPLETE=bash_source fair > ~/.config/fish/.fair-complete.fish
echo '. ~/.config/fish/.fair-complete.fish' >> ~/.bashrc
```
## Uninstallation
To uninstall the CLI run:
```
fair purge --all
pip uninstall fair
```
## The User Configuration File
Job runs are configured via `config.yaml` files. Upon initialisation of a project, FAIR-CLI automatically generates a starter configuration file with all requirements in place. To execute a process (e.g. perform a model run from a compiled binary/script) an additional key of either `script` or `script_path` must be provided. Alternatively the command `fair run bash` can be used to append the key and run a command directly.
By default the shell used to execute a process is `sh` or `batch` for UNIX and Windows systems respectively. This can be overwritten by assigning the optional `shell` key with one of the following values (where `{0}` is the script file):
| **Shell** | **Command** |
| ------------ | ------------------------------- |
| `bash` | `bash -eo pipefail {0}` |
| `java` | `java {0}` |
| `julia` | `julia {0}` |
| `powershell` | `powershell -command ". '{0}'"` |
| `pwsh` | `pwsh -command ". '{0}'"` |
| `python2` | `python2 {0}` |
| `python3` | `python3 {0}` |
| `python` | `python {0}` |
| `R` | `R -f {0}` |
| `sh` | `sh -e {0}` |
| `batch` | `{0}` |
A full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).
## Available Commands
### `init`
Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.
#### Custom CLI Configuration
After setup is complete, the current CLI configuration can also be saved using the command:
```
fair init --export
```
the created file can then be re-read at a later point during setup. Alternatively, if creating a configuration from scratch the YAML file should contain the following information:
```yaml
namespaces:
input: testing
output: testing
registries:
local:
data_store: /path/to/local/data_store/,
directory: /local/registry/install/directory
uri: http://127.0.0.1:8000/api/
origin:
data_store: /remote/registry/data/store/path/
token: /path/to/remote/token
uri: https://data.fairdatapipeline.org/api/'
user:
email: 'test@noreply',
family_name: 'Test'
given_names: 'Interface'
orcid: None,
uuid: '2ddb2358-84bf-43ff-b2aa-3ac7dc3b49f1'
git:
local_repo: /local/repo/path
remote: origin
description: Testing Project
```
this file is then read during the initialisation:
```sh
fair init --using <cli-config.yaml file>
```
For integration into a CI workflow, the setup can be skipped by running:
```sh
fair init --ci
```
which will create temporary directories for some of the required location paths.
### `run`
The purpose of `run` is to execute a model/submission run and submit results to the local registry. Outputs of a run will be stored within the `coderun` folder in the directory specified under the `data_store` tag in the `config.yaml`, by default this is `$HOME/.fair/data/coderun`.
```sh
fair run
```
If you wish to use an alternative `config.yaml` then specify it as an additional argument:
```sh
fair run /path/to/config.yaml
```
You can also launch a bash command directly, this will be automatically written into the `config.yaml`:
```sh
fair run --script 'echo "Hello World"'
```
note the command itself must be quoted as it is a single argument.
By default the CLI will not allow the user to perform a run if the state of the analysis repository is such that it is behind the git remote, or contains uncommitted changes. To override this behaviour use the `--dirty` flag.
### `pull`
The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
```sh
fair pull /path/to/config.yaml
```
### `status`
This command displays objects which are awaiting staging or have been staged behaving in a manner similar to `git status`:
```sh
fair status
```
staged changes are displayed in green, and unstaged in red.
### `add`
Before changes can be pushed to the remote registry they must be staged. This command allows you to stage objects displayed when running `fair status` so that they can be sent to the remote registry. Data products are displayed and staged in the form `namespace:data_product_name@version`:
```sh
fair add my_namespace:data_object@v0.1.0
```
### `push`
The `push` command will push any staged data products to the remote registry:
```sh
fair push
```
### `purge`
The `purge` command removes setup of the current project so it can bereinitialised:
```sh
fair purge
```
To remove all configurations entirely (including those global to all projects) run:
```sh
fair purge --global
```
To remove the data directory itself run:
```sh
fair purge --data
```
**WARNING**: This is not recommended as the registry may still have entries pointing to this location!
Finally to remove everything run:
```sh
fair purge --all
```
this will remove the current repository `.fair` folder and the global FAIR directory which also contains the local registry.
You can skip any confirmation messages by running:
```sh
fair purge --yes
```
### `registry`
By default the CLI will launch the registry whenever a synchronisation or run is called. The server will only be halted once all ongoing CLI processes (in the case of multiple parallel calls) have been completed.
However the user may also specify a manual launch that will override this behaviour, instead leaving the server running constantly allowing them to view the registry in the browser.
The commands:
```sh
fair registry start
```
and
```sh
fair registry stop
```
will launch and halt the server respectively.
The registry can be installed using the CLI as well by running:
```sh
fair registry install
```
with the additional options to specify the installation location, and the data registry repository tag to install from:
```sh
fair registry install --directory ~/.fair/my_registry --version v1.0-rc5
```
### `log`
Runs are logged locally within the local FAIR repository. A full list of runs is shown by running:
```sh
fair log
```
This will present a list of runs in a summary analogous to a `git log` call:
```yaml
run 0db35c20946a1ebeaafdc3b30103cd74a57eb6b6
Author: Joe Bloggs <jbloggs@noreply.uk>
Date: Wed Jun 30 09:09:30 2021
```
| **NOTE** |
| ----------------------------------------------------------------------------------------------------------------------------------- |
| The SHA for a job is *not* related to a registry code run identifier as multiple code runs can be executed within a single job. |
### `view`
To view the `stdout` of a run given its SHA as shown by running `fair log` use the command:
```sh
fair view <sha>
```
you do not need to specify the full SHA but rather the first few unique characters.
## Template Variables
Within the `config.yaml` file, template variables can be specified by using the notation `${{ VAR }}`, the following variables are currently recognised:
| **Variable** | **Description** |
| ------------------- | -------------------------------------------------------------------------------- |
| `DATE` | Date in the form `%Y%m%d` |
| `DATETIME` | Date and time in the form `%Y-%m-%sT%H:%M:S` |
| `DATETIME-%Y%H%M` | Date and time in custom format (where `%Y%H%M` can be any valid form) |
| `USER` | The current user as defined in the CLI |
| `USER_ID` | The unique identifier for the current user |
| `REPO_DIR` | The FAIR repository root directory |
| `CONFIG_DIR` | The directory containing the `config.yaml` after template substitution |
| `LOCAL_TOKEN` | The token for access to the local registry |
| `SOURCE_CONFIG` | Path of the user defined `config.yaml` |
| `GIT_BRANCH` | Current branch of the `git` repository |
| `GIT_REMOTE` | The URI of the git repository specified during setup |
| `GIT_TAG` | The latest tag on `git` |
Raw data
{
"_id": null,
"home_page": "https://www.fairdatapipeline.org/",
"name": "fair-cli",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.0,<4.0.0",
"maintainer_email": "",
"keywords": "FAIR Data Pipeline,FAIR,Data Management,Provenance",
"author": "Richard Reeve",
"author_email": "richard.reeve@glasgow.ac.uk",
"download_url": "https://files.pythonhosted.org/packages/04/ce/8aa0792219c0db899d7f2a7e8eb36f134fe0e22e8154f3c9ec4673a2bacc/fair_cli-0.9.4.tar.gz",
"platform": null,
"description": "# FAIR Data Pipeline Command Line Interface\n\n[](https://pypi.org/project/fair-cli/) [](https://pypi.org/project/fair-cli/) [](https://zenodo.org/badge/latestdoi/377398464)  [](https://bestpractices.coreinfrastructure.org/projects/5411)\n\n[](https://github.com/FAIRDataPipeline/FAIR-CLI/actions/workflows/fair-cli.yaml) [](https://codecov.io/gh/FAIRDataPipeline/FAIR-CLI) [](https://sonarcloud.io/summary/new_code?id=FAIRDataPipeline_FAIR-CLI)\n\nFAIR-CLI forms the main interface for synchronising changes between local and shared remote FAIR Data Pipeline registries, it is also used to instantiate model runs/data submissions to the pipeline. Full documentation of the FAIR Data Pipeline can be found on the project [website](https://www.fairdatapipeline.org/).\n\n## Installation\n\nThe package is installed using Pip:\n\n```sh\npip install fair-cli\n```\n\nTo enable tab completion you need to modify your shell:\n\n### Bash\n```\n_FAIR_COMPLETE=bash_source fair > ~/.config/.fair-complete.bash\necho '. ~/.config/.fair-complete.bash' >> ~/.bashrc\n```\n\n### zsh\n```\n_FAIR_COMPLETE=zsh_source fair > ~/.fair-complete.zsh\necho '. ~/.fair-complete.zsh' >> ~/.bashrc\n```\n\n### Fish\n```\n_FAIR_COMPLETE=bash_source fair > ~/.config/fish/.fair-complete.fish\necho '. ~/.config/fish/.fair-complete.fish' >> ~/.bashrc\n```\n\n## Uninstallation\nTo uninstall the CLI run:\n```\nfair purge --all\npip uninstall fair\n```\n\n## The User Configuration File\nJob runs are configured via `config.yaml` files. Upon initialisation of a project, FAIR-CLI automatically generates a starter configuration file with all requirements in place. To execute a process (e.g. perform a model run from a compiled binary/script) an additional key of either `script` or `script_path` must be provided. Alternatively the command `fair run bash` can be used to append the key and run a command directly.\n\nBy default the shell used to execute a process is `sh` or `batch` for UNIX and Windows systems respectively. This can be overwritten by assigning the optional `shell` key with one of the following values (where `{0}` is the script file):\n\n| **Shell** | **Command** |\n| ------------ | ------------------------------- |\n| `bash` | `bash -eo pipefail {0}` |\n| `java` | `java {0}` |\n| `julia` | `julia {0}` |\n| `powershell` | `powershell -command \". '{0}'\"` |\n| `pwsh` | `pwsh -command \". '{0}'\"` |\n| `python2` | `python2 {0}` |\n| `python3` | `python3 {0}` |\n| `python` | `python {0}` |\n| `R` | `R -f {0}` |\n| `sh` | `sh -e {0}` |\n| `batch` | `{0}` |\n\nA full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).\n\n## Available Commands\n\n### `init`\n\nInitialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.\n\n#### Custom CLI Configuration\nAfter setup is complete, the current CLI configuration can also be saved using the command:\n```\nfair init --export\n```\nthe created file can then be re-read at a later point during setup. Alternatively, if creating a configuration from scratch the YAML file should contain the following information:\n\n```yaml\nnamespaces:\n input: testing\n output: testing\nregistries:\n local:\n data_store: /path/to/local/data_store/,\n directory: /local/registry/install/directory\n uri: http://127.0.0.1:8000/api/\n origin:\n data_store: /remote/registry/data/store/path/\n token: /path/to/remote/token\n uri: https://data.fairdatapipeline.org/api/'\nuser:\n email: 'test@noreply',\n family_name: 'Test'\n given_names: 'Interface'\n orcid: None,\n uuid: '2ddb2358-84bf-43ff-b2aa-3ac7dc3b49f1'\ngit:\n local_repo: /local/repo/path\n remote: origin\ndescription: Testing Project\n```\nthis file is then read during the initialisation:\n\n```sh\nfair init --using <cli-config.yaml file>\n```\n\nFor integration into a CI workflow, the setup can be skipped by running:\n\n```sh\nfair init --ci\n```\n\nwhich will create temporary directories for some of the required location paths.\n\n\n### `run`\n\nThe purpose of `run` is to execute a model/submission run and submit results to the local registry. Outputs of a run will be stored within the `coderun` folder in the directory specified under the `data_store` tag in the `config.yaml`, by default this is `$HOME/.fair/data/coderun`.\n\n```sh\nfair run\n```\n\nIf you wish to use an alternative `config.yaml` then specify it as an additional argument:\n\n```sh\nfair run /path/to/config.yaml\n```\n\nYou can also launch a bash command directly, this will be automatically written into the `config.yaml`:\n\n```sh\nfair run --script 'echo \"Hello World\"'\n```\n\nnote the command itself must be quoted as it is a single argument.\n\nBy default the CLI will not allow the user to perform a run if the state of the analysis repository is such that it is behind the git remote, or contains uncommitted changes. To override this behaviour use the `--dirty` flag.\n\n### `pull`\n\nThe command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.\n\n```sh\nfair pull /path/to/config.yaml\n```\n\n### `status`\nThis command displays objects which are awaiting staging or have been staged behaving in a manner similar to `git status`:\n```sh\nfair status\n```\nstaged changes are displayed in green, and unstaged in red.\n\n### `add`\nBefore changes can be pushed to the remote registry they must be staged. This command allows you to stage objects displayed when running `fair status` so that they can be sent to the remote registry. Data products are displayed and staged in the form `namespace:data_product_name@version`:\n```sh\nfair add my_namespace:data_object@v0.1.0\n```\n\n### `push`\nThe `push` command will push any staged data products to the remote registry:\n\n```sh\nfair push\n```\n\n### `purge`\n\nThe `purge` command removes setup of the current project so it can bereinitialised:\n\n```sh\nfair purge\n```\n\nTo remove all configurations entirely (including those global to all projects) run:\n\n```sh\nfair purge --global\n```\n\nTo remove the data directory itself run:\n\n```sh\nfair purge --data\n```\n\n**WARNING**: This is not recommended as the registry may still have entries pointing to this location!\n\nFinally to remove everything run:\n\n```sh\nfair purge --all\n```\n\nthis will remove the current repository `.fair` folder and the global FAIR directory which also contains the local registry.\n\nYou can skip any confirmation messages by running:\n\n```sh\nfair purge --yes\n```\n\n### `registry`\n\nBy default the CLI will launch the registry whenever a synchronisation or run is called. The server will only be halted once all ongoing CLI processes (in the case of multiple parallel calls) have been completed.\n\nHowever the user may also specify a manual launch that will override this behaviour, instead leaving the server running constantly allowing them to view the registry in the browser.\n\nThe commands:\n\n```sh\nfair registry start\n```\n\nand\n\n```sh\nfair registry stop\n```\n\nwill launch and halt the server respectively.\n\nThe registry can be installed using the CLI as well by running:\n```sh\nfair registry install\n```\nwith the additional options to specify the installation location, and the data registry repository tag to install from:\n```sh\nfair registry install --directory ~/.fair/my_registry --version v1.0-rc5\n```\n\n### `log`\n\nRuns are logged locally within the local FAIR repository. A full list of runs is shown by running:\n\n```sh\nfair log\n```\n\nThis will present a list of runs in a summary analogous to a `git log` call:\n\n```yaml\nrun 0db35c20946a1ebeaafdc3b30103cd74a57eb6b6\nAuthor: Joe Bloggs <jbloggs@noreply.uk>\nDate: Wed Jun 30 09:09:30 2021\n```\n\n| **NOTE** |\n| ----------------------------------------------------------------------------------------------------------------------------------- |\n| The SHA for a job is *not* related to a registry code run identifier as multiple code runs can be executed within a single job. |\n\n### `view`\n\nTo view the `stdout` of a run given its SHA as shown by running `fair log` use the command:\n\n```sh\nfair view <sha>\n```\n\nyou do not need to specify the full SHA but rather the first few unique characters.\n\n## Template Variables\n\nWithin the `config.yaml` file, template variables can be specified by using the notation `${{ VAR }}`, the following variables are currently recognised:\n\n| **Variable** | **Description** |\n| ------------------- | -------------------------------------------------------------------------------- |\n| `DATE` | Date in the form `%Y%m%d` |\n| `DATETIME` | Date and time in the form `%Y-%m-%sT%H:%M:S` |\n| `DATETIME-%Y%H%M` | Date and time in custom format (where `%Y%H%M` can be any valid form) |\n| `USER` | The current user as defined in the CLI |\n| `USER_ID` | The unique identifier for the current user |\n| `REPO_DIR` | The FAIR repository root directory |\n| `CONFIG_DIR` | The directory containing the `config.yaml` after template substitution |\n| `LOCAL_TOKEN` | The token for access to the local registry |\n| `SOURCE_CONFIG` | Path of the user defined `config.yaml` |\n| `GIT_BRANCH` | Current branch of the `git` repository |\n| `GIT_REMOTE` | The URI of the git repository specified during setup |\n| `GIT_TAG` | The latest tag on `git` |\n",
"bugtrack_url": null,
"license": "BSD-2-Clause",
"summary": "Synchronization interface for the SCRC FAIR Data Pipeline registry",
"version": "0.9.4",
"project_urls": {
"Documentation": "https://www.fairdatapipeline.org/docs/interface/fdp/",
"Homepage": "https://www.fairdatapipeline.org/",
"Issue Tracker": "https://github.com/FAIRDataPipeline/FAIR-CLI/issues",
"Repository": "https://github.com/FAIRDataPipeline/FAIR-CLI"
},
"split_keywords": [
"fair data pipeline",
"fair",
"data management",
"provenance"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "21ecf5b8c66c6cfb2a64ed253f4c2d84e73543d211bfc7957aef9f6054960523",
"md5": "07f207628d77541eb577bca044d729f8",
"sha256": "30835988cec2a8eef5478b164e5d412d527b0f1550cdad6087d76ed4df74e8e8"
},
"downloads": -1,
"filename": "fair_cli-0.9.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "07f207628d77541eb577bca044d729f8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0,<4.0.0",
"size": 102983,
"upload_time": "2023-09-14T08:56:54",
"upload_time_iso_8601": "2023-09-14T08:56:54.986533Z",
"url": "https://files.pythonhosted.org/packages/21/ec/f5b8c66c6cfb2a64ed253f4c2d84e73543d211bfc7957aef9f6054960523/fair_cli-0.9.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "04ce8aa0792219c0db899d7f2a7e8eb36f134fe0e22e8154f3c9ec4673a2bacc",
"md5": "d2ada353715720b1f115f1e6fb501530",
"sha256": "2483c498c5ffaa6743983bc4524e93b6b1672ff601f62147e4b44d91cf8d644e"
},
"downloads": -1,
"filename": "fair_cli-0.9.4.tar.gz",
"has_sig": false,
"md5_digest": "d2ada353715720b1f115f1e6fb501530",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0,<4.0.0",
"size": 88756,
"upload_time": "2023-09-14T08:56:57",
"upload_time_iso_8601": "2023-09-14T08:56:57.071962Z",
"url": "https://files.pythonhosted.org/packages/04/ce/8aa0792219c0db899d7f2a7e8eb36f134fe0e22e8154f3c9ec4673a2bacc/fair_cli-0.9.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-14 08:56:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "FAIRDataPipeline",
"github_project": "FAIR-CLI",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "fair-cli"
}
JSON
