| Name | model-config-tests JSON |
| Version |
0.0.7
JSON |
| download |
| home_page | None |
| Summary | Test for ACCESS model (payu) configurations |
| upload_time | 2024-08-13 05:54:44 |
| maintainer | None |
| docs_url | None |
| author | ACCESS-NRI |
| requires_python | >=3.9 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Model Configuration Pytests and CI
This repository houses pytests that are used as part CI checks for model configurations, as well as reusable components of model configuration CI.
The checksum pytests are used for reproducibility CI checks in this repository. The quick configuration tests are used in any repository that calls `config-pr-1-ci.yml` or is templated by [`ACCESS-NRI/model-configs-template](https://github.com/ACCESS-NRI/model-configs-template). For example, [ACCESS-NRI/access-om2-configs](https://github.com/ACCESS-NRI/access-om2-configs).
Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [
bit reproducibility tests](https://github.com/COSIMA/access-om2/blob/master/test/test_bit_reproducibility.py).
## Pytests
### How to run pytests manually on NCI
1. Create and activate a python virtual environment for installing and running tests
```sh
module load python3/3.11.0
python3 -m venv <path/to/test-venv>
source <path/to/test-venv>/bin/activate
```
2. Either pip install a released version of `model-config-tests`,
```sh
pip install model-config-tests==0.0.1
```
Or to install `model-config-tests` in "editable" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.
```sh
git clone https://github.com/ACCESS-NRI/model-config-tests/ <path/to/model-config-tests>
pip install -e <path/to/model-config-tests>
```
3. Checkout an experiment (in this case it is using an ACCESS-OM2 config)
```sh
git clone https://github.com/ACCESS-NRI/access-om2-configs/ <path/to/experiment>
cd <path/to/experiment>
git checkout <branch/tag>
```
4. Run the pytests
```sh
model-config-tests
```
5. Once done with testing, deactivate the virtual environment, and if the environment is no longer needed, remove the environment
```sh
deactivate
rm -rf <path/to/test-venv> # Deletes the test environment
```
### Pytest Options
The output directory for pytests defaults to `$TMPDIR/test-model-repro` and contains the following subdirectories:
- `control` - contains copies of the model configuration used for each experiment run in the tests.
- `lab` - contains `payu` model output directories containing `work` and `archive` sub-directories.
This output directory also contains files generated by pytests, including the `checksum` directory which is used as part of reproducibility CI workflows.
To specify a different folder for pytest outputs, use `--output-path` command flag, for example:
```sh
model-config-tests --output-path /some/other/path/for/output
```
By default, the control directory, e.g. the model configuration to test, is the current working directory. This can be set similarly to above by using the
`--control-path` command flag.
The path containing the checksum file to check against can also be set using
`--checksum-path` command flag. The default is the `testing/checksum/historical-<default-model-runtime>hr-checksums.json` file which is stored in the control directory.
To run only CI reproducibility checksum tests, use `-m checksum`, e.g.
```sh
model-config-tests -m checksum
```
To run quick configuration tests, use the `config` marker. To additionally run
ACCESS-OM2 specific quick configuration tests, use `access_om2` marker,
e.g.:
```sh
model-config-tests -m "config or access_om2"
```
## CI/CD
The `.github` directory contains many different workflows and actions. This section describes how they are used.
### CI/CD For This Repository
`CI.yml` and `CD.yml` are used to test, package and upload the `model-config-tests` package that is used by `model-configs`-style repositories across the ACCESS-NRI. These are the only workflows that run on this repository. The others are reusable workflows called by `model-configs`-style repositories, among others.
### Reusable CI
The `config-*.yml`, `generate-checksums.yml` and `test-repro.yml` workflows are called by `model-configs`-style repositories to test model configurations. They are stored in this repository to allow a central place to update generic CI used by all model configuration repositories.
Below is information on the use of these workflows.
#### `config-pr-*.yml` Pipeline
The `config-pr-*` Pipeline is a series of workflows that govern the testing, ChatOps and merging procedures of pull requests for model configuration repositories, such as [`ACCESS-NRI/access-om2-configs`](https://github.com/ACCESS-NRI/access-om2-configs).
Essentially, these files work on two types of configuration branch pull requests in the model configuration repository. More information on the terminology used in model configuration repositories can be found in the `README.md` of the `ACCESS-NRI/model-configs-template` repository. The types of pull requests are:
- Pull requests into `dev-*`: Allows quick checks of configuration metadata and common mistakes in configurations during PRs into the `dev-*` configuration branches.
- Pull requests from `dev-*` into `release-*`: Allows both quick checks, as well as a longer, more comprehensive check on the reproducibility of the changes being brought into the `release-*` configuration branch, compared to the previous `release-*` commit. It also acts on 'comment commands' run during the pull request, like `!bump` for updating the version of the configuration ([see the 'Config Tags' section](https://github.com/ACCESS-NRI/model-configs-template/blob/main/README.md) in the `ACCESS-NRI/model-configs-template` repository for more). It is also responsible for the creation of the final config tag and release, once merged.
#### `config-schedule-*.yml` Pipeline
The `config-schedule-*` Pipeline is a series of workflows used to check the reproducibility of certain config tags against themselves, every month. This is used as a kind of canary to make sure that we continue to get the same results on the same deployment targets.
#### `generate-checksums` Reusable Workflow
This workflow is used to easily generate the checksums used in the reproducibility checks, for a specific branch of a model configuration repository, if they don't already exist. This is most often used for the initial commit of a checksum to the `release-*` configuration branch.
#### `test-repro` Reusable Workflow
This workflow is used to test the reproducibility of a given model repository against historical checksums, and can be used as a standalone workflow.
Using it has some requirements outside of just filling in the inputs: One must have a valid GitHub Environment (specified by the `environment-name` input) in the calling repository, that has the following `secrets` and `vars` defined:
- `secrets.SSH_HOST` - hostname for the deployment target
- `secrets.SSH_HOST_DATA` - hostname for the data mover on the deployment target (if it exists)
- `secrets.SSH_KEY` - private key for access to the deployment target
- `secrets.SSH_USER` - username for access to the deployment target
- `vars.EXPERIMENTS_LOCATION` - directory on the deployment target that will contain all the experiments used during testing of reproducibility across multiple runs of this workflow (ex. `/scratch/some/directory/experiments`)
Raw data
{
"_id": null,
"home_page": null,
"name": "model-config-tests",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "ACCESS-NRI",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/d9/d8/836f93e34472aa4aeddbdd158abe5ec08d9c507f60213ae9725b7a3f7846/model_config_tests-0.0.7.tar.gz",
"platform": null,
"description": "# Model Configuration Pytests and CI\n\nThis repository houses pytests that are used as part CI checks for model configurations, as well as reusable components of model configuration CI.\n\nThe checksum pytests are used for reproducibility CI checks in this repository. The quick configuration tests are used in any repository that calls `config-pr-1-ci.yml` or is templated by [`ACCESS-NRI/model-configs-template](https://github.com/ACCESS-NRI/model-configs-template). For example, [ACCESS-NRI/access-om2-configs](https://github.com/ACCESS-NRI/access-om2-configs).\n\nCode from these pytests is adapted from COSIMAS's ACCESS-OM2's [\nbit reproducibility tests](https://github.com/COSIMA/access-om2/blob/master/test/test_bit_reproducibility.py).\n\n## Pytests\n\n### How to run pytests manually on NCI\n\n1. Create and activate a python virtual environment for installing and running tests\n ```sh\n module load python3/3.11.0\n python3 -m venv <path/to/test-venv>\n source <path/to/test-venv>/bin/activate\n ```\n\n2. Either pip install a released version of `model-config-tests`,\n ```sh\n pip install model-config-tests==0.0.1\n ```\n Or to install `model-config-tests` in \"editable\" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.\n ```sh\n git clone https://github.com/ACCESS-NRI/model-config-tests/ <path/to/model-config-tests>\n pip install -e <path/to/model-config-tests>\n ```\n\n3. Checkout an experiment (in this case it is using an ACCESS-OM2 config)\n\n ```sh\n git clone https://github.com/ACCESS-NRI/access-om2-configs/ <path/to/experiment>\n cd <path/to/experiment>\n git checkout <branch/tag>\n ```\n\n4. Run the pytests\n ```sh\n model-config-tests\n ```\n\n5. Once done with testing, deactivate the virtual environment, and if the environment is no longer needed, remove the environment\n ```sh\n deactivate\n rm -rf <path/to/test-venv> # Deletes the test environment\n ```\n\n### Pytest Options\n\nThe output directory for pytests defaults to `$TMPDIR/test-model-repro` and contains the following subdirectories:\n\n- `control` - contains copies of the model configuration used for each experiment run in the tests.\n- `lab` - contains `payu` model output directories containing `work` and `archive` sub-directories.\n\nThis output directory also contains files generated by pytests, including the `checksum` directory which is used as part of reproducibility CI workflows.\n\nTo specify a different folder for pytest outputs, use `--output-path` command flag, for example:\n\n```sh\nmodel-config-tests --output-path /some/other/path/for/output\n```\n\nBy default, the control directory, e.g. the model configuration to test, is the current working directory. This can be set similarly to above by using the\n`--control-path` command flag.\n\nThe path containing the checksum file to check against can also be set using\n`--checksum-path` command flag. The default is the `testing/checksum/historical-<default-model-runtime>hr-checksums.json` file which is stored in the control directory.\n\nTo run only CI reproducibility checksum tests, use `-m checksum`, e.g.\n\n```sh\nmodel-config-tests -m checksum\n```\n\nTo run quick configuration tests, use the `config` marker. To additionally run\nACCESS-OM2 specific quick configuration tests, use `access_om2` marker,\ne.g.:\n\n```sh\nmodel-config-tests -m \"config or access_om2\"\n```\n\n## CI/CD\n\nThe `.github` directory contains many different workflows and actions. This section describes how they are used.\n\n### CI/CD For This Repository\n\n`CI.yml` and `CD.yml` are used to test, package and upload the `model-config-tests` package that is used by `model-configs`-style repositories across the ACCESS-NRI. These are the only workflows that run on this repository. The others are reusable workflows called by `model-configs`-style repositories, among others.\n\n### Reusable CI\n\nThe `config-*.yml`, `generate-checksums.yml` and `test-repro.yml` workflows are called by `model-configs`-style repositories to test model configurations. They are stored in this repository to allow a central place to update generic CI used by all model configuration repositories.\n\nBelow is information on the use of these workflows.\n\n#### `config-pr-*.yml` Pipeline\n\nThe `config-pr-*` Pipeline is a series of workflows that govern the testing, ChatOps and merging procedures of pull requests for model configuration repositories, such as [`ACCESS-NRI/access-om2-configs`](https://github.com/ACCESS-NRI/access-om2-configs).\n\nEssentially, these files work on two types of configuration branch pull requests in the model configuration repository. More information on the terminology used in model configuration repositories can be found in the `README.md` of the `ACCESS-NRI/model-configs-template` repository. The types of pull requests are:\n\n- Pull requests into `dev-*`: Allows quick checks of configuration metadata and common mistakes in configurations during PRs into the `dev-*` configuration branches.\n- Pull requests from `dev-*` into `release-*`: Allows both quick checks, as well as a longer, more comprehensive check on the reproducibility of the changes being brought into the `release-*` configuration branch, compared to the previous `release-*` commit. It also acts on 'comment commands' run during the pull request, like `!bump` for updating the version of the configuration ([see the 'Config Tags' section](https://github.com/ACCESS-NRI/model-configs-template/blob/main/README.md) in the `ACCESS-NRI/model-configs-template` repository for more). It is also responsible for the creation of the final config tag and release, once merged.\n\n#### `config-schedule-*.yml` Pipeline\n\nThe `config-schedule-*` Pipeline is a series of workflows used to check the reproducibility of certain config tags against themselves, every month. This is used as a kind of canary to make sure that we continue to get the same results on the same deployment targets.\n\n#### `generate-checksums` Reusable Workflow\n\nThis workflow is used to easily generate the checksums used in the reproducibility checks, for a specific branch of a model configuration repository, if they don't already exist. This is most often used for the initial commit of a checksum to the `release-*` configuration branch.\n\n#### `test-repro` Reusable Workflow\n\nThis workflow is used to test the reproducibility of a given model repository against historical checksums, and can be used as a standalone workflow.\n\nUsing it has some requirements outside of just filling in the inputs: One must have a valid GitHub Environment (specified by the `environment-name` input) in the calling repository, that has the following `secrets` and `vars` defined:\n\n- `secrets.SSH_HOST` - hostname for the deployment target\n- `secrets.SSH_HOST_DATA` - hostname for the data mover on the deployment target (if it exists)\n- `secrets.SSH_KEY` - private key for access to the deployment target\n- `secrets.SSH_USER` - username for access to the deployment target\n- `vars.EXPERIMENTS_LOCATION` - directory on the deployment target that will contain all the experiments used during testing of reproducibility across multiple runs of this workflow (ex. `/scratch/some/directory/experiments`)\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Test for ACCESS model (payu) configurations",
"version": "0.0.7",
"project_urls": {
"Homepage": "https://github.com/ACCESS-NRI/model-config-tests/",
"Issues": "https://github.com/ACCESS-NRI/model-config-tests/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ac3aab9fb0d5cb6a3df9c0bb7d1f34f762ec8f347af72a3d7ea3c3ef75a1a523",
"md5": "af051e2f5a0a5dd4c8725644ce5e6337",
"sha256": "d37a1aa39cbf170744f770da07d92e810376b3a103bd5373a734c8b05c3ffa87"
},
"downloads": -1,
"filename": "model_config_tests-0.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "af051e2f5a0a5dd4c8725644ce5e6337",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 31119,
"upload_time": "2024-08-13T05:54:43",
"upload_time_iso_8601": "2024-08-13T05:54:43.546656Z",
"url": "https://files.pythonhosted.org/packages/ac/3a/ab9fb0d5cb6a3df9c0bb7d1f34f762ec8f347af72a3d7ea3c3ef75a1a523/model_config_tests-0.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d9d8836f93e34472aa4aeddbdd158abe5ec08d9c507f60213ae9725b7a3f7846",
"md5": "6d38026e2c8ab7d0ae27f03f20a5e8df",
"sha256": "98954136a83d8b5a72246046624bd65303c52c9eadcb7e9b51d6da7678caa37c"
},
"downloads": -1,
"filename": "model_config_tests-0.0.7.tar.gz",
"has_sig": false,
"md5_digest": "6d38026e2c8ab7d0ae27f03f20a5e8df",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 30279,
"upload_time": "2024-08-13T05:54:44",
"upload_time_iso_8601": "2024-08-13T05:54:44.910578Z",
"url": "https://files.pythonhosted.org/packages/d9/d8/836f93e34472aa4aeddbdd158abe5ec08d9c507f60213ae9725b7a3f7846/model_config_tests-0.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-13 05:54:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ACCESS-NRI",
"github_project": "model-config-tests",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "model-config-tests"
}
