| Name | access-mopper JSON |
| Version |
2.3.0a4
JSON |
| download |
| home_page | None |
| Summary | ACCESS Model Output Post-Processor, maps raw model output to CMIP-style defined variables and produce post-processed output using CMOR3 |
| upload_time | 2025-08-04 23:32:51 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | Apache-2.0 |
| keywords |
access
post-processing
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ACCESS Model Output Post-Processor (ACCESS-MOPPeR) v2
[](https://access-mopper-v2.readthedocs.io/en/latest/?badge=latest)
[](https://badge.fury.io/py/access_mopper)
[](https://anaconda.org/conda-forge/access_mopper)
ACCESS-MOPPeR is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets. This version represents a significant rewrite focusing on usability, flexibility, and integration with modern Python workflows.
## Key Features
- **Python API** for integration into notebooks and scripts
- **Batch processing system** for HPC environments with PBS
- **Real-time monitoring** with web-based dashboard
- **Flexible CMORisation** of individual variables
- **Dask-enabled** for scalable parallel processing
- **Cross-platform compatibility** (not limited to NCI Gadi)
- **CMIP6 and CMIP7 FastTrack support**
## Installation
ACCESS-MOPPeR requires Python >= 3.11. Install with:
```bash
pip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit
pip install .
```
## Quick Start
### Interactive Usage (Python API)
```python
import glob
from access_mopper import ACCESS_ESM_CMORiser
# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")
# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
input_paths=files,
compound_name="Amon.pr", # table.variable format
experiment_id="historical",
source_id="ACCESS-ESM1-5",
variant_label="r1i1p1f1",
grid_label="gn",
activity_id="CMIP",
output_path="/path/to/output"
)
# Run CMORisation
cmoriser.run()
cmoriser.write()
```
### Batch Processing (HPC/PBS)
For large-scale processing on HPC systems:
1. **Create a configuration file** (`batch_config.yml`):
```yaml
variables:
- Amon.pr
- Omon.tos
- Amon.ts
experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn
input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"
file_patterns:
Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"
worker_init: |
module load conda
conda activate your_environment
```
2. **Submit batch job**:
```bash
mopper-cmorise batch_config.yml
```
3. **Monitor progress** at http://localhost:8501
## Batch Processing Features
The batch processing system provides:
- **Parallel execution**: Each variable processed as a separate PBS job
- **Real-time monitoring**: Web dashboard showing job status and progress
- **Automatic tracking**: SQLite database maintains job history and status
- **Error handling**: Failed jobs can be easily identified and resubmitted
- **Resource optimization**: Configurable CPU, memory, and storage requirements
- **Environment management**: Automatic setup of conda/module environments
### Monitoring Tools
- **Streamlit Dashboard**: Real-time web interface at http://localhost:8501
- **Command line**: Use standard PBS commands (`qstat`, `qdel`)
- **Database**: SQLite tracking at `{output_folder}/cmor_tasks.db`
- **Log files**: Individual stdout/stderr for each job
### File Organization
```
work_directory/
├── batch_config.yml # Your configuration
├── cmor_job_scripts/ # Generated PBS scripts and logs
│ ├── cmor_Amon_pr.sh # PBS script
│ ├── cmor_Amon_pr.py # Python processing script
│ ├── cmor_Amon_pr.out # Job output
│ └── cmor_Amon_pr.err # Job errors
└── output_folder/
├── cmor_tasks.db # Progress tracking
└── [CMORised files] # Final output
```
## Documentation
- **Getting Started**: `docs/source/getting_started.rst`
- **Example Configuration**: `src/access_mopper/examples/batch_config.yml`
- **API Reference**: [Coming soon]
## Current Limitations
- **Alpha version**: Intended for evaluation only
- **Ocean variables**: Limited support in current release
- **Variable mapping**: Under review for CMIP6/CMIP7 compliance
## Support
- **Issues**: Submit via GitHub Issues
- **Questions**: Contact ACCESS-NRI support
- **Contributions**: Welcome via Pull Requests
## License
ACCESS-MOPPeR is licensed under the Apache-2.0 License.
---
**Background**: ACCESS-MOPPeR v2 is a complete rewrite using modern Python libraries (xarray, dask) instead of CMOR, providing improved flexibility and integration with contemporary data science workflows.
Raw data
{
"_id": null,
"home_page": null,
"name": "access-mopper",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "ACCESS, post-processing",
"author": null,
"author_email": "Romain Beucher <romain.beucher@anu.edu.au>",
"download_url": "https://files.pythonhosted.org/packages/63/2b/090b4fc4ba23f24bfbdd8bd1f9e5ceda4cbe8ec60257edc0a33efa38ff49/access_mopper-2.3.0a4.tar.gz",
"platform": null,
"description": "# ACCESS Model Output Post-Processor (ACCESS-MOPPeR) v2\n\n[](https://access-mopper-v2.readthedocs.io/en/latest/?badge=latest)\n[](https://badge.fury.io/py/access_mopper)\n[](https://anaconda.org/conda-forge/access_mopper)\n\nACCESS-MOPPeR is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets. This version represents a significant rewrite focusing on usability, flexibility, and integration with modern Python workflows.\n\n## Key Features\n\n- **Python API** for integration into notebooks and scripts\n- **Batch processing system** for HPC environments with PBS\n- **Real-time monitoring** with web-based dashboard\n- **Flexible CMORisation** of individual variables\n- **Dask-enabled** for scalable parallel processing\n- **Cross-platform compatibility** (not limited to NCI Gadi)\n- **CMIP6 and CMIP7 FastTrack support**\n\n## Installation\n\nACCESS-MOPPeR requires Python >= 3.11. Install with:\n\n```bash\npip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit\npip install .\n```\n\n## Quick Start\n\n### Interactive Usage (Python API)\n\n```python\nimport glob\nfrom access_mopper import ACCESS_ESM_CMORiser\n\n# Select input files\nfiles = glob.glob(\"/path/to/model/output/*mon.nc\")\n\n# Create CMORiser instance\ncmoriser = ACCESS_ESM_CMORiser(\n input_paths=files,\n compound_name=\"Amon.pr\", # table.variable format\n experiment_id=\"historical\",\n source_id=\"ACCESS-ESM1-5\",\n variant_label=\"r1i1p1f1\",\n grid_label=\"gn\",\n activity_id=\"CMIP\",\n output_path=\"/path/to/output\"\n)\n\n# Run CMORisation\ncmoriser.run()\ncmoriser.write()\n```\n\n### Batch Processing (HPC/PBS)\n\nFor large-scale processing on HPC systems:\n\n1. **Create a configuration file** (`batch_config.yml`):\n\n```yaml\nvariables:\n - Amon.pr\n - Omon.tos\n - Amon.ts\n\nexperiment_id: piControl\nsource_id: ACCESS-ESM1-5\nvariant_label: r1i1p1f1\ngrid_label: gn\n\ninput_folder: \"/g/data/project/model/output\"\noutput_folder: \"/scratch/project/cmor_output\"\n\nfile_patterns:\n Amon.pr: \"output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc\"\n Omon.tos: \"output[0-4][0-9][0-9]/ocean/*temp*.nc\"\n Amon.ts: \"output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc\"\n\n# PBS configuration\nqueue: normal\ncpus_per_node: 16\nmem: 32GB\nwalltime: \"02:00:00\"\nscheduler_options: \"#PBS -P your_project\"\nstorage: \"gdata/project+scratch/project\"\n\nworker_init: |\n module load conda\n conda activate your_environment\n```\n\n2. **Submit batch job**:\n\n```bash\nmopper-cmorise batch_config.yml\n```\n\n3. **Monitor progress** at http://localhost:8501\n\n## Batch Processing Features\n\nThe batch processing system provides:\n\n- **Parallel execution**: Each variable processed as a separate PBS job\n- **Real-time monitoring**: Web dashboard showing job status and progress\n- **Automatic tracking**: SQLite database maintains job history and status\n- **Error handling**: Failed jobs can be easily identified and resubmitted\n- **Resource optimization**: Configurable CPU, memory, and storage requirements\n- **Environment management**: Automatic setup of conda/module environments\n\n### Monitoring Tools\n\n- **Streamlit Dashboard**: Real-time web interface at http://localhost:8501\n- **Command line**: Use standard PBS commands (`qstat`, `qdel`)\n- **Database**: SQLite tracking at `{output_folder}/cmor_tasks.db`\n- **Log files**: Individual stdout/stderr for each job\n\n### File Organization\n\n```\nwork_directory/\n\u251c\u2500\u2500 batch_config.yml # Your configuration\n\u251c\u2500\u2500 cmor_job_scripts/ # Generated PBS scripts and logs\n\u2502 \u251c\u2500\u2500 cmor_Amon_pr.sh # PBS script\n\u2502 \u251c\u2500\u2500 cmor_Amon_pr.py # Python processing script\n\u2502 \u251c\u2500\u2500 cmor_Amon_pr.out # Job output\n\u2502 \u2514\u2500\u2500 cmor_Amon_pr.err # Job errors\n\u2514\u2500\u2500 output_folder/\n \u251c\u2500\u2500 cmor_tasks.db # Progress tracking\n \u2514\u2500\u2500 [CMORised files] # Final output\n```\n\n## Documentation\n\n- **Getting Started**: `docs/source/getting_started.rst`\n- **Example Configuration**: `src/access_mopper/examples/batch_config.yml`\n- **API Reference**: [Coming soon]\n\n## Current Limitations\n\n- **Alpha version**: Intended for evaluation only\n- **Ocean variables**: Limited support in current release\n- **Variable mapping**: Under review for CMIP6/CMIP7 compliance\n\n## Support\n\n- **Issues**: Submit via GitHub Issues\n- **Questions**: Contact ACCESS-NRI support\n- **Contributions**: Welcome via Pull Requests\n\n## License\n\nACCESS-MOPPeR is licensed under the Apache-2.0 License.\n\n---\n\n**Background**: ACCESS-MOPPeR v2 is a complete rewrite using modern Python libraries (xarray, dask) instead of CMOR, providing improved flexibility and integration with contemporary data science workflows.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "ACCESS Model Output Post-Processor, maps raw model output to CMIP-style defined variables and produce post-processed output using CMOR3",
"version": "2.3.0a4",
"project_urls": null,
"split_keywords": [
"access",
" post-processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cf49d802037a779f20cddd7d8d0966d467697c99e3b6f094ae3aa96e7dfc3595",
"md5": "12a8c05433af98c52c8916b5a3002482",
"sha256": "e352747538715b4872c9161bbd6499b4c13b891623b4e6e289b36328a76412b3"
},
"downloads": -1,
"filename": "access_mopper-2.3.0a4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "12a8c05433af98c52c8916b5a3002482",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 345108,
"upload_time": "2025-08-04T23:32:49",
"upload_time_iso_8601": "2025-08-04T23:32:49.523248Z",
"url": "https://files.pythonhosted.org/packages/cf/49/d802037a779f20cddd7d8d0966d467697c99e3b6f094ae3aa96e7dfc3595/access_mopper-2.3.0a4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "632b090b4fc4ba23f24bfbdd8bd1f9e5ceda4cbe8ec60257edc0a33efa38ff49",
"md5": "940185969ee2c174753406c6e87da87d",
"sha256": "dfdb248076866b4a6bcc6e706244f09c342b460aec59516c811c26fd08480968"
},
"downloads": -1,
"filename": "access_mopper-2.3.0a4.tar.gz",
"has_sig": false,
"md5_digest": "940185969ee2c174753406c6e87da87d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 309588,
"upload_time": "2025-08-04T23:32:51",
"upload_time_iso_8601": "2025-08-04T23:32:51.433992Z",
"url": "https://files.pythonhosted.org/packages/63/2b/090b4fc4ba23f24bfbdd8bd1f9e5ceda4cbe8ec60257edc0a33efa38ff49/access_mopper-2.3.0a4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-04 23:32:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "access-mopper"
}
