# pytorch-caney
Python package for lots of Pytorch tools.
[](https://zenodo.org/badge/latestdoi/472450059)
[](https://github.com/psf/black)
[](https://coveralls.io/github/nasa-nccs-hpda/pytorch-caney?branch=main)
## Documentation
- Latest: https://nasa-nccs-hpda.github.io/pytorch-caney/latest
## Objectives
- Library to process remote sensing imagery using GPU and CPU parallelization.
- Machine Learning and Deep Learning image classification and regression.
- Agnostic array and vector-like data structures.
- User interface environments via Notebooks for easy to use AI/ML projects.
- Example notebooks for quick AI/ML start with your own data.
## Installation
The following library is intended to be used to accelerate the development of data science products
for remote sensing satellite imagery, or any other applications. pytorch-caney can be installed
by itself, but instructions for installing the full environments are listed under the requirements
directory so projects, examples, and notebooks can be run.
Note: PIP installations do not include CUDA libraries for GPU support. Make sure NVIDIA libraries
are installed locally in the system if not using conda/mamba.
```bash
module load singularity # if a module needs to be loaded
singularity build --sandbox pytorch-caney-container docker://nasanccs/pytorch-caney:latest
```
## Why Caney?
"Caney" means longhouse in Taíno.
## Contributors
- Jordan Alexis Caraballo-Vega, jordan.a.caraballo-vega@nasa.gov
- Caleb Spradlin, caleb.s.spradlin@nasa.gov
## Contributing
Please see our [guide for contributing to pytorch-caney](CONTRIBUTING.md).
## SatVision
| name | pretrain | resolution | #params |
| :---: | :---: | :---: | :---: |
| SatVision-B | MODIS-1.9-M | 192x192 | 84.5M |
## SatVision Datasets
| name | bands | resolution | #chips |
| :---: | :---: | :---: | :---: |
| MODIS-Small | 7 | 128x128 | 1,994,131 |
## Pre-training with Masked Image Modeling
To pre-train the swinv2 base model with masked image modeling pre-training, run:
```bash
torchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg <config-file> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp
```
For example to run on a compute node with 4 GPUs and a batch size of 128 on the MODIS SatVision pre-training dataset with a base swinv2 model, run:
```bash
singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
Singularity> export PYTHONPATH=$PWD:$PWD/pytorch-caney
Singularity> torchrun --nproc_per_node 4 pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg pytorch-caney/examples/satvision/mim_pretrain_swinv2_satvision_base_192_window12_800ep.yaml --dataset MODIS --data-paths /explore/nobackup/projects/ilab/data/satvision/pretraining/training_* --batch-size 128 --output . --enable-amp
```
This example script runs the exact configuration used to make the SatVision-base model pre-training with MiM and the MODIS pre-training dataset.
```bash
singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
Singularity> cd pytorch-caney/examples/satvision
Singularity> ./run_satvision_pretrain.sh
```
## Fine-tuning Satvision-base
To fine-tune the satvision-base pre-trained model, run:
```bash
torchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/finetuning/finetune.py --cfg <config-file> --pretrained <path-to-pretrained> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp
```
See example config files pytorch-caney/examples/satvision/finetune_satvision_base_*.yaml to see how to structure your config file for fine-tuning.
## Testing
For unittests, run this bash command to run linting and unit test runs. This will execute unit tests and linting in a temporary venv environment only used for testing.
```bash
git clone git@github.com:nasa-nccs-hpda/pytorch-caney.git
cd pytorch-caney; bash test.sh
```
or run unit tests directly with container or anaconda env
```bash
git clone git@github.com:nasa-nccs-hpda/pytorch-caney.git
singularity build --sandbox pytorch-caney-container docker://nasanccs/pytorch-caney:latest
singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
cd pytorch-caney; python -m unittest discover pytorch_caney/tests
```
```bash
git clone git@github.com:nasa-nccs-hpda/pytorch-caney.git
cd pytorch-caney; conda env create -f requirements/environment_gpu.yml;
conda activate pytorch-caney
python -m unittest discover pytorch_caney/tests
```
## References
- [Pytorch Lightning](https://github.com/Lightning-AI/lightning)
- [Swin Transformer](https://github.com/microsoft/Swin-Transformer)
- [SimMIM](https://github.com/microsoft/SimMIM)
Raw data
{
"_id": null,
"home_page": "https://github.com/nasa-nccs-hpda/pytorch-caney",
"name": "pytorch-caney",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "pytorch-caney,deep-learning,machine-learning",
"author": "jordancaraballo",
"author_email": "jordan.a.caraballo-vega@nasa.gov",
"download_url": "https://files.pythonhosted.org/packages/9b/cb/7558ac22c1aca9eda92a74fcb8fc762630758b24ed9d28e852b0ecbcd594/pytorch-caney-0.1.0.tar.gz",
"platform": null,
"description": "# pytorch-caney\n\nPython package for lots of Pytorch tools.\n\n[](https://zenodo.org/badge/latestdoi/472450059)\n\n\n\n[](https://github.com/psf/black)\n[](https://coveralls.io/github/nasa-nccs-hpda/pytorch-caney?branch=main)\n\n## Documentation\n\n- Latest: https://nasa-nccs-hpda.github.io/pytorch-caney/latest\n\n## Objectives\n\n- Library to process remote sensing imagery using GPU and CPU parallelization.\n- Machine Learning and Deep Learning image classification and regression.\n- Agnostic array and vector-like data structures.\n- User interface environments via Notebooks for easy to use AI/ML projects.\n- Example notebooks for quick AI/ML start with your own data.\n\n## Installation\n\nThe following library is intended to be used to accelerate the development of data science products\nfor remote sensing satellite imagery, or any other applications. pytorch-caney can be installed\nby itself, but instructions for installing the full environments are listed under the requirements\ndirectory so projects, examples, and notebooks can be run.\n\nNote: PIP installations do not include CUDA libraries for GPU support. Make sure NVIDIA libraries\nare installed locally in the system if not using conda/mamba.\n\n```bash\nmodule load singularity # if a module needs to be loaded\nsingularity build --sandbox pytorch-caney-container docker://nasanccs/pytorch-caney:latest\n```\n\n## Why Caney?\n\n\"Caney\" means longhouse in Ta\u00edno.\n\n## Contributors\n\n- Jordan Alexis Caraballo-Vega, jordan.a.caraballo-vega@nasa.gov\n- Caleb Spradlin, caleb.s.spradlin@nasa.gov\n\n## Contributing\n\nPlease see our [guide for contributing to pytorch-caney](CONTRIBUTING.md).\n\n## SatVision\n\n| name | pretrain | resolution | #params |\n| :---: | :---: | :---: | :---: |\n| SatVision-B | MODIS-1.9-M | 192x192 | 84.5M |\n\n## SatVision Datasets\n\n| name | bands | resolution | #chips |\n| :---: | :---: | :---: | :---: |\n| MODIS-Small | 7 | 128x128 | 1,994,131 |\n## Pre-training with Masked Image Modeling\nTo pre-train the swinv2 base model with masked image modeling pre-training, run:\n```bash\ntorchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg <config-file> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp\n```\n\nFor example to run on a compute node with 4 GPUs and a batch size of 128 on the MODIS SatVision pre-training dataset with a base swinv2 model, run:\n\n```bash\nsingularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container\nSingularity> export PYTHONPATH=$PWD:$PWD/pytorch-caney\nSingularity> torchrun --nproc_per_node 4 pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg pytorch-caney/examples/satvision/mim_pretrain_swinv2_satvision_base_192_window12_800ep.yaml --dataset MODIS --data-paths /explore/nobackup/projects/ilab/data/satvision/pretraining/training_* --batch-size 128 --output . --enable-amp\n```\n\nThis example script runs the exact configuration used to make the SatVision-base model pre-training with MiM and the MODIS pre-training dataset.\n```bash\nsingularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container\nSingularity> cd pytorch-caney/examples/satvision\nSingularity> ./run_satvision_pretrain.sh\n```\n\n## Fine-tuning Satvision-base\nTo fine-tune the satvision-base pre-trained model, run:\n```bash\ntorchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/finetuning/finetune.py --cfg <config-file> --pretrained <path-to-pretrained> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp\n```\n\nSee example config files pytorch-caney/examples/satvision/finetune_satvision_base_*.yaml to see how to structure your config file for fine-tuning.\n\n\n## Testing\nFor unittests, run this bash command to run linting and unit test runs. This will execute unit tests and linting in a temporary venv environment only used for testing.\n```bash\ngit clone git@github.com:nasa-nccs-hpda/pytorch-caney.git\ncd pytorch-caney; bash test.sh\n```\nor run unit tests directly with container or anaconda env\n\n```bash\ngit clone git@github.com:nasa-nccs-hpda/pytorch-caney.git\nsingularity build --sandbox pytorch-caney-container docker://nasanccs/pytorch-caney:latest\nsingularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container\ncd pytorch-caney; python -m unittest discover pytorch_caney/tests\n```\n\n```bash\ngit clone git@github.com:nasa-nccs-hpda/pytorch-caney.git\ncd pytorch-caney; conda env create -f requirements/environment_gpu.yml;\nconda activate pytorch-caney\npython -m unittest discover pytorch_caney/tests\n```\n## References\n\n- [Pytorch Lightning](https://github.com/Lightning-AI/lightning)\n- [Swin Transformer](https://github.com/microsoft/Swin-Transformer)\n- [SimMIM](https://github.com/microsoft/SimMIM)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Methods for pytorch deep learning applications",
"version": "0.1.0",
"project_urls": {
"Documentation": "https://github.com/nasa-nccs-hpda/pytorch-caney",
"Homepage": "https://github.com/nasa-nccs-hpda/pytorch-caney",
"Issues": "https://github.com/nasa-nccs-hpda/pytorch-caney/issues",
"Source": "https://github.com/nasa-nccs-hpda/pytorch-caney"
},
"split_keywords": [
"pytorch-caney",
"deep-learning",
"machine-learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a4b9a6122a4d9a594dde3afb3895df23719528b6231cd0c86724387e56625fd8",
"md5": "6114e363563baccbabb322ed50b67ac6",
"sha256": "4d8f96165595a557f0f477d85ae70bf95420b28b7e574ac9bc6e514df96a3abc"
},
"downloads": -1,
"filename": "pytorch_caney-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6114e363563baccbabb322ed50b67ac6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 37813,
"upload_time": "2023-08-21T18:17:55",
"upload_time_iso_8601": "2023-08-21T18:17:55.003762Z",
"url": "https://files.pythonhosted.org/packages/a4/b9/a6122a4d9a594dde3afb3895df23719528b6231cd0c86724387e56625fd8/pytorch_caney-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9bcb7558ac22c1aca9eda92a74fcb8fc762630758b24ed9d28e852b0ecbcd594",
"md5": "5b40f6986b64e2dd920d2ec616f15726",
"sha256": "99e00d8b96e513d24744160553cd7e70e351ff5d6131a60e09d82713c5cbd7a7"
},
"downloads": -1,
"filename": "pytorch-caney-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "5b40f6986b64e2dd920d2ec616f15726",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 33464,
"upload_time": "2023-08-21T18:17:56",
"upload_time_iso_8601": "2023-08-21T18:17:56.817977Z",
"url": "https://files.pythonhosted.org/packages/9b/cb/7558ac22c1aca9eda92a74fcb8fc762630758b24ed9d28e852b0ecbcd594/pytorch-caney-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-21 18:17:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nasa-nccs-hpda",
"github_project": "pytorch-caney",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pytorch-caney"
}
JSON
