# paramsurvey-tooling
## Install
```
pip install paramsurvey-tooling
```
## paramsurvey usage on clusters with batch queues
Make 3 batch scripts: head, driver, and child. They all have a similar header:
```
#FOO batch header lines
. ~/setup-my-software-environment.sh
```
The script to set up your environment should activate your Conda environment, or
load modules or whatever.
Then each of these scripts gets one of the following commands:
* `pstool start head`
* `pstool start driver ./my-script.py foo bar baz`
* `pstool start child`
Finally, submit the head job, then the driver, and finally a bunch of children.
Instead of using a driver batch script, you can also run the driver on a head
node:
```
pstool submit ./my-script.py foo bar baz # NOT YET WORKING
```
## Containers and paramsurvey
Many compute clusters do not support Docker containers, for security reasons.
These clusters often do support Singularity containers, and it's not hard to
turn an arbitrary Docker container into a Singularity container. Build
the Docker container on a machine that does have docker installed, and
then export it to a file:
```
docker save IMAGE_ID | gzip > my_docker_image.tar.gz
```
Transfer that file to a host with `paramsurvey-tooling` and `singularity`
installed, and do:
```
pstool build my_docker_image.tar.gz
```
## If you find a bug
This tool attempts to support many systems that the author can not
directly test. If you find a bug in your environment, run this:
```
pstool debug > DEBUG
```
and attach this file `DEBUG` to your bug report.
## Documentation
Installing code in an environment:
* Conda
* OS packages
* modules
Containerized environments:
* Docker
* Singularity
Execution environment:
* [Laptop or a single server](Execution-Laptop.md)
* [Cluster with batch queue:](Execution-Cluster.md)
* Philosophy
* Details
* slurm -- Harvard cluster
* PBSPro -- ASIAA cluster
* Grid Engine -- Smithsonian Hydra cluster
* Tearing down
* Cloud-native:
* Philosophy
* Open Science Grid
* https://bhpire.arizona.edu/2022/09/12/new-webinar-performing-large-scale-parameter-surveys-with-osg-services/
* Ray Cluster on AWS or GCE
* Ray Cluster over Kubernetes
Raw data
{
"_id": null,
"home_page": "https://github.com/wumpus/paramsurvey-tooling",
"name": "paramsurvey-tooling",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6.*",
"maintainer_email": "",
"keywords": "",
"author": "Greg Lindahl and others",
"author_email": "lindahl@pbm.com",
"download_url": "https://files.pythonhosted.org/packages/be/29/9323f4ac23b4fd5bbd14c084cfc08f6b3d32846806868ee16a915863d603/paramsurvey_tooling-0.3.8.tar.gz",
"platform": null,
"description": "# paramsurvey-tooling\n\n## Install\n\n```\npip install paramsurvey-tooling\n```\n\n## paramsurvey usage on clusters with batch queues\n\nMake 3 batch scripts: head, driver, and child. They all have a similar header:\n\n```\n#FOO batch header lines\n\n. ~/setup-my-software-environment.sh\n```\n\nThe script to set up your environment should activate your Conda environment, or\nload modules or whatever.\n\nThen each of these scripts gets one of the following commands:\n\n* `pstool start head`\n* `pstool start driver ./my-script.py foo bar baz`\n* `pstool start child`\n\nFinally, submit the head job, then the driver, and finally a bunch of children.\n\nInstead of using a driver batch script, you can also run the driver on a head\nnode:\n\n```\npstool submit ./my-script.py foo bar baz # NOT YET WORKING\n```\n\n## Containers and paramsurvey\n\nMany compute clusters do not support Docker containers, for security reasons.\nThese clusters often do support Singularity containers, and it's not hard to\nturn an arbitrary Docker container into a Singularity container. Build\nthe Docker container on a machine that does have docker installed, and\nthen export it to a file:\n\n```\ndocker save IMAGE_ID | gzip > my_docker_image.tar.gz\n```\n\nTransfer that file to a host with `paramsurvey-tooling` and `singularity`\ninstalled, and do:\n\n```\npstool build my_docker_image.tar.gz\n```\n\n## If you find a bug\n\nThis tool attempts to support many systems that the author can not\ndirectly test. If you find a bug in your environment, run this:\n\n```\npstool debug > DEBUG\n```\n\nand attach this file `DEBUG` to your bug report.\n\n## Documentation\n\nInstalling code in an environment:\n* Conda\n* OS packages\n* modules\n\nContainerized environments:\n* Docker\n* Singularity\n\nExecution environment:\n* [Laptop or a single server](Execution-Laptop.md)\n* [Cluster with batch queue:](Execution-Cluster.md)\n * Philosophy\n * Details\n * slurm -- Harvard cluster\n * PBSPro -- ASIAA cluster\n * Grid Engine -- Smithsonian Hydra cluster\n * Tearing down\n* Cloud-native:\n * Philosophy\n * Open Science Grid\n * https://bhpire.arizona.edu/2022/09/12/new-webinar-performing-large-scale-parameter-surveys-with-osg-services/\n * Ray Cluster on AWS or GCE\n * Ray Cluster over Kubernetes\n\n\n\n\n\n\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A toolkit to make paramsurvey easy to use on clusters and cloud",
"version": "0.3.8",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "be299323f4ac23b4fd5bbd14c084cfc08f6b3d32846806868ee16a915863d603",
"md5": "0dbe2bd7f5bd8c4fecabee140de1c528",
"sha256": "75175a7f56221ca5967b155fb56e5bbefc92b04f088b6eee88bb58901d0dcb09"
},
"downloads": -1,
"filename": "paramsurvey_tooling-0.3.8.tar.gz",
"has_sig": false,
"md5_digest": "0dbe2bd7f5bd8c4fecabee140de1c528",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6.*",
"size": 13322,
"upload_time": "2023-01-18T02:32:21",
"upload_time_iso_8601": "2023-01-18T02:32:21.908929Z",
"url": "https://files.pythonhosted.org/packages/be/29/9323f4ac23b4fd5bbd14c084cfc08f6b3d32846806868ee16a915863d603/paramsurvey_tooling-0.3.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-18 02:32:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "wumpus",
"github_project": "paramsurvey-tooling",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "paramsurvey-tooling"
}
JSON
