pysqa


Namepysqa JSON
Version 0.2.2 PyPI version JSON
download
home_pageNone
SummarySimple HPC queuing system adapter for Python on based jinja templates to automate the submission script creation.
upload_time2024-11-09 06:33:36
maintainerNone
docs_urlNone
authorNone
requires_python<3.14,>=3.9
licenseBSD 3-Clause License Copyright (c) 2019, Jan Janssen All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
keywords pyiron
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # pysqa

[![Unittests](https://github.com/pyiron/pysqa/actions/workflows/unittest.yml/badge.svg)](https://github.com/pyiron/pysqa/actions/workflows/unittest.yml)
[![Documentation Status](https://readthedocs.org/projects/pysqa/badge/?version=latest)](https://pysqa.readthedocs.io/en/latest/?badge=latest)
[![Coverage Status](https://coveralls.io/repos/github/pyiron/pysqa/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pysqa?branch=main)
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pysqa/HEAD?labpath=example_config.ipynb)

High-performance computing (HPC) does not have to be hard. In this context the aim of the Python Simple Queuing System 
Adapter (`pysqa`) is to simplify the submission of tasks from python to HPC clusters as easy as starting another 
`subprocess` locally. This is achieved based on the assumption that even though modern HPC queuing systems offer a wide 
range of different configuration options, most users submit the majority of their jobs with very similar parameters. 

Therefore, in `pysqa` users define submission script templates once and reuse them to submit many different tasks and 
workflows afterwards. These templates are defined in the [jinja2 template language](https://palletsprojects.com/p/jinja/),
so current submission scripts can be easily converted to templates. In addition, to the submission of new tasks to HPC
queuing systems, `pysqa` also allows the users to track the progress of their tasks, delete them or enable reservations 
using the built-in functionality of the queuing system. Finally, `pysqa` enables remote connections to HPC clusters 
using SSH including support for two factor authentication via [pyauthenticator](https://github.com/jan-janssen/pyauthenticator), 
this allows the users to submit task from a python process on their local workstation to remote HPC clusters.

All this functionality is available from both the [Python interface](https://pysqa.readthedocs.io/en/latest/example.html) 
as well as the [command line interface](https://pysqa.readthedocs.io/en/latest/command.html). 

## Features
The core feature of `pysqa` is the communication to HPC queuing systems including ([Flux](https://pysqa.readthedocs.io/en/latest/queue.html#flux), 
[LFS](https://pysqa.readthedocs.io/en/latest/queue.html#lfs), [MOAB](https://pysqa.readthedocs.io/en/latest/queue.html#moab), 
[SGE](https://pysqa.readthedocs.io/en/latest/queue.html#sge), [SLURM](https://pysqa.readthedocs.io/en/latest/queue.html#slurm) 
and [TORQUE](https://pysqa.readthedocs.io/en/latest/queue.html#torque)). This includes: 

* `QueueAdapter().submit_job()` - Submission of new tasks to the queuing system. 
* `QueueAdapter().get_queue_status()` - List of calculation currently waiting or running on the queuing system. 
* `QueueAdapter().delete_job()` - Deleting calculation which are currently waiting or running on the queuing system. 
* `QueueAdapter().queue_list` - List of available queue templates created by the user.
* `QueueAdapter().config` - Templates to a specific number of cores, run time or other computing resources. With 
  integrated checks if a given submitted task follows these restrictions. 

In addition to these core features, `pysqa` is continuously extended to support more use cases for a larger group of 
users. These new features include the support for remote queuing systems: 

* Remote connection via the secure shell protocol (SSH) to access remote HPC clusters.
* Transfer of files to and from remote HPC clusters, based on a predefined mapping of the remote file system into the 
  local file system. 
* Support for both individual connections as well as continuous connections depending on the network availability. 

Finally, there is current work in progress to support a combination of [multiple local and remote queuing systems](https://pysqa.readthedocs.io/en/latest/advanced.html) 
from within `pysqa`, which are represented to the user as a single resource. 

## Documentation

* [Installation](https://pysqa.readthedocs.io/en/latest/installation.html)
  * [pypi-based installation](https://pysqa.readthedocs.io/en/latest/installation.html#pypi-based-installation)
  * [conda-based installation](https://pysqa.readthedocs.io/en/latest/installation.html#conda-based-installation)
* [Queuing Systems](https://pysqa.readthedocs.io/en/latest/queue.html)
  * [Flux](https://pysqa.readthedocs.io/en/latest/queue.html#flux)
  * [LFS](https://pysqa.readthedocs.io/en/latest/queue.html#lfs)
  * [MOAB](https://pysqa.readthedocs.io/en/latest/queue.html#moab)
  * [SGE](https://pysqa.readthedocs.io/en/latest/queue.html#sge)
  * [SLURM](https://pysqa.readthedocs.io/en/latest/queue.html#slurm)
  * [TORQUE](https://pysqa.readthedocs.io/en/latest/queue.html#torque)
* [Python Interface Dynamic](https://pysqa.readthedocs.io/en/latest/example_queue_type.html)
  * [Submit job to queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#submit-job-to-queue)
  * [Show jobs in queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#show-jobs-in-queue)
  * [Delete job from queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#delete-job-from-queue)
* [Python Interface Config](https://pysqa.readthedocs.io/en/latest/example_config.html)
  * [List available queues](https://pysqa.readthedocs.io/en/latest/example_config.html#list-available-queues)
  * [Submit job to queue](https://pysqa.readthedocs.io/en/latest/example_config.html#submit-job-to-queue)
  * [Show jobs in queue](https://pysqa.readthedocs.io/en/latest/example_config.html#show-jobs-in-queue)
  * [Delete job from queue](https://pysqa.readthedocs.io/en/latest/example_config.html#delete-job-from-queue)
* [Command Line Interface](https://pysqa.readthedocs.io/en/latest/command.html)
  * [Submit job](https://pysqa.readthedocs.io/en/latest/command.html#submit-job)
  * [Enable reservation](https://pysqa.readthedocs.io/en/latest/command.html#enable-reservation)
  * [List jobs](https://pysqa.readthedocs.io/en/latest/command.html#list-jobs)
  * [Delete job](https://pysqa.readthedocs.io/en/latest/command.html#delete-job)
  * [List files](https://pysqa.readthedocs.io/en/latest/command.html#list-files)
  * [Help](https://pysqa.readthedocs.io/en/latest/command.html#help)
* [Advanced Configuration](https://pysqa.readthedocs.io/en/latest/advanced.html)
  * [Remote HPC Configuration](https://pysqa.readthedocs.io/en/latest/advanced.html#remote-hpc-configuration)
  * [Access to Multiple HPCs](https://pysqa.readthedocs.io/en/latest/advanced.html#access-to-multiple-hpcs)
* [Debugging](https://pysqa.readthedocs.io/en/latest/debug.html)
  * [Local Queuing System](https://pysqa.readthedocs.io/en/latest/debug.html#local-queuing-system)
  * [Remote HPC](https://pysqa.readthedocs.io/en/latest/debug.html#remote-hpc)

## License
`pysqa` is released under the [BSD license](https://github.com/pyiron/pysqa/blob/main/LICENSE) . It is a spin-off of the 
[pyiron project](https://pyiron.org) therefore if you use `pysqa` for calculation which result in a scientific 
publication, please cite: 

    @article{pyiron-paper,
      title = {pyiron: An integrated development environment for computational materials science},
      journal = {Computational Materials Science},
      volume = {163},
      pages = {24 - 36},
      year = {2019},
      issn = {0927-0256},
      doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},
      url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},
      author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer},
      keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},
    }

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "pysqa",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<3.14,>=3.9",
    "maintainer_email": null,
    "keywords": "pyiron",
    "author": null,
    "author_email": "Jan Janssen <janssen@mpie.de>",
    "download_url": "https://files.pythonhosted.org/packages/8f/36/3c84eb365e5e7ab057f115ee44e679d3514a5490600c91f0ac3cc6b14e4c/pysqa-0.2.2.tar.gz",
    "platform": null,
    "description": "# pysqa\n\n[![Unittests](https://github.com/pyiron/pysqa/actions/workflows/unittest.yml/badge.svg)](https://github.com/pyiron/pysqa/actions/workflows/unittest.yml)\n[![Documentation Status](https://readthedocs.org/projects/pysqa/badge/?version=latest)](https://pysqa.readthedocs.io/en/latest/?badge=latest)\n[![Coverage Status](https://coveralls.io/repos/github/pyiron/pysqa/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pysqa?branch=main)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pysqa/HEAD?labpath=example_config.ipynb)\n\nHigh-performance computing (HPC) does not have to be hard. In this context the aim of the Python Simple Queuing System \nAdapter (`pysqa`) is to simplify the submission of tasks from python to HPC clusters as easy as starting another \n`subprocess` locally. This is achieved based on the assumption that even though modern HPC queuing systems offer a wide \nrange of different configuration options, most users submit the majority of their jobs with very similar parameters. \n\nTherefore, in `pysqa` users define submission script templates once and reuse them to submit many different tasks and \nworkflows afterwards. These templates are defined in the [jinja2 template language](https://palletsprojects.com/p/jinja/),\nso current submission scripts can be easily converted to templates. In addition, to the submission of new tasks to HPC\nqueuing systems, `pysqa` also allows the users to track the progress of their tasks, delete them or enable reservations \nusing the built-in functionality of the queuing system. Finally, `pysqa` enables remote connections to HPC clusters \nusing SSH including support for two factor authentication via [pyauthenticator](https://github.com/jan-janssen/pyauthenticator), \nthis allows the users to submit task from a python process on their local workstation to remote HPC clusters.\n\nAll this functionality is available from both the [Python interface](https://pysqa.readthedocs.io/en/latest/example.html) \nas well as the [command line interface](https://pysqa.readthedocs.io/en/latest/command.html). \n\n## Features\nThe core feature of `pysqa` is the communication to HPC queuing systems including ([Flux](https://pysqa.readthedocs.io/en/latest/queue.html#flux), \n[LFS](https://pysqa.readthedocs.io/en/latest/queue.html#lfs), [MOAB](https://pysqa.readthedocs.io/en/latest/queue.html#moab), \n[SGE](https://pysqa.readthedocs.io/en/latest/queue.html#sge), [SLURM](https://pysqa.readthedocs.io/en/latest/queue.html#slurm) \nand [TORQUE](https://pysqa.readthedocs.io/en/latest/queue.html#torque)). This includes: \n\n* `QueueAdapter().submit_job()` - Submission of new tasks to the queuing system. \n* `QueueAdapter().get_queue_status()` - List of calculation currently waiting or running on the queuing system. \n* `QueueAdapter().delete_job()` - Deleting calculation which are currently waiting or running on the queuing system. \n* `QueueAdapter().queue_list` - List of available queue templates created by the user.\n* `QueueAdapter().config` - Templates to a specific number of cores, run time or other computing resources. With \n  integrated checks if a given submitted task follows these restrictions. \n\nIn addition to these core features, `pysqa` is continuously extended to support more use cases for a larger group of \nusers. These new features include the support for remote queuing systems: \n\n* Remote connection via the secure shell protocol (SSH) to access remote HPC clusters.\n* Transfer of files to and from remote HPC clusters, based on a predefined mapping of the remote file system into the \n  local file system. \n* Support for both individual connections as well as continuous connections depending on the network availability. \n\nFinally, there is current work in progress to support a combination of [multiple local and remote queuing systems](https://pysqa.readthedocs.io/en/latest/advanced.html) \nfrom within `pysqa`, which are represented to the user as a single resource. \n\n## Documentation\n\n* [Installation](https://pysqa.readthedocs.io/en/latest/installation.html)\n  * [pypi-based installation](https://pysqa.readthedocs.io/en/latest/installation.html#pypi-based-installation)\n  * [conda-based installation](https://pysqa.readthedocs.io/en/latest/installation.html#conda-based-installation)\n* [Queuing Systems](https://pysqa.readthedocs.io/en/latest/queue.html)\n  * [Flux](https://pysqa.readthedocs.io/en/latest/queue.html#flux)\n  * [LFS](https://pysqa.readthedocs.io/en/latest/queue.html#lfs)\n  * [MOAB](https://pysqa.readthedocs.io/en/latest/queue.html#moab)\n  * [SGE](https://pysqa.readthedocs.io/en/latest/queue.html#sge)\n  * [SLURM](https://pysqa.readthedocs.io/en/latest/queue.html#slurm)\n  * [TORQUE](https://pysqa.readthedocs.io/en/latest/queue.html#torque)\n* [Python Interface Dynamic](https://pysqa.readthedocs.io/en/latest/example_queue_type.html)\n  * [Submit job to queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#submit-job-to-queue)\n  * [Show jobs in queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#show-jobs-in-queue)\n  * [Delete job from queue](https://pysqa.readthedocs.io/en/latest/example_queue_type.html#delete-job-from-queue)\n* [Python Interface Config](https://pysqa.readthedocs.io/en/latest/example_config.html)\n  * [List available queues](https://pysqa.readthedocs.io/en/latest/example_config.html#list-available-queues)\n  * [Submit job to queue](https://pysqa.readthedocs.io/en/latest/example_config.html#submit-job-to-queue)\n  * [Show jobs in queue](https://pysqa.readthedocs.io/en/latest/example_config.html#show-jobs-in-queue)\n  * [Delete job from queue](https://pysqa.readthedocs.io/en/latest/example_config.html#delete-job-from-queue)\n* [Command Line Interface](https://pysqa.readthedocs.io/en/latest/command.html)\n  * [Submit job](https://pysqa.readthedocs.io/en/latest/command.html#submit-job)\n  * [Enable reservation](https://pysqa.readthedocs.io/en/latest/command.html#enable-reservation)\n  * [List jobs](https://pysqa.readthedocs.io/en/latest/command.html#list-jobs)\n  * [Delete job](https://pysqa.readthedocs.io/en/latest/command.html#delete-job)\n  * [List files](https://pysqa.readthedocs.io/en/latest/command.html#list-files)\n  * [Help](https://pysqa.readthedocs.io/en/latest/command.html#help)\n* [Advanced Configuration](https://pysqa.readthedocs.io/en/latest/advanced.html)\n  * [Remote HPC Configuration](https://pysqa.readthedocs.io/en/latest/advanced.html#remote-hpc-configuration)\n  * [Access to Multiple HPCs](https://pysqa.readthedocs.io/en/latest/advanced.html#access-to-multiple-hpcs)\n* [Debugging](https://pysqa.readthedocs.io/en/latest/debug.html)\n  * [Local Queuing System](https://pysqa.readthedocs.io/en/latest/debug.html#local-queuing-system)\n  * [Remote HPC](https://pysqa.readthedocs.io/en/latest/debug.html#remote-hpc)\n\n## License\n`pysqa` is released under the [BSD license](https://github.com/pyiron/pysqa/blob/main/LICENSE) . It is a spin-off of the \n[pyiron project](https://pyiron.org) therefore if you use `pysqa` for calculation which result in a scientific \npublication, please cite: \n\n    @article{pyiron-paper,\n      title = {pyiron: An integrated development environment for computational materials science},\n      journal = {Computational Materials Science},\n      volume = {163},\n      pages = {24 - 36},\n      year = {2019},\n      issn = {0927-0256},\n      doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},\n      url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},\n      author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and J\u00f6rg Neugebauer},\n      keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},\n    }\n",
    "bugtrack_url": null,
    "license": "BSD 3-Clause License  Copyright (c) 2019, Jan Janssen All rights reserved.  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:  * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.  * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.  * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ",
    "summary": "Simple HPC queuing system adapter for Python on based jinja templates to automate the submission script creation.",
    "version": "0.2.2",
    "project_urls": {
        "Documentation": "https://pysqa.readthedocs.io",
        "Homepage": "https://github.com/pyiron/pysqa",
        "Repository": "https://github.com/pyiron/pysqa"
    },
    "split_keywords": [
        "pyiron"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "121a6fd1cd9c7accdfc0fa05425166e0279b8fa2b8114e8d40ace7b13f12eb31",
                "md5": "4b7487e07fdda377b9f2fe7b9bd8333c",
                "sha256": "ca8d44563d93b302bb3b5dfe754b365a6ae0e0bc905ee80f39993f8c71f1a7f8"
            },
            "downloads": -1,
            "filename": "pysqa-0.2.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4b7487e07fdda377b9f2fe7b9bd8333c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<3.14,>=3.9",
            "size": 42731,
            "upload_time": "2024-11-09T06:33:34",
            "upload_time_iso_8601": "2024-11-09T06:33:34.743882Z",
            "url": "https://files.pythonhosted.org/packages/12/1a/6fd1cd9c7accdfc0fa05425166e0279b8fa2b8114e8d40ace7b13f12eb31/pysqa-0.2.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "8f363c84eb365e5e7ab057f115ee44e679d3514a5490600c91f0ac3cc6b14e4c",
                "md5": "e6509efebb1d9c2df5e27a4f4770c247",
                "sha256": "b09bcb9725c7d2fdd7c61b820c8ab2947716b1b58dbc7d4cbba9223031c60231"
            },
            "downloads": -1,
            "filename": "pysqa-0.2.2.tar.gz",
            "has_sig": false,
            "md5_digest": "e6509efebb1d9c2df5e27a4f4770c247",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<3.14,>=3.9",
            "size": 34601,
            "upload_time": "2024-11-09T06:33:36",
            "upload_time_iso_8601": "2024-11-09T06:33:36.729612Z",
            "url": "https://files.pythonhosted.org/packages/8f/36/3c84eb365e5e7ab057f115ee44e679d3514a5490600c91f0ac3cc6b14e4c/pysqa-0.2.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-09 06:33:36",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "pyiron",
    "github_project": "pysqa",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "pysqa"
}
        
Elapsed time: 0.32915s