----------------------------------------------
pyina: MPI parallel map and cluster scheduling
----------------------------------------------
About Pyina
===========
The ``pyina`` package provides several basic tools to make MPI-based
parallel computing more accessable to the end user. The goal of ``pyina``
is to allow the user to extend their own code to MPI-based parallel
computing with minimal refactoring.
The central element of ``pyina`` is the parallel map algorithm.
``pyina`` currently provides two strategies for executing the parallel-map,
where a strategy is the algorithm for distributing the work list of
jobs across the availble nodes. These strategies can be used *"in-the-raw"*
(i.e. directly) to provide the map algorithm to a user's own mpi-aware code.
Further, in ``pyina.mpi`` ``pyina`` provides pipe and map implementations
(known as *"easy map"*) that hide the MPI internals from the user. With the
*"easy map"*, the user can launch their code in parallel batch mode -- using
standard Python and without ever having to write a line of MPI code.
There are several ways that a user would typically launch their code in
parallel -- directly with ``mpirun`` or ``mpiexec``, or through the use of a
scheduler such as *torque* or *slurm*. ``pyina`` encapsulates several of these
*"launchers"*, and provides a common interface to the different methods of
launching a MPI job.
``pyina`` is part of ``pathos``, a Python framework for heterogeneous computing.
``pyina`` is in active development, so any user feedback, bug reports, comments,
or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/pyina/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.
Major Features
==============
``pyina`` provides a highly configurable parallel map interface
to running MPI jobs, with:
- a map interface that extends the Python ``map`` standard
- the ability to submit batch jobs to a selection of schedulers
- the ability to customize node and process launch configurations
- the ability to launch parallel MPI jobs with standard Python
- ease in selecting different strategies for processing a work list
Current Release
===============
The latest released version of ``pyina`` is available at:
https://pypi.org/project/pyina
``pyina`` is distributed under a 3-clause BSD license.
Development Version
===================
You can get the latest development version with all the shiny new features at:
https://github.com/uqfoundation
If you have a new contribution, please submit a pull request.
Installation
============
``pyina`` can be installed with ``pip``::
$ pip install pyina
A version of MPI must also be installed. Launchers in ``pyina`` that
submit to a scheduler will throw errors if the underlying scheduler is
not available, however a scheduler is not required for ``pyina`` to execute.
Requirements
============
``pyina`` requires:
- ``python`` (or ``pypy``), **>=3.8**
- ``setuptools``, **>=42**
- ``cython``, **>=0.29.30**
- ``numpy``, **>=1.0**
- ``mpi4py``, **>=1.3**
- ``dill``, **>=0.3.8**
- ``pox``, **>=0.3.4**
- ``pathos``, **>=0.3.2**
More Information
================
Probably the best way to get started is to look at the documentation at
http://pyina.rtfd.io. Also see https://github.com/uqfoundation/pyina/tree/master/examples and ``pyina.tests`` for a set of scripts that demonstrate the
configuration and launching of mpi-based parallel jobs using the *"easy map"*
interface. You can run the tests with ``python -m pyina.tests``. A script is
included for querying, setting up, and tearing down an MPI environment, see
``python -m pyina`` for more information. The source code is generally well
documented, so further questions may be resolved by inspecting the code itself.
Please feel free to submit a ticket on github, or ask a question on
stackoverflow (**@Mike McKerns**). If you would like to share how you use
``pyina`` in your work, please send an email (to **mmckerns at uqfoundation dot
org**).
Important classes and functions are found here:
- ``pyina.mpi`` [the map API definition]
- ``pyina.schedulers`` [all available schedulers]
- ``pyina.launchers`` [all available launchers]
Mapping strategies are found here:
- ``pyina.mpi_scatter`` [the scatter-gather strategy]
- ``pyina.mpi_pool`` [the worker pool strategy]
``pyina`` also provides a convience script that helps navigate the
MPI environment. This script can be run from anywhere with::
$ mpi_world
If may also be convienent to set a shell alias for the launch of 'raw'
mpi-python jobs. Set something like the following (for bash)::
$ alias mpython1='mpiexec -np 1 `which python`'
$ alias mpython2='mpiexec -np 2 `which python`'
$ ...
Citation
========
If you use ``pyina`` to do research that leads to publication, we ask that you
acknowledge use of ``pyina`` by citing the following in your publication::
M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
"Building a framework for predictive science", Proceedings of
the 10th Python in Science Conference, 2011;
http://arxiv.org/pdf/1202.1056
Michael McKerns and Michael Aivazis,
"pathos: a framework for heterogeneous computing", 2010- ;
https://uqfoundation.github.io/project/pathos
Please see https://uqfoundation.github.io/project/pathos or
http://arxiv.org/pdf/1202.1056 for further information.
Raw data
{
"_id": null,
"home_page": "https://github.com/uqfoundation/pyina",
"name": "pyina",
"maintainer": "Mike McKerns",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "mmckerns@uqfoundation.org",
"keywords": "",
"author": "Mike McKerns",
"author_email": "mmckerns@uqfoundation.org",
"download_url": "https://files.pythonhosted.org/packages/d7/93/7dd267f05e6e829d2f2cfee47f36a56f6a4e8b35f05ef7c618d714bdad89/pyina-0.2.9.tar.gz",
"platform": "Linux",
"description": "----------------------------------------------\npyina: MPI parallel map and cluster scheduling\n----------------------------------------------\n\nAbout Pyina\n===========\n\nThe ``pyina`` package provides several basic tools to make MPI-based\nparallel computing more accessable to the end user. The goal of ``pyina``\nis to allow the user to extend their own code to MPI-based parallel\ncomputing with minimal refactoring.\n\nThe central element of ``pyina`` is the parallel map algorithm.\n``pyina`` currently provides two strategies for executing the parallel-map,\nwhere a strategy is the algorithm for distributing the work list of\njobs across the availble nodes. These strategies can be used *\"in-the-raw\"*\n(i.e. directly) to provide the map algorithm to a user's own mpi-aware code.\nFurther, in ``pyina.mpi`` ``pyina`` provides pipe and map implementations\n(known as *\"easy map\"*) that hide the MPI internals from the user. With the\n*\"easy map\"*, the user can launch their code in parallel batch mode -- using\nstandard Python and without ever having to write a line of MPI code.\n\nThere are several ways that a user would typically launch their code in\nparallel -- directly with ``mpirun`` or ``mpiexec``, or through the use of a\nscheduler such as *torque* or *slurm*. ``pyina`` encapsulates several of these\n*\"launchers\"*, and provides a common interface to the different methods of\nlaunching a MPI job.\n\n``pyina`` is part of ``pathos``, a Python framework for heterogeneous computing.\n``pyina`` is in active development, so any user feedback, bug reports, comments,\nor suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/pyina/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.\n\n\nMajor Features\n==============\n\n``pyina`` provides a highly configurable parallel map interface\nto running MPI jobs, with:\n\n - a map interface that extends the Python ``map`` standard\n - the ability to submit batch jobs to a selection of schedulers\n - the ability to customize node and process launch configurations\n - the ability to launch parallel MPI jobs with standard Python\n - ease in selecting different strategies for processing a work list\n\n\nCurrent Release\n===============\n\nThe latest released version of ``pyina`` is available at:\n\n https://pypi.org/project/pyina\n\n``pyina`` is distributed under a 3-clause BSD license.\n\n\nDevelopment Version\n===================\n\nYou can get the latest development version with all the shiny new features at:\n\n https://github.com/uqfoundation\n\nIf you have a new contribution, please submit a pull request.\n\n\nInstallation\n============\n\n``pyina`` can be installed with ``pip``::\n\n $ pip install pyina\n\nA version of MPI must also be installed. Launchers in ``pyina`` that\nsubmit to a scheduler will throw errors if the underlying scheduler is\nnot available, however a scheduler is not required for ``pyina`` to execute.\n\n\nRequirements\n============\n\n``pyina`` requires:\n\n - ``python`` (or ``pypy``), **>=3.8**\n - ``setuptools``, **>=42**\n - ``cython``, **>=0.29.30**\n - ``numpy``, **>=1.0**\n - ``mpi4py``, **>=1.3**\n - ``dill``, **>=0.3.8**\n - ``pox``, **>=0.3.4**\n - ``pathos``, **>=0.3.2**\n\n\nMore Information\n================\n\nProbably the best way to get started is to look at the documentation at\nhttp://pyina.rtfd.io. Also see https://github.com/uqfoundation/pyina/tree/master/examples and ``pyina.tests`` for a set of scripts that demonstrate the\nconfiguration and launching of mpi-based parallel jobs using the *\"easy map\"*\ninterface. You can run the tests with ``python -m pyina.tests``. A script is\nincluded for querying, setting up, and tearing down an MPI environment, see\n``python -m pyina`` for more information. The source code is generally well\ndocumented, so further questions may be resolved by inspecting the code itself.\nPlease feel free to submit a ticket on github, or ask a question on\nstackoverflow (**@Mike McKerns**). If you would like to share how you use\n``pyina`` in your work, please send an email (to **mmckerns at uqfoundation dot\norg**).\n\nImportant classes and functions are found here:\n\n - ``pyina.mpi`` [the map API definition]\n - ``pyina.schedulers`` [all available schedulers] \n - ``pyina.launchers`` [all available launchers] \n\nMapping strategies are found here:\n\n - ``pyina.mpi_scatter`` [the scatter-gather strategy]\n - ``pyina.mpi_pool`` [the worker pool strategy]\n\n``pyina`` also provides a convience script that helps navigate the\nMPI environment. This script can be run from anywhere with::\n\n $ mpi_world\n\nIf may also be convienent to set a shell alias for the launch of 'raw'\nmpi-python jobs. Set something like the following (for bash)::\n\n $ alias mpython1='mpiexec -np 1 `which python`'\n $ alias mpython2='mpiexec -np 2 `which python`'\n $ ...\n\n\nCitation\n========\n\nIf you use ``pyina`` to do research that leads to publication, we ask that you\nacknowledge use of ``pyina`` by citing the following in your publication::\n\n M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,\n \"Building a framework for predictive science\", Proceedings of\n the 10th Python in Science Conference, 2011;\n http://arxiv.org/pdf/1202.1056\n\n Michael McKerns and Michael Aivazis,\n \"pathos: a framework for heterogeneous computing\", 2010- ;\n https://uqfoundation.github.io/project/pathos\n\nPlease see https://uqfoundation.github.io/project/pathos or\nhttp://arxiv.org/pdf/1202.1056 for further information.\n\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "MPI parallel map and cluster scheduling",
"version": "0.2.9",
"project_urls": {
"Bug Tracker": "https://github.com/uqfoundation/pyina/issues",
"Documentation": "http://pyina.rtfd.io",
"Download": "https://pypi.org/project/pyina/#files",
"Homepage": "https://github.com/uqfoundation/pyina",
"Source Code": "https://github.com/uqfoundation/pyina"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87319841ac6c17a57e759ae7ca771c3a624f93e6bca955a38d0eb389d00f2fb2",
"md5": "29adf5808d1312edb5a14ad657803d09",
"sha256": "fdf55e2a9cf39ebd830ba732c6e71527fb595a9da23b720d6322be49dc274c2a"
},
"downloads": -1,
"filename": "pyina-0.2.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "29adf5808d1312edb5a14ad657803d09",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 45845,
"upload_time": "2024-01-28T20:31:01",
"upload_time_iso_8601": "2024-01-28T20:31:01.797472Z",
"url": "https://files.pythonhosted.org/packages/87/31/9841ac6c17a57e759ae7ca771c3a624f93e6bca955a38d0eb389d00f2fb2/pyina-0.2.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d7937dd267f05e6e829d2f2cfee47f36a56f6a4e8b35f05ef7c618d714bdad89",
"md5": "497521d7003a0a8e7141421eb6aad53f",
"sha256": "dd0c394f8e01dac20a5e17ef39f5ba1806012ee90fdc6ec22d290ff94ef0035f"
},
"downloads": -1,
"filename": "pyina-0.2.9.tar.gz",
"has_sig": false,
"md5_digest": "497521d7003a0a8e7141421eb6aad53f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 132709,
"upload_time": "2024-01-28T20:31:04",
"upload_time_iso_8601": "2024-01-28T20:31:04.243555Z",
"url": "https://files.pythonhosted.org/packages/d7/93/7dd267f05e6e829d2f2cfee47f36a56f6a4e8b35f05ef7c618d714bdad89/pyina-0.2.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-28 20:31:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "uqfoundation",
"github_project": "pyina",
"travis_ci": true,
"coveralls": true,
"github_actions": false,
"tox": true,
"lcname": "pyina"
}
JSON
