pathos


Namepathos JSON
Version 0.3.3 PyPI version JSON
download
home_pagehttps://github.com/uqfoundation/pathos
Summaryparallel graph management and execution in heterogeneous computing
upload_time2024-09-30 19:30:56
maintainerMike McKerns
docs_urlNone
authorMike McKerns
requires_python>=3.8
licenseBSD-3-Clause
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI
coveralls test coverage
            --------------------------------------------------------------------------
pathos: parallel graph management and execution in heterogeneous computing
--------------------------------------------------------------------------

About the Pathos Framework
==========================

``pathos`` is a framework for heterogeneous computing. It provides a consistent
high-level interface for configuring and launching parallel computations
across heterogeneous resources. ``pathos`` provides configurable launchers for
parallel and distributed computing, where each launcher contains the
syntactic logic to configure and launch jobs in an execution environment.
Examples of launchers that plug into ``pathos`` are: a queue-less MPI-based
launcher (in ``pyina``), a ssh-based launcher (in ``pathos``), and a multi-process
launcher (in ``multiprocess``).

``pathos`` provides a consistent interface for parallel and/or distributed
versions of ``map`` and ``apply`` for each launcher, thus lowering the barrier
for users to extend their code to parallel and/or distributed resources.
The guiding design principle behind ``pathos`` is that ``map`` and ``apply``
should be drop-in replacements in otherwise serial code, and thus switching
to one or more of the ``pathos`` launchers is all that is needed to enable
code to leverage the selected parallel or distributed computing resource.
This not only greatly reduces the time to convert a code to parallel, but it
also enables a single code-base to be maintained instead of requiring
parallel, serial, and distributed versions of a code. ``pathos`` maps can be
nested, thus hierarchical heterogeneous computing is possible by merely
selecting the desired hierarchy of ``map`` and ``pipe`` (``apply``) objects.

The ``pathos`` framework is composed of several interoperating packages:

    - ``dill``: serialize all of Python
    - ``pox``: utilities for filesystem exploration and automated builds
    - ``klepto``: persistent caching to memory, disk, or database
    - ``multiprocess``: better multiprocessing and multithreading in Python
    - ``ppft``: distributed and parallel Python
    - ``pyina``: MPI parallel ``map`` and cluster scheduling
    - ``pathos``: graph management and execution in heterogeneous computing


About Pathos
============

The ``pathos`` package provides a few basic tools to make parallel and
distributed computing more accessible to the end user. The goal of ``pathos``
is to enable the user to extend their own code to parallel and distributed
computing with minimal refactoring.

``pathos`` provides methods for configuring, launching, monitoring, and
controlling a service on a remote host. One of the most basic features
of ``pathos`` is the ability to configure and launch a RPC-based service
on a remote host. ``pathos`` seeds the remote host with the  ``portpicker``
script, which allows the remote host to inform the localhost of a port
that is available for communication.

Beyond the ability to establish a RPC service, and then post requests,
is the ability to launch code in parallel. Unlike parallel computing
performed at the node level (typically with MPI), ``pathos`` enables the
user to launch jobs in parallel across heterogeneous distributed resources.
``pathos`` provides distributed ``map`` and ``pipe`` algorithms, where a mix of
local processors and distributed workers can be selected.  ``pathos``
also provides a very basic automated load balancing service, as well as
the ability for the user to directly select the resources.

The high-level ``pool.map`` interface, yields a ``map`` implementation that
hides the RPC internals from the user. With ``pool.map``, the user can launch
their code in parallel, and as a distributed service, using standard Python
and without writing a line of server or parallel batch code.

RPC servers and communication in general is known to be insecure.  However,
instead of attempting to make the RPC communication itself secure, ``pathos``
provides the ability to automatically wrap any distributes service or
communication in a ssh-tunnel. Ssh is a universally trusted method.
Using ssh-tunnels, ``pathos`` has launched several distributed calculations
on national lab clusters, and to date has performed test calculations
that utilize node-to-node communication between several national lab clusters
and a user's laptop.  ``pathos`` allows the user to configure and launch
at a very atomistic level, through raw access to ssh and scp. 

``pathos`` is the core of a Python framework for heterogeneous computing.
``pathos`` is in active development, so any user feedback, bug reports, comments,
or suggestions are highly appreciated.  A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.


Major Features
==============

``pathos`` provides a configurable distributed parallel ``map`` interface
to launching RPC service calls, with:

    - a ``map`` interface that meets and extends the Python ``map`` standard
    - the ability to submit service requests to a selection of servers
    - the ability to tunnel server communications with ssh

The ``pathos`` core is built on low-level communication to remote hosts using
ssh. The interface to ssh, scp, and ssh-tunneled connections can:

    - configure and launch remote processes with ssh
    - configure and copy file objects with scp
    - establish an tear-down a ssh-tunnel

To get up and running quickly, ``pathos`` also provides infrastructure to:

    - easily establish a ssh-tunneled connection to a RPC server


Current Release
===============

The latest released version of ``pathos`` is available from:

    https://pypi.org/project/pathos

``pathos`` is distributed under a 3-clause BSD license.


Development Version
===================

You can get the latest development version with all the shiny new features at:

    https://github.com/uqfoundation

If you have a new contribution, please submit a pull request.


Installation
============

``pathos`` can be installed with ``pip``::

    $ pip install pathos


Requirements
============

``pathos`` requires:

    - ``python`` (or ``pypy``), **>=3.8**
    - ``setuptools``, **>=42**
    - ``pox``, **>=0.3.5**
    - ``dill``, **>=0.3.9**
    - ``ppft``, **>=1.7.6.9**
    - ``multiprocess``, **>=0.70.17**


More Information
================

Probably the best way to get started is to look at the documentation at
http://pathos.rtfd.io. Also see ``pathos.tests`` and https://github.com/uqfoundation/pathos/tree/master/examples for a set of scripts that demonstrate the
configuration and launching of communications with ssh and scp, and demonstrate
the configuration and execution of jobs in a hierarchical parallel workflow.
You can run the test suite with ``python -m pathos.tests``. Tunnels and other
connections to remote servers can be established with the ``pathos_connect``
script (or with ``python -m pathos``). See ``pathos_connect --help`` for more
information.  ``pathos`` also provides a ``portpicker`` script to select an
open port (also available with ``python -m pathos.portpicker``). The source
code is generally well documented, so further questions may be resolved by
inspecting the code itself.  Please feel free to submit a ticket on github,
or ask a question on stackoverflow (**@Mike McKerns**). If you would like to
share how you use ``pathos`` in your work, please send an email (to **mmckerns
at uqfoundation dot org**).

Important classes and functions are found here:

    - ``pathos.abstract_launcher``           [the worker pool API definition]
    - ``pathos.pools``                       [all of the pathos worker pools]
    - ``pathos.core``                        [the high-level command interface] 
    - ``pathos.hosts``                       [the hostname registry interface] 
    - ``pathos.serial.SerialPool``           [the serial Python worker pool]
    - ``pathos.parallel.ParallelPool``       [the parallelpython worker pool]
    - ``pathos.multiprocessing.ProcessPool`` [the multiprocessing worker pool]
    - ``pathos.threading.ThreadPool``        [the multithreading worker pool]
    - ``pathos.connection.Pipe``             [the launcher base class]
    - ``pathos.secure.Pipe``                 [the secure launcher base class]
    - ``pathos.secure.Copier``               [the secure copier  base class]
    - ``pathos.secure.Tunnel``               [the secure tunnel base class]
    - ``pathos.selector.Selector``           [the selector base class]
    - ``pathos.server.Server``               [the server base class]
    - ``pathos.profile``                     [profiling in threads and processes]
    - ``pathos.maps``                        [standalone map instances]

``pathos`` also provides two convenience scripts that are used to establish
secure distributed connections. These scripts are installed to a directory
on the user's ``$PATH``, and thus can be run from anywhere:

    - ``portpicker``                         [get the portnumber of an open port]
    - ``pathos_connect``                     [establish tunnel and/or RPC server]

Typing ``--help`` as an argument to any of the above scripts will print out an
instructive help message.


Citation
========

If you use ``pathos`` to do research that leads to publication, we ask that you
acknowledge use of ``pathos`` by citing the following in your publication::

    M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
    "Building a framework for predictive science", Proceedings of
    the 10th Python in Science Conference, 2011;
    http://arxiv.org/pdf/1202.1056

    Michael McKerns and Michael Aivazis,
    "pathos: a framework for heterogeneous computing", 2010- ;
    https://uqfoundation.github.io/project/pathos

Please see https://uqfoundation.github.io/project/pathos or
http://arxiv.org/pdf/1202.1056 for further information.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/uqfoundation/pathos",
    "name": "pathos",
    "maintainer": "Mike McKerns",
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "mmckerns@uqfoundation.org",
    "keywords": null,
    "author": "Mike McKerns",
    "author_email": "mmckerns@uqfoundation.org",
    "download_url": "https://files.pythonhosted.org/packages/5b/a4/6274bddc49a00873d3269b7612c1553763bae6466c0c82913e16810abd51/pathos-0.3.3.tar.gz",
    "platform": "Linux",
    "description": "--------------------------------------------------------------------------\npathos: parallel graph management and execution in heterogeneous computing\n--------------------------------------------------------------------------\n\nAbout the Pathos Framework\n==========================\n\n``pathos`` is a framework for heterogeneous computing. It provides a consistent\nhigh-level interface for configuring and launching parallel computations\nacross heterogeneous resources. ``pathos`` provides configurable launchers for\nparallel and distributed computing, where each launcher contains the\nsyntactic logic to configure and launch jobs in an execution environment.\nExamples of launchers that plug into ``pathos`` are: a queue-less MPI-based\nlauncher (in ``pyina``), a ssh-based launcher (in ``pathos``), and a multi-process\nlauncher (in ``multiprocess``).\n\n``pathos`` provides a consistent interface for parallel and/or distributed\nversions of ``map`` and ``apply`` for each launcher, thus lowering the barrier\nfor users to extend their code to parallel and/or distributed resources.\nThe guiding design principle behind ``pathos`` is that ``map`` and ``apply``\nshould be drop-in replacements in otherwise serial code, and thus switching\nto one or more of the ``pathos`` launchers is all that is needed to enable\ncode to leverage the selected parallel or distributed computing resource.\nThis not only greatly reduces the time to convert a code to parallel, but it\nalso enables a single code-base to be maintained instead of requiring\nparallel, serial, and distributed versions of a code. ``pathos`` maps can be\nnested, thus hierarchical heterogeneous computing is possible by merely\nselecting the desired hierarchy of ``map`` and ``pipe`` (``apply``) objects.\n\nThe ``pathos`` framework is composed of several interoperating packages:\n\n    - ``dill``: serialize all of Python\n    - ``pox``: utilities for filesystem exploration and automated builds\n    - ``klepto``: persistent caching to memory, disk, or database\n    - ``multiprocess``: better multiprocessing and multithreading in Python\n    - ``ppft``: distributed and parallel Python\n    - ``pyina``: MPI parallel ``map`` and cluster scheduling\n    - ``pathos``: graph management and execution in heterogeneous computing\n\n\nAbout Pathos\n============\n\nThe ``pathos`` package provides a few basic tools to make parallel and\ndistributed computing more accessible to the end user. The goal of ``pathos``\nis to enable the user to extend their own code to parallel and distributed\ncomputing with minimal refactoring.\n\n``pathos`` provides methods for configuring, launching, monitoring, and\ncontrolling a service on a remote host. One of the most basic features\nof ``pathos`` is the ability to configure and launch a RPC-based service\non a remote host. ``pathos`` seeds the remote host with the  ``portpicker``\nscript, which allows the remote host to inform the localhost of a port\nthat is available for communication.\n\nBeyond the ability to establish a RPC service, and then post requests,\nis the ability to launch code in parallel. Unlike parallel computing\nperformed at the node level (typically with MPI), ``pathos`` enables the\nuser to launch jobs in parallel across heterogeneous distributed resources.\n``pathos`` provides distributed ``map`` and ``pipe`` algorithms, where a mix of\nlocal processors and distributed workers can be selected.  ``pathos``\nalso provides a very basic automated load balancing service, as well as\nthe ability for the user to directly select the resources.\n\nThe high-level ``pool.map`` interface, yields a ``map`` implementation that\nhides the RPC internals from the user. With ``pool.map``, the user can launch\ntheir code in parallel, and as a distributed service, using standard Python\nand without writing a line of server or parallel batch code.\n\nRPC servers and communication in general is known to be insecure.  However,\ninstead of attempting to make the RPC communication itself secure, ``pathos``\nprovides the ability to automatically wrap any distributes service or\ncommunication in a ssh-tunnel. Ssh is a universally trusted method.\nUsing ssh-tunnels, ``pathos`` has launched several distributed calculations\non national lab clusters, and to date has performed test calculations\nthat utilize node-to-node communication between several national lab clusters\nand a user's laptop.  ``pathos`` allows the user to configure and launch\nat a very atomistic level, through raw access to ssh and scp. \n\n``pathos`` is the core of a Python framework for heterogeneous computing.\n``pathos`` is in active development, so any user feedback, bug reports, comments,\nor suggestions are highly appreciated.  A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.\n\n\nMajor Features\n==============\n\n``pathos`` provides a configurable distributed parallel ``map`` interface\nto launching RPC service calls, with:\n\n    - a ``map`` interface that meets and extends the Python ``map`` standard\n    - the ability to submit service requests to a selection of servers\n    - the ability to tunnel server communications with ssh\n\nThe ``pathos`` core is built on low-level communication to remote hosts using\nssh. The interface to ssh, scp, and ssh-tunneled connections can:\n\n    - configure and launch remote processes with ssh\n    - configure and copy file objects with scp\n    - establish an tear-down a ssh-tunnel\n\nTo get up and running quickly, ``pathos`` also provides infrastructure to:\n\n    - easily establish a ssh-tunneled connection to a RPC server\n\n\nCurrent Release\n===============\n\nThe latest released version of ``pathos`` is available from:\n\n    https://pypi.org/project/pathos\n\n``pathos`` is distributed under a 3-clause BSD license.\n\n\nDevelopment Version\n===================\n\nYou can get the latest development version with all the shiny new features at:\n\n    https://github.com/uqfoundation\n\nIf you have a new contribution, please submit a pull request.\n\n\nInstallation\n============\n\n``pathos`` can be installed with ``pip``::\n\n    $ pip install pathos\n\n\nRequirements\n============\n\n``pathos`` requires:\n\n    - ``python`` (or ``pypy``), **>=3.8**\n    - ``setuptools``, **>=42**\n    - ``pox``, **>=0.3.5**\n    - ``dill``, **>=0.3.9**\n    - ``ppft``, **>=1.7.6.9**\n    - ``multiprocess``, **>=0.70.17**\n\n\nMore Information\n================\n\nProbably the best way to get started is to look at the documentation at\nhttp://pathos.rtfd.io. Also see ``pathos.tests`` and https://github.com/uqfoundation/pathos/tree/master/examples for a set of scripts that demonstrate the\nconfiguration and launching of communications with ssh and scp, and demonstrate\nthe configuration and execution of jobs in a hierarchical parallel workflow.\nYou can run the test suite with ``python -m pathos.tests``. Tunnels and other\nconnections to remote servers can be established with the ``pathos_connect``\nscript (or with ``python -m pathos``). See ``pathos_connect --help`` for more\ninformation.  ``pathos`` also provides a ``portpicker`` script to select an\nopen port (also available with ``python -m pathos.portpicker``). The source\ncode is generally well documented, so further questions may be resolved by\ninspecting the code itself.  Please feel free to submit a ticket on github,\nor ask a question on stackoverflow (**@Mike McKerns**). If you would like to\nshare how you use ``pathos`` in your work, please send an email (to **mmckerns\nat uqfoundation dot org**).\n\nImportant classes and functions are found here:\n\n    - ``pathos.abstract_launcher``           [the worker pool API definition]\n    - ``pathos.pools``                       [all of the pathos worker pools]\n    - ``pathos.core``                        [the high-level command interface] \n    - ``pathos.hosts``                       [the hostname registry interface] \n    - ``pathos.serial.SerialPool``           [the serial Python worker pool]\n    - ``pathos.parallel.ParallelPool``       [the parallelpython worker pool]\n    - ``pathos.multiprocessing.ProcessPool`` [the multiprocessing worker pool]\n    - ``pathos.threading.ThreadPool``        [the multithreading worker pool]\n    - ``pathos.connection.Pipe``             [the launcher base class]\n    - ``pathos.secure.Pipe``                 [the secure launcher base class]\n    - ``pathos.secure.Copier``               [the secure copier  base class]\n    - ``pathos.secure.Tunnel``               [the secure tunnel base class]\n    - ``pathos.selector.Selector``           [the selector base class]\n    - ``pathos.server.Server``               [the server base class]\n    - ``pathos.profile``                     [profiling in threads and processes]\n    - ``pathos.maps``                        [standalone map instances]\n\n``pathos`` also provides two convenience scripts that are used to establish\nsecure distributed connections. These scripts are installed to a directory\non the user's ``$PATH``, and thus can be run from anywhere:\n\n    - ``portpicker``                         [get the portnumber of an open port]\n    - ``pathos_connect``                     [establish tunnel and/or RPC server]\n\nTyping ``--help`` as an argument to any of the above scripts will print out an\ninstructive help message.\n\n\nCitation\n========\n\nIf you use ``pathos`` to do research that leads to publication, we ask that you\nacknowledge use of ``pathos`` by citing the following in your publication::\n\n    M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,\n    \"Building a framework for predictive science\", Proceedings of\n    the 10th Python in Science Conference, 2011;\n    http://arxiv.org/pdf/1202.1056\n\n    Michael McKerns and Michael Aivazis,\n    \"pathos: a framework for heterogeneous computing\", 2010- ;\n    https://uqfoundation.github.io/project/pathos\n\nPlease see https://uqfoundation.github.io/project/pathos or\nhttp://arxiv.org/pdf/1202.1056 for further information.\n",
    "bugtrack_url": null,
    "license": "BSD-3-Clause",
    "summary": "parallel graph management and execution in heterogeneous computing",
    "version": "0.3.3",
    "project_urls": {
        "Bug Tracker": "https://github.com/uqfoundation/pathos/issues",
        "Documentation": "http://pathos.rtfd.io",
        "Download": "https://pypi.org/project/pathos/#files",
        "Homepage": "https://github.com/uqfoundation/pathos",
        "Source Code": "https://github.com/uqfoundation/pathos"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "77f6a459cf58ff6b2d1c0a1961ee7084f4bb549d50e288f064e7e7be5ae3a7ab",
                "md5": "b91bf81b3dbf522a2443187a1f113704",
                "sha256": "e04616c6448608ad1f809360be22e3f2078d949a36a81e6991da6c2dd1f82513"
            },
            "downloads": -1,
            "filename": "pathos-0.3.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b91bf81b3dbf522a2443187a1f113704",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 82142,
            "upload_time": "2024-09-30T19:30:55",
            "upload_time_iso_8601": "2024-09-30T19:30:55.241467Z",
            "url": "https://files.pythonhosted.org/packages/77/f6/a459cf58ff6b2d1c0a1961ee7084f4bb549d50e288f064e7e7be5ae3a7ab/pathos-0.3.3-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5ba46274bddc49a00873d3269b7612c1553763bae6466c0c82913e16810abd51",
                "md5": "5369d9e5aa032b9014b085c33a46f8f8",
                "sha256": "dcb2a5f321aa34ca541c1c1861011ea49df357bb908379c21dd5741f666e0a58"
            },
            "downloads": -1,
            "filename": "pathos-0.3.3.tar.gz",
            "has_sig": false,
            "md5_digest": "5369d9e5aa032b9014b085c33a46f8f8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 166953,
            "upload_time": "2024-09-30T19:30:56",
            "upload_time_iso_8601": "2024-09-30T19:30:56.365514Z",
            "url": "https://files.pythonhosted.org/packages/5b/a4/6274bddc49a00873d3269b7612c1553763bae6466c0c82913e16810abd51/pathos-0.3.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-09-30 19:30:56",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "uqfoundation",
    "github_project": "pathos",
    "travis_ci": true,
    "coveralls": true,
    "github_actions": false,
    "tox": true,
    "lcname": "pathos"
}
        
Elapsed time: 0.35150s