dispy


Namedispy JSON
Version 4.15.2 PyPI version JSON
download
home_pagehttps://dispy.org
SummaryDistributed and Parallel Computing with/for Python.
upload_time2022-10-06 14:26:46
maintainer
docs_urlNone
authorGiridhar Pemmasani
requires_python
licenseApache 2.0
keywords distributed computing parallel processing mapreduce hadoop job scheduler
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            dispy
######

    .. note:: Full documentation for dispy is now available at `dispy.org
              <https://dispy.org>`_.

`dispy <https://dispy.org>`_ is a comprehensive, yet easy
to use framework for creating and using compute clusters to execute computations
in parallel across multiple processors in a single machine (SMP), among many
machines in a cluster, grid or cloud.  dispy is well suited for data parallel
(SIMD) paradigm where a computation is evaluated with different (large) datasets
independently with no communication among computation tasks (except for
computation tasks sending intermediate results to the client).

dispy works with Python versions 2.7+ and 3.1+ on Linux, Mac OS X and Windows; it may
work on other platforms (e.g., FreeBSD and other BSD variants) too.

Features
--------

* dispy is implemented with `pycos <https://pycos.org>`_,
  an independent framework for asynchronous, concurrent, distributed, network
  programming with tasks (without threads). pycos uses non-blocking sockets with
  I/O notification mechanisms epoll, kqueue and poll, and Windows I/O Completion
  Ports (IOCP) for high performance and scalability, so dispy works efficiently
  with a single node or large cluster(s) of nodes. pycos itself has support for
  distributed/parallel computing, including transferring computations, files
  etc., and message passing (for communicating with client and other computation
  tasks).  While dispy can be used to schedule jobs of a computation to get the
  results, pycos can be used to create `distributed communicating processes
  <https://pycos.org/dispycos.html>`_, for broad range of use cases.

* Computations (Python functions or standalone programs) and their
  dependencies (files, Python functions, classes, modules) are
  distributed automatically.

* Computation nodes can be anywhere on the network (local or
  remote). For security, either simple hash based authentication or
  SSL encryption can be used.

* After each execution is finished, the results of execution, output,
  errors and exception trace are made available for further
  processing.

* Nodes may become available dynamically: dispy will schedule jobs
  whenever a node is available and computations can use that node.

* If callback function is provided, dispy executes that function
  when a job is finished; this can be used for processing job
  results as they become available.

* Client-side and server-side fault recovery are supported:

  If user program (client) terminates unexpectedly (e.g., due to
  uncaught exception), the nodes continue to execute scheduled
  jobs. If client-side fault recover option is used when creating a
  cluster, the results of the scheduled (but unfinished at the time of
  crash) jobs for that cluster can be retrieved later.

  If a computation is marked reentrant when a cluster is created and a
  node (server) executing jobs for that computation fails, dispy
  automatically resubmits those jobs to other available nodes.

* dispy can be used in a single process to use all the nodes
  exclusively (with ``JobCluster`` - simpler to use) or in multiple
  processes simultaneously sharing the nodes (with
  ``SharedJobCluster`` and *dispyscheduler* program).

* Cluster can be `monitored and managed
  <https:/dispy.org/httpd.html>`_ with web browser.

Dependencies
------------

dispy requires pycos_ for concurrent, asynchronous network programming with tasks. pycos is
automatically installed if dispy is installed with pip. Under Windows efficient polling notifier
I/O Completion Ports (IOCP) is supported only if `pywin32 <https://github.com/mhammond/pywin32>`_
is installed; otherwise, inefficient *select* notifier is used.

Installation
------------
To install dispy, run::

   python -m pip install dispy

Release Notes
-------------

Short summary of changes for each release can be found at `News
<https://pycos.com/forum/viewforum.php?f=11>`_. Detailed logs / changes are at
github `commits <https://github.com/pgiri/dispy/commits/master>`_.

Authors
-------
* Giridhar Pemmasani

Links
-----
* Documentation is at `dispy.org`_.
* `Examples <https://dispy.org/examples.html>`_.
* `Github (Code Respository) <https://github.com/pgiri/dispy>`_.



            

Raw data

            {
    "_id": null,
    "home_page": "https://dispy.org",
    "name": "dispy",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "distributed computing,parallel processing,mapreduce,hadoop,job scheduler",
    "author": "Giridhar Pemmasani",
    "author_email": "pgiri@yahoo.com",
    "download_url": "https://files.pythonhosted.org/packages/59/88/b2bd984a81db9ba0d73a47645ded9da8e0bcfddc231644f9017668aeabcc/dispy-4.15.2.tar.gz",
    "platform": "any",
    "description": "dispy\n######\n\n    .. note:: Full documentation for dispy is now available at `dispy.org\n              <https://dispy.org>`_.\n\n`dispy <https://dispy.org>`_ is a comprehensive, yet easy\nto use framework for creating and using compute clusters to execute computations\nin parallel across multiple processors in a single machine (SMP), among many\nmachines in a cluster, grid or cloud.  dispy is well suited for data parallel\n(SIMD) paradigm where a computation is evaluated with different (large) datasets\nindependently with no communication among computation tasks (except for\ncomputation tasks sending intermediate results to the client).\n\ndispy works with Python versions 2.7+ and 3.1+ on Linux, Mac OS X and Windows; it may\nwork on other platforms (e.g., FreeBSD and other BSD variants) too.\n\nFeatures\n--------\n\n* dispy is implemented with `pycos <https://pycos.org>`_,\n  an independent framework for asynchronous, concurrent, distributed, network\n  programming with tasks (without threads). pycos uses non-blocking sockets with\n  I/O notification mechanisms epoll, kqueue and poll, and Windows I/O Completion\n  Ports (IOCP) for high performance and scalability, so dispy works efficiently\n  with a single node or large cluster(s) of nodes. pycos itself has support for\n  distributed/parallel computing, including transferring computations, files\n  etc., and message passing (for communicating with client and other computation\n  tasks).  While dispy can be used to schedule jobs of a computation to get the\n  results, pycos can be used to create `distributed communicating processes\n  <https://pycos.org/dispycos.html>`_, for broad range of use cases.\n\n* Computations (Python functions or standalone programs) and their\n  dependencies (files, Python functions, classes, modules) are\n  distributed automatically.\n\n* Computation nodes can be anywhere on the network (local or\n  remote). For security, either simple hash based authentication or\n  SSL encryption can be used.\n\n* After each execution is finished, the results of execution, output,\n  errors and exception trace are made available for further\n  processing.\n\n* Nodes may become available dynamically: dispy will schedule jobs\n  whenever a node is available and computations can use that node.\n\n* If callback function is provided, dispy executes that function\n  when a job is finished; this can be used for processing job\n  results as they become available.\n\n* Client-side and server-side fault recovery are supported:\n\n  If user program (client) terminates unexpectedly (e.g., due to\n  uncaught exception), the nodes continue to execute scheduled\n  jobs. If client-side fault recover option is used when creating a\n  cluster, the results of the scheduled (but unfinished at the time of\n  crash) jobs for that cluster can be retrieved later.\n\n  If a computation is marked reentrant when a cluster is created and a\n  node (server) executing jobs for that computation fails, dispy\n  automatically resubmits those jobs to other available nodes.\n\n* dispy can be used in a single process to use all the nodes\n  exclusively (with ``JobCluster`` - simpler to use) or in multiple\n  processes simultaneously sharing the nodes (with\n  ``SharedJobCluster`` and *dispyscheduler* program).\n\n* Cluster can be `monitored and managed\n  <https:/dispy.org/httpd.html>`_ with web browser.\n\nDependencies\n------------\n\ndispy requires pycos_ for concurrent, asynchronous network programming with tasks. pycos is\nautomatically installed if dispy is installed with pip. Under Windows efficient polling notifier\nI/O Completion Ports (IOCP) is supported only if `pywin32 <https://github.com/mhammond/pywin32>`_\nis installed; otherwise, inefficient *select* notifier is used.\n\nInstallation\n------------\nTo install dispy, run::\n\n   python -m pip install dispy\n\nRelease Notes\n-------------\n\nShort summary of changes for each release can be found at `News\n<https://pycos.com/forum/viewforum.php?f=11>`_. Detailed logs / changes are at\ngithub `commits <https://github.com/pgiri/dispy/commits/master>`_.\n\nAuthors\n-------\n* Giridhar Pemmasani\n\nLinks\n-----\n* Documentation is at `dispy.org`_.\n* `Examples <https://dispy.org/examples.html>`_.\n* `Github (Code Respository) <https://github.com/pgiri/dispy>`_.\n\n\n",
    "bugtrack_url": null,
    "license": "Apache 2.0",
    "summary": "Distributed and Parallel Computing with/for Python.",
    "version": "4.15.2",
    "split_keywords": [
        "distributed computing",
        "parallel processing",
        "mapreduce",
        "hadoop",
        "job scheduler"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "e5b0af9f122a047f68d2f20c12408120",
                "sha256": "8e88bf23673a8269ba34887ebcf4a7c8a51016fbd27f898fefef57dd2257258c"
            },
            "downloads": -1,
            "filename": "dispy-4.15.2.tar.gz",
            "has_sig": false,
            "md5_digest": "e5b0af9f122a047f68d2f20c12408120",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 427923,
            "upload_time": "2022-10-06T14:26:46",
            "upload_time_iso_8601": "2022-10-06T14:26:46.243724Z",
            "url": "https://files.pythonhosted.org/packages/59/88/b2bd984a81db9ba0d73a47645ded9da8e0bcfddc231644f9017668aeabcc/dispy-4.15.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-10-06 14:26:46",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "dispy"
}
        
Elapsed time: 0.01422s