| Name | slurminade JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | |
| Summary | A decorator-based slurm runner. |
| upload_time | 2024-02-27 15:42:07 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | |
| keywords |
slurm
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
setuptools
simple-slurm
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
slurminade - A decorator-based slurm runner for Python-code.
============================================================
|PyPI version| |CI status| |License|
*slurminade* makes using the workload manager
`slurm <https://slurm.schedmd.com/documentation.html>`__ with Python
beautiful. It is based on
`simple_slurm <https://github.com/amq92/simple_slurm>`__, but instead of
just allowing to comfortably execute shell commands in slurm, it allows
to directly distribute Python-functions. A function decorated with
``@slurminade.slurmify(partition="alg")`` will automatically be executed
by a node of the partition ``alg`` by just calling
``.distribute(yes_also_args_are_allowed)``. The general idea is that the
corresponding Python-code exists on both machines, thus, the slurm-node
can also call the functions of the original code if you tell if which
one and what arguments to use. This is similar to
`celery <https://github.com/celery/celery>`__ but you do not need to
install anything, just make sure the same Python-environment is
available on the nodes (usually the case in a proper slurm setup).
Please check the documentation of
`simple_slurm <https://github.com/amq92/simple_slurm>`__ to get to know
more about the possible parameters. You can also call simple_slurm
directly by ``srun`` and ``sbatch`` (automatically with the
configuration specified with slurminade).
*slurminade* has two design goals:
1. Pythonic slurm: Allowing to use slurm in a Pythonic-way, without any
shell commands etc.
2. Compatibility: Scripts can also run without slurm. You can share a
script and also people without slurm can execute it without any
changes.
We use it to empirically evaluate optimization algorithms for research
papers on hundreds of instances that can require 15min each to solve.
With slurminade, we can distribute the workload by just changing a few
lines of code in our local Python scripts (those that you use for
probing and development before running big experiments). An example of
such a usage can be found here: `Example of an empirical algorithm
performance study for graph coloring heuristics using slurminade and
AlgBench <https://github.com/d-krupke/AlgBench/tree/main/examples/graph_coloring>`__.
You will find the `original
runner <https://github.com/d-krupke/AlgBench/blob/main/examples/graph_coloring/02_run_benchmark.py>`__
and the `slurmified
runner <https://github.com/d-krupke/AlgBench/blob/main/examples/graph_coloring/02b_run_benchmark_with_slurminade.py>`__,
showing the simplicity of distributing your experiments with slurminade.
A simple script could look like this:
.. code:: python
import slurminade
slurminade.update_default_configuration(
partition="alg", exclusive=True
) # global options for slurm
# If no slurm environment is found, the functions are called directly to make scripts
# compatible with any environment.
# You can enforce slurm with `slurminade.set_dispatcher(slurminade.SlurmDispatcher())`
@slurminade.node_setup
def setup():
print("I will run automatically on every slurm node at the beginning!")
# use this decorator to make a function distributable with slurm
@slurminade.slurmify(
constraint="alggen02"
) # function specific options can be specified
def prepare():
print("Prepare")
@slurminade.slurmify()
def f(foobar):
print(f"f({foobar})")
@slurminade.slurmify()
def clean_up():
print("Clean up")
if __name__ == "__main__":
prepare.distribute()
slurminade.join() # make sure that no job runs before prepare has finished
with slurminade.JobBundling(max_size=20): # automatically bundles up to 20 tasks
# run 100x f after `prepare` has finished
for i in range(100):
f.distribute(i)
slurminade.join() # make sure that the clean up jobs runs after all f-jobs have finished
clean_up.distribute()
If slurm is not available, ``distribute`` results in a local function
call. Analogous for ``srun`` and ``sbatch`` (giving some extra value on
top of just forwarding to *simple_slurm*).
.. warning::
Always use ``JobBundling`` when distributing many small tasks to few nodes. Slurm
jobs have a certain overhead and you do not want to spam your
infrastructure with too many jobs. However, function calls
joined by ``JobBundling`` are considered as a single job by slurm, thus,
not shared across nodes.
**What are the limitations of slurminade?** Slurminade reconstructs the
environment by basically loading the code on the slurm node (without the
``__main__``-part) and then calling the slurmified function with
parameters serialized as JSONSs. This means that the code must be
written in a common ``.py``-file and all (distributed) function
arguments must be JSON-serializable. Also, the function must not use any
global state (e.g., global variables, file or database connections)
initialized in the ``__main__``-part. Additionally, the
Python-environment must be available under the same path on the slurm
node as slurminade will use the same paths on the slurm node to
reconstruct the environment (allowing to use virtual environments).
**Does slurminade work with Python 2?** No, it is a Python 3 project. We
tested it with Python 3.7 and higher.
**Does slurminade work with Windows?** Probably not, but I never saw a
slurm cluster running on Windows. The (automatic) slurm-less mode should
work on Windows. So your code will run, but all function calls will be
local.
**Are multi-file projects supported?** Yes, as long as the files are
available on the slurm node.
**Does slurminade work with virtual environments?** Yes. We recommend to
use *slurminade* with `conda <https://docs.conda.io/en/latest/>`__. We
have not tested it with other virtual environments.
**Can I run my slurmified code outside a slurm environment?** Yes, if
you do not have slurm, the distributed functions are run as normal
Python function calls. This means that you can share the same code with
people that do not have slurm. It was important to us that the
experimental evaluations we run on our slurm cluster can also be run in
a common Python environment by reviewers without any changes.
**Can I receive the return value of a slurmified function?** No, the
return value is not transmitted back to the caller. Note that the
distribute-calls are non-blocking, i.e., the function returns
immediately. Return values could be implemented via a *Promise*-object
like for other distributed computing frameworks, but we did not see the
need for it yet. We are usually saving the results in a database or
files, e.g., using `AlgBench <https://github.com/d-krupke/AlgBench>`__.
**Can I use command line arguments ``sys.argv`` in my scripts?** Yes,
but only in your ``__main__``-part. The arguments are not transmitted to
the slurm nodes as they are not part of the function call. You can add
these as normal function arguments to your slurmified functions if
needed. It is important that your global objects to not rely on these
arguments for initialization, as the ``__main__``-part is not executed
on the slurm node. It is theoretically possible to transmit the
arguments to the slurm node, but we did not see the need for it. Let us
know if you need it and we may implement it.
The code is super simple and open source, don’t be afraid to create a
fork that fits your own needs.
.. note::
Talk with you system administrator or supervisor to get the
proper slurm configuration.
Installation
------------
You can install *slurminade* with ``pip install slurminade``.
Usage
-----
You can set task specific slurm arguments within the decorator, e.g.,
``@slurminade.slurmify(constraint="alggen03")``. These arguments are
directly passed to *simple_slurm*, such that all its arguments are
supported.
In order for *slurminade* to work, the code needs to be in a Python
file/project shared by all slurm-nodes. Otherwise, *slurminade* will not
find the corresponding function. The slurmified functions also must be
importable, i.e., on the top level. Currently, all function names must
be unique as *slurminade* will only transmit the function’s name.
Don’t do:
---------
Bad: Non blocking system calls
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: python
import slurminade
import os
import subprocess
@slurminade.slurmify()
def run_shell_command():
# non-blocking system call
subprocess.Popen("complex call")
# BAD! The system call will run outside of slurm! The slurm task directly terminates.
instead use
.. code:: python
import slurminade
if __name__ == "__main__":
slurminade.sbatch(
"complex call"
) # forwards your call to simple_slurm that is better used for such things.
Bad: Global variables in the ``__main__`` part
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: python
import slurminade
FLAG = True
@slurminade.slurmify()
def bad_global(args):
if FLAG: # BAD! Will be True because the __main__ Part is not executed on the node.
pass
else:
pass
if __name__ == "__main__":
FLAG = False
bad_global.distribute("args")
instead do
.. code:: python
import slurminade
@slurminade.slurmify()
def bad_global(
args, FLAG
): # Now the flag is passed correctly as an argument. Note that only json-compatible arguments are possible.
if FLAG:
pass
else:
pass
# Without the `if`, the node would also execute this part (*slurminade* will abort automatically)
if __name__ == "__main__":
FLAG = False
bad_global.distribute("args", FLAG)
..
.. warning::
The same is true for any global state such as file or
database connections. You can use global variables, but be wary of
side effects.
Error: Complex objects as arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: python
import slurminade
@slurminade.slurmify()
def sec_order_func(func):
func()
def f():
print("hello")
def g():
print("world!")
if __name__ == "__main__":
sec_order_func.distribute(f) # will throw an exception
sec_order_func.distribute(g)
Instead, create individual slurmified functions for each call or pass a
simple identifier that lets the function deduce, what to do, e.g., a
switch-case. If you really need to pass complex objects, you could also
pickle the object and only pass the file name.
Default configuration
---------------------
You can set up a default configuration in
``~/.slurminade_default.json``. This should simply be a dictionary of
arguments for *simple_slurm*. For example
.. code:: json
{
"partition": "alg"
}
The current version checks the following files and overwrites values in the following order:
1. ``~/.slurminade_default.json``
2. ``~/$XDG_CONFIG_HOME/slurminade/.slurminade_default.json``
3. ``./.slurminade_default.json``
Debugging
---------
You can use
.. code:: python
import slurminade
slurminade.set_dispatcher(slurminade.TestDispatcher())
to see the serialization or
.. code:: python
import slurminade
slurminade.set_dispatcher(slurminade.SubprocessDispatcher())
to distribute the tasks without slurm using subprocesses.
If there is a bug, you will directly see it in the output (at least for
most bugs).
Project structure
-----------------
The project is reasonably easy:
- bundling.py: Contains code for bundling tasks, so we don’t spam slurm
with too many.
- conf.py: Contains code for managing the configuration of slurm.
- dispatcher.py: Contains code for actually dispatching tasks to slurm.
- execute.py: Contains code to execute the task on the slurm node.
- function.py: Contains the code for making a function
slurm-compatible.
- function_map.py: Saves all the slurmified functions.
- guard.py: Contains code to prevent you accidentally DDoSing your
infrastructure.
- options.py: Contains a simple data structure to save slurm options.
Changes
-------
- 1.1.0: Slurminade can now be called from iPython, too! `exec` has been renamed `shell` to prevent confusion with the Python call `exec` which will evaluate a string as Python code.
- 1.0.1: Dispatcher now return jobs references instead of job ids. This allows to do some fancier stuff in the future, when the jobs infos are only available a short time after the job has been submitted.
- 0.10.1: FIX: Listing functions will no longer execute setup functions.
- 0.10.0: `Batch` is now named `JobBundling`. There is a method `join` for easier synchronization. `exec` allows to executed commands just like `srun` and `sbatch`, but uniform syntax with other slurmified functions. Functions can now also be called with `distribute_and_wait`. If you call `python3 -m slurminade.check --partition YOUR_PARTITION --constraint YOUR_CONSTRAINT` you can check if your slurm configuration is running correctly.
- 0.9.0: Lots of improvements.
- 0.8.1: Bugfix and automatic detection of wrong usage when using ``Batch`` with ``wait_for``.
- 0.8.0: Added extensive logging and improved typing.
- 0.7.0: Warning if a Batch is flushed multiple times, as we noticed this to be a common indentation error.
- 0.6.2: Fixes recursive distribution guard, which seemed to be broken.
- 0.6.1: Bugfixes in naming
- 0.6.0: Autmatic naming of tasks.
- 0.5.5: Fixing bug guard bug in subprocess dispatcher.
- 0.5.4: Dispatched function calls that are too long for the command
line now use a temporary file instead.
- 0.5.3: Fixed a bug that caused the dispatch limit to have no effect.
- 0.5.2: Added pyproject.toml for PEP compliance
- 0.5.1: ``Batch`` will now flush on delete, in case you forgot.
- 0.5.0:
- Functions now have a ``wait_for``-option and return job ids.
- Braking changes: Batches have a new API.
- ``add`` is no longer needed.
- ``AutoBatch`` is now called ``Batch``.
- Fundamental code changes under the hood.
- <0.5.0:
- Lots of experiments on finding the right interface.
Contributors
-------------
This project is developed at the Algorithms Group at TU Braunschweig,
Germany. The lead developer is `Dominik Krupke <https://krupke.cc>`__.
Further contributors are Matthias Konitzny and Patrick Blumenberg.
.. |PyPI version| image:: https://badge.fury.io/py/slurminade.svg
:target: https://badge.fury.io/py/slurminade
.. |CI status| image:: https://github.com/d-krupke/slurminade/actions/workflows/pytest.yml/badge.svg
:target: https://github.com/d-krupke/slurminade
.. |License| image:: https://img.shields.io/github/license/d-krupke/slurminade
:target: https://github.com/d-krupke/slurminade
Similar Projects
----------------
* This project is greatly inspired by `Celery <https://github.com/celery/celery>`__, but does not require any additional infrastructure except for slurm.
* If you want a more powerful library to, e.g., also distribute lambdas or functions with complex arguments, check out `submitit <https://github.com/facebookincubator/submitit/>`__. It is a great project, and we may use it as a backend in the future. However, it does not support the slurm-less mode and can easily hide non-deterministic errors. Slurminade on the other hand is restricted on purpose to write reproducible scripts that can also be run without slurm.
Raw data
{
"_id": null,
"home_page": "",
"name": "slurminade",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "slurm",
"author": "",
"author_email": "\"TU Braunschweig, IBR, Algorithms Group (Dominik Krupke)\" <krupke@ibr.cs.tu-bs.de>",
"download_url": "https://files.pythonhosted.org/packages/ab/79/934bb10b01a8f1cbca5c11b720da5294723685159f3d4f71c8b950dbb46c/slurminade-1.1.0.tar.gz",
"platform": null,
"description": "slurminade - A decorator-based slurm runner for Python-code.\n============================================================\n\n|PyPI version| |CI status| |License|\n\n*slurminade* makes using the workload manager\n`slurm <https://slurm.schedmd.com/documentation.html>`__ with Python\nbeautiful. It is based on\n`simple_slurm <https://github.com/amq92/simple_slurm>`__, but instead of\njust allowing to comfortably execute shell commands in slurm, it allows\nto directly distribute Python-functions. A function decorated with\n``@slurminade.slurmify(partition=\"alg\")`` will automatically be executed\nby a node of the partition ``alg`` by just calling\n``.distribute(yes_also_args_are_allowed)``. The general idea is that the\ncorresponding Python-code exists on both machines, thus, the slurm-node\ncan also call the functions of the original code if you tell if which\none and what arguments to use. This is similar to\n`celery <https://github.com/celery/celery>`__ but you do not need to\ninstall anything, just make sure the same Python-environment is\navailable on the nodes (usually the case in a proper slurm setup).\n\nPlease check the documentation of\n`simple_slurm <https://github.com/amq92/simple_slurm>`__ to get to know\nmore about the possible parameters. You can also call simple_slurm\ndirectly by ``srun`` and ``sbatch`` (automatically with the\nconfiguration specified with slurminade).\n\n*slurminade* has two design goals:\n\n1. Pythonic slurm: Allowing to use slurm in a Pythonic-way, without any\n shell commands etc.\n2. Compatibility: Scripts can also run without slurm. You can share a\n script and also people without slurm can execute it without any\n changes.\n\nWe use it to empirically evaluate optimization algorithms for research\npapers on hundreds of instances that can require 15min each to solve.\nWith slurminade, we can distribute the workload by just changing a few\nlines of code in our local Python scripts (those that you use for\nprobing and development before running big experiments). An example of\nsuch a usage can be found here: `Example of an empirical algorithm\nperformance study for graph coloring heuristics using slurminade and\nAlgBench <https://github.com/d-krupke/AlgBench/tree/main/examples/graph_coloring>`__.\nYou will find the `original\nrunner <https://github.com/d-krupke/AlgBench/blob/main/examples/graph_coloring/02_run_benchmark.py>`__\nand the `slurmified\nrunner <https://github.com/d-krupke/AlgBench/blob/main/examples/graph_coloring/02b_run_benchmark_with_slurminade.py>`__,\nshowing the simplicity of distributing your experiments with slurminade.\n\nA simple script could look like this:\n\n.. code:: python\n\n import slurminade\n\n slurminade.update_default_configuration(\n partition=\"alg\", exclusive=True\n ) # global options for slurm\n\n # If no slurm environment is found, the functions are called directly to make scripts\n # compatible with any environment.\n # You can enforce slurm with `slurminade.set_dispatcher(slurminade.SlurmDispatcher())`\n\n\n @slurminade.node_setup\n def setup():\n print(\"I will run automatically on every slurm node at the beginning!\")\n\n\n # use this decorator to make a function distributable with slurm\n @slurminade.slurmify(\n constraint=\"alggen02\"\n ) # function specific options can be specified\n def prepare():\n print(\"Prepare\")\n\n\n @slurminade.slurmify()\n def f(foobar):\n print(f\"f({foobar})\")\n\n\n @slurminade.slurmify()\n def clean_up():\n print(\"Clean up\")\n\n\n if __name__ == \"__main__\":\n prepare.distribute()\n slurminade.join() # make sure that no job runs before prepare has finished\n with slurminade.JobBundling(max_size=20): # automatically bundles up to 20 tasks\n # run 100x f after `prepare` has finished\n for i in range(100):\n f.distribute(i)\n\n slurminade.join() # make sure that the clean up jobs runs after all f-jobs have finished\n clean_up.distribute()\n\nIf slurm is not available, ``distribute`` results in a local function\ncall. Analogous for ``srun`` and ``sbatch`` (giving some extra value on\ntop of just forwarding to *simple_slurm*).\n\n.. warning::\n Always use ``JobBundling`` when distributing many small tasks to few nodes. Slurm\n jobs have a certain overhead and you do not want to spam your\n infrastructure with too many jobs. However, function calls\n joined by ``JobBundling`` are considered as a single job by slurm, thus,\n not shared across nodes.\n\n**What are the limitations of slurminade?** Slurminade reconstructs the\nenvironment by basically loading the code on the slurm node (without the\n``__main__``-part) and then calling the slurmified function with\nparameters serialized as JSONSs. This means that the code must be\nwritten in a common ``.py``-file and all (distributed) function\narguments must be JSON-serializable. Also, the function must not use any\nglobal state (e.g., global variables, file or database connections)\ninitialized in the ``__main__``-part. Additionally, the\nPython-environment must be available under the same path on the slurm\nnode as slurminade will use the same paths on the slurm node to\nreconstruct the environment (allowing to use virtual environments).\n\n**Does slurminade work with Python 2?** No, it is a Python 3 project. We\ntested it with Python 3.7 and higher.\n\n**Does slurminade work with Windows?** Probably not, but I never saw a\nslurm cluster running on Windows. The (automatic) slurm-less mode should\nwork on Windows. So your code will run, but all function calls will be\nlocal.\n\n**Are multi-file projects supported?** Yes, as long as the files are\navailable on the slurm node.\n\n**Does slurminade work with virtual environments?** Yes. We recommend to\nuse *slurminade* with `conda <https://docs.conda.io/en/latest/>`__. We\nhave not tested it with other virtual environments.\n\n**Can I run my slurmified code outside a slurm environment?** Yes, if\nyou do not have slurm, the distributed functions are run as normal\nPython function calls. This means that you can share the same code with\npeople that do not have slurm. It was important to us that the\nexperimental evaluations we run on our slurm cluster can also be run in\na common Python environment by reviewers without any changes.\n\n**Can I receive the return value of a slurmified function?** No, the\nreturn value is not transmitted back to the caller. Note that the\ndistribute-calls are non-blocking, i.e., the function returns\nimmediately. Return values could be implemented via a *Promise*-object\nlike for other distributed computing frameworks, but we did not see the\nneed for it yet. We are usually saving the results in a database or\nfiles, e.g., using `AlgBench <https://github.com/d-krupke/AlgBench>`__.\n\n**Can I use command line arguments ``sys.argv`` in my scripts?** Yes,\nbut only in your ``__main__``-part. The arguments are not transmitted to\nthe slurm nodes as they are not part of the function call. You can add\nthese as normal function arguments to your slurmified functions if\nneeded. It is important that your global objects to not rely on these\narguments for initialization, as the ``__main__``-part is not executed\non the slurm node. It is theoretically possible to transmit the\narguments to the slurm node, but we did not see the need for it. Let us\nknow if you need it and we may implement it.\n\nThe code is super simple and open source, don\u2019t be afraid to create a\nfork that fits your own needs.\n\n.. note::\n Talk with you system administrator or supervisor to get the\n proper slurm configuration.\n\nInstallation\n------------\n\nYou can install *slurminade* with ``pip install slurminade``.\n\nUsage\n-----\n\nYou can set task specific slurm arguments within the decorator, e.g.,\n``@slurminade.slurmify(constraint=\"alggen03\")``. These arguments are\ndirectly passed to *simple_slurm*, such that all its arguments are\nsupported.\n\nIn order for *slurminade* to work, the code needs to be in a Python\nfile/project shared by all slurm-nodes. Otherwise, *slurminade* will not\nfind the corresponding function. The slurmified functions also must be\nimportable, i.e., on the top level. Currently, all function names must\nbe unique as *slurminade* will only transmit the function\u2019s name.\n\nDon\u2019t do:\n---------\n\nBad: Non blocking system calls\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: python\n\n import slurminade\n import os\n import subprocess\n\n\n @slurminade.slurmify()\n def run_shell_command():\n # non-blocking system call\n subprocess.Popen(\"complex call\")\n # BAD! The system call will run outside of slurm! The slurm task directly terminates.\n\ninstead use\n\n.. code:: python\n\n import slurminade\n\n if __name__ == \"__main__\":\n slurminade.sbatch(\n \"complex call\"\n ) # forwards your call to simple_slurm that is better used for such things.\n\nBad: Global variables in the ``__main__`` part\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: python\n\n import slurminade\n\n FLAG = True\n\n\n @slurminade.slurmify()\n def bad_global(args):\n if FLAG: # BAD! Will be True because the __main__ Part is not executed on the node.\n pass\n else:\n pass\n\n\n if __name__ == \"__main__\":\n FLAG = False\n bad_global.distribute(\"args\")\n\ninstead do\n\n.. code:: python\n\n import slurminade\n\n\n @slurminade.slurmify()\n def bad_global(\n args, FLAG\n ): # Now the flag is passed correctly as an argument. Note that only json-compatible arguments are possible.\n if FLAG:\n pass\n else:\n pass\n\n\n # Without the `if`, the node would also execute this part (*slurminade* will abort automatically)\n if __name__ == \"__main__\":\n FLAG = False\n bad_global.distribute(\"args\", FLAG)\n\n..\n\n.. warning::\n The same is true for any global state such as file or\n database connections. You can use global variables, but be wary of\n side effects.\n\nError: Complex objects as arguments\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: python\n\n import slurminade\n\n\n @slurminade.slurmify()\n def sec_order_func(func):\n func()\n\n\n def f():\n print(\"hello\")\n\n\n def g():\n print(\"world!\")\n\n\n if __name__ == \"__main__\":\n sec_order_func.distribute(f) # will throw an exception\n sec_order_func.distribute(g)\n\nInstead, create individual slurmified functions for each call or pass a\nsimple identifier that lets the function deduce, what to do, e.g., a\nswitch-case. If you really need to pass complex objects, you could also\npickle the object and only pass the file name.\n\nDefault configuration\n---------------------\n\nYou can set up a default configuration in\n``~/.slurminade_default.json``. This should simply be a dictionary of\narguments for *simple_slurm*. For example\n\n.. code:: json\n\n {\n \"partition\": \"alg\"\n }\n\nThe current version checks the following files and overwrites values in the following order:\n\n1. ``~/.slurminade_default.json``\n2. ``~/$XDG_CONFIG_HOME/slurminade/.slurminade_default.json``\n3. ``./.slurminade_default.json``\n\nDebugging\n---------\n\nYou can use\n\n.. code:: python\n\n import slurminade\n\n slurminade.set_dispatcher(slurminade.TestDispatcher())\n\nto see the serialization or\n\n.. code:: python\n\n import slurminade\n\n slurminade.set_dispatcher(slurminade.SubprocessDispatcher())\n\nto distribute the tasks without slurm using subprocesses.\n\nIf there is a bug, you will directly see it in the output (at least for\nmost bugs).\n\nProject structure\n-----------------\n\nThe project is reasonably easy:\n\n- bundling.py: Contains code for bundling tasks, so we don\u2019t spam slurm\n with too many.\n- conf.py: Contains code for managing the configuration of slurm.\n- dispatcher.py: Contains code for actually dispatching tasks to slurm.\n- execute.py: Contains code to execute the task on the slurm node.\n- function.py: Contains the code for making a function\n slurm-compatible.\n- function_map.py: Saves all the slurmified functions.\n- guard.py: Contains code to prevent you accidentally DDoSing your\n infrastructure.\n- options.py: Contains a simple data structure to save slurm options.\n\nChanges\n-------\n\n- 1.1.0: Slurminade can now be called from iPython, too! `exec` has been renamed `shell` to prevent confusion with the Python call `exec` which will evaluate a string as Python code.\n- 1.0.1: Dispatcher now return jobs references instead of job ids. This allows to do some fancier stuff in the future, when the jobs infos are only available a short time after the job has been submitted.\n- 0.10.1: FIX: Listing functions will no longer execute setup functions.\n- 0.10.0: `Batch` is now named `JobBundling`. There is a method `join` for easier synchronization. `exec` allows to executed commands just like `srun` and `sbatch`, but uniform syntax with other slurmified functions. Functions can now also be called with `distribute_and_wait`. If you call `python3 -m slurminade.check --partition YOUR_PARTITION --constraint YOUR_CONSTRAINT` you can check if your slurm configuration is running correctly.\n- 0.9.0: Lots of improvements.\n- 0.8.1: Bugfix and automatic detection of wrong usage when using ``Batch`` with ``wait_for``.\n- 0.8.0: Added extensive logging and improved typing.\n- 0.7.0: Warning if a Batch is flushed multiple times, as we noticed this to be a common indentation error.\n- 0.6.2: Fixes recursive distribution guard, which seemed to be broken.\n- 0.6.1: Bugfixes in naming\n- 0.6.0: Autmatic naming of tasks.\n- 0.5.5: Fixing bug guard bug in subprocess dispatcher.\n- 0.5.4: Dispatched function calls that are too long for the command\n line now use a temporary file instead.\n- 0.5.3: Fixed a bug that caused the dispatch limit to have no effect.\n- 0.5.2: Added pyproject.toml for PEP compliance\n- 0.5.1: ``Batch`` will now flush on delete, in case you forgot.\n- 0.5.0:\n\n - Functions now have a ``wait_for``-option and return job ids.\n - Braking changes: Batches have a new API.\n\n - ``add`` is no longer needed.\n - ``AutoBatch`` is now called ``Batch``.\n\n - Fundamental code changes under the hood.\n\n- <0.5.0:\n\n - Lots of experiments on finding the right interface.\n\nContributors\n-------------\n\nThis project is developed at the Algorithms Group at TU Braunschweig,\nGermany. The lead developer is `Dominik Krupke <https://krupke.cc>`__.\nFurther contributors are Matthias Konitzny and Patrick Blumenberg.\n\n.. |PyPI version| image:: https://badge.fury.io/py/slurminade.svg\n :target: https://badge.fury.io/py/slurminade\n.. |CI status| image:: https://github.com/d-krupke/slurminade/actions/workflows/pytest.yml/badge.svg\n :target: https://github.com/d-krupke/slurminade\n\n.. |License| image:: https://img.shields.io/github/license/d-krupke/slurminade\n :target: https://github.com/d-krupke/slurminade\n\n\nSimilar Projects\n----------------\n\n* This project is greatly inspired by `Celery <https://github.com/celery/celery>`__, but does not require any additional infrastructure except for slurm.\n* If you want a more powerful library to, e.g., also distribute lambdas or functions with complex arguments, check out `submitit <https://github.com/facebookincubator/submitit/>`__. It is a great project, and we may use it as a backend in the future. However, it does not support the slurm-less mode and can easily hide non-deterministic errors. Slurminade on the other hand is restricted on purpose to write reproducible scripts that can also be run without slurm.\n",
"bugtrack_url": null,
"license": "",
"summary": "A decorator-based slurm runner.",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/d-krupke/slurminade",
"Issues": "https://github.com/d-krupke/slurminade/issues"
},
"split_keywords": [
"slurm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d67182996385f857e477c215a68ad57a4b45b66709d2c3f4c3d0347685b6f5e4",
"md5": "31b43bc11f6b81e8a026e620b559daba",
"sha256": "2531f30c2e2324e8041b82e65b04b75864d649601a6e10ea37640e8a6db74e2f"
},
"downloads": -1,
"filename": "slurminade-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "31b43bc11f6b81e8a026e620b559daba",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 27233,
"upload_time": "2024-02-27T15:42:03",
"upload_time_iso_8601": "2024-02-27T15:42:03.696165Z",
"url": "https://files.pythonhosted.org/packages/d6/71/82996385f857e477c215a68ad57a4b45b66709d2c3f4c3d0347685b6f5e4/slurminade-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ab79934bb10b01a8f1cbca5c11b720da5294723685159f3d4f71c8b950dbb46c",
"md5": "1c8a455d672e74605b3ae439b5fa5ec3",
"sha256": "541b126d1cdfdabf9125c2039fd63f2490b427fe40c454a5fd327b4ad1a937d2"
},
"downloads": -1,
"filename": "slurminade-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "1c8a455d672e74605b3ae439b5fa5ec3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 35464,
"upload_time": "2024-02-27T15:42:07",
"upload_time_iso_8601": "2024-02-27T15:42:07.235607Z",
"url": "https://files.pythonhosted.org/packages/ab/79/934bb10b01a8f1cbca5c11b720da5294723685159f3d4f71c8b950dbb46c/slurminade-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-27 15:42:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "d-krupke",
"github_project": "slurminade",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "setuptools",
"specs": [
[
">=",
"58.0.4"
]
]
},
{
"name": "simple-slurm",
"specs": [
[
">=",
"0.2.3"
]
]
}
],
"lcname": "slurminade"
}
