<div align="center">
<img alt="Labtech logo - FIFO the Labrador in a lab coat and goggles working on a laptop with the labtech 'lt' icon." src="https://ben-denham.github.io/labtech/images/fifo.svg">
<h1>labtech</h1>
<a href="">
<img alt="PyPI" src="https://img.shields.io/pypi/v/labtech">
</a>
<p>
<a href="https://github.com/ben-denham/labtech">GitHub</a> - <a href="https://ben-denham.github.io/labtech">Documentation</a>
</p>
</div>
Labtech makes it easy to define multi-step experiment pipelines and
run them with maximal parallelism and result caching:
* **Defining tasks is simple**; write a class with a single `run()`
method and parameters as dataclass-style attributes.
* **Flexible experiment configuration**; simply create task objects
for all of your parameter permutations.
* **Handles pipelines of tasks**; any task parameter that is itself a
task will be executed first and make its result available to its
dependent task(s).
* **Implicit parallelism**; Labtech resolves task dependencies and
runs tasks in sub-processes with as much parallelism as possible.
* **Implicit caching and loading of task results**; configurable and
extensible options for how and where task results are cached.
* **Integration with [mlflow](https://mlflow.org/)**; Automatically
log task runs to mlflow with all of their parameters.
## Installation
```
pip install labtech
```
## Usage
<!-- N.B. keep this code in-sync with tests/integration/readme/usage.py -->
```python
from time import sleep
import labtech
# Decorate your task class with @labtech.task:
@labtech.task
class Experiment:
# Each Experiment task instance will take `base` and `power` parameters:
base: int
power: int
def run(self) -> int:
# Define the task's run() method to return the result of the experiment:
labtech.logger.info(f'Raising {self.base} to the power of {self.power}')
sleep(1)
return self.base ** self.power
def main():
# Configure Experiment parameter permutations
experiments = [
Experiment(
base=base,
power=power,
)
for base in range(5)
for power in range(5)
]
# Configure a Lab to run the experiments:
lab = labtech.Lab(
# Specify a directory to cache results in (running the experiments a second
# time will just load results from the cache!):
storage='demo_lab',
# Control the degree of parallelism:
max_workers=5,
)
# Run the experiments!
results = lab.run_tasks(experiments)
print([results[experiment] for experiment in experiments])
if __name__ == '__main__':
main()
```
Labtech can also produce graphical progress bars in
[Jupyter](https://jupyter.org/) notebooks:
Tasks parameters can be any of the following types:
* Simple scalar types: `str`, `bool`, `float`, `int`, `None`
* Collections of any of these types: `list`, `tuple`, `dict`, `Enum`
* Task types: A task parameter is a "nested task" that will be
executed before its parent so that it may make use of the nested
result.
Here's an example of defining a single long-running task to produce a
result for a large number of dependent tasks:
<!-- N.B. keep this code in-sync with tests/integration/readme/dependents_and_mermaid.py -->
```python
from time import sleep
import labtech
@labtech.task
class SlowTask:
base: int
def run(self) -> int:
sleep(5)
return self.base ** 2
@labtech.task
class DependentTask:
slow_task: SlowTask
multiplier: int
def run(self) -> int:
return self.multiplier * self.slow_task.result
def main():
some_slow_task = SlowTask(base=42)
dependent_tasks = [
DependentTask(
slow_task=some_slow_task,
multiplier=multiplier,
)
for multiplier in range(10)
]
lab = labtech.Lab(storage='demo_lab')
results = lab.run_tasks(dependent_tasks)
print([results[task] for task in dependent_tasks])
if __name__ == '__main__':
main()
```
Labtech can even generate a [Mermaid diagram](https://mermaid.js.org/syntax/classDiagram.html)
to visualise your tasks:
<!-- N.B. keep this code in-sync with tests/integration/readme/dependents_and_mermaid.py -->
```python
from labtech.diagram import display_task_diagram
some_slow_task = SlowTask(base=42)
dependent_tasks = [
DependentTask(
slow_task=some_slow_task,
multiplier=multiplier,
)
for multiplier in range(10)
]
display_task_diagram(dependent_tasks)
```
```mermaid
classDiagram
direction BT
class DependentTask
DependentTask : SlowTask slow_task
DependentTask : int multiplier
DependentTask : run() int
class SlowTask
SlowTask : int base
SlowTask : run() int
DependentTask <-- SlowTask: slow_task
```
To learn more, dive into the following resources:
* [The labtech tutorial](https://ben-denham.github.io/labtech/tutorial) ([as an interactive notebook](https://mybinder.org/v2/gh/ben-denham/labtech/main?filepath=examples/tutorial.ipynb))
* [Cookbook of common patterns](https://ben-denham.github.io/labtech/cookbook) ([as an interactive notebook](https://mybinder.org/v2/gh/ben-denham/labtech/main?filepath=examples/cookbook.ipynb))
* [API reference for Labs and Tasks](https://ben-denham.github.io/labtech/core)
* [More options for cache formats and storage providers](https://ben-denham.github.io/labtech/caching)
* [Diagramming tools](https://ben-denham.github.io/labtech/diagram)
* [More examples](https://github.com/ben-denham/labtech/tree/main/examples)
## Mypy Plugin
For [mypy](https://mypy-lang.org/) type-checking of classes decorated
with `labtech.task`, simply enable the labtech mypy plugin in your
`mypy.ini` file:
```INI
[mypy]
plugins = labtech.mypy_plugin
```
## Contributing
* Install Poetry dependencies with `make deps`
* Run linting, mypy, and tests with `make check`
* Documentation:
* Run local server: `make docs-serve`
* Build docs: `make docs-build`
* Deploy docs to GitHub Pages: `make docs-github`
* Docstring style follows the [Google style guide](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)
Raw data
{
"_id": null,
"home_page": "https://github.com/ben-denham/labtech",
"name": "labtech",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Ben Denham",
"author_email": "ben@denham.nz",
"download_url": "https://files.pythonhosted.org/packages/26/90/759fa1c085a73a1f0190c9d4467ff8c893b37168c2c6b24f620b05b4981b/labtech-0.6.1.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n<img alt=\"Labtech logo - FIFO the Labrador in a lab coat and goggles working on a laptop with the labtech 'lt' icon.\" src=\"https://ben-denham.github.io/labtech/images/fifo.svg\">\n\n<h1>labtech</h1>\n\n<a href=\"\">\n <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/labtech\">\n</a>\n\n<p>\n <a href=\"https://github.com/ben-denham/labtech\">GitHub</a> - <a href=\"https://ben-denham.github.io/labtech\">Documentation</a>\n</p>\n\n</div>\n\nLabtech makes it easy to define multi-step experiment pipelines and\nrun them with maximal parallelism and result caching:\n\n* **Defining tasks is simple**; write a class with a single `run()`\n method and parameters as dataclass-style attributes.\n* **Flexible experiment configuration**; simply create task objects\n for all of your parameter permutations.\n* **Handles pipelines of tasks**; any task parameter that is itself a\n task will be executed first and make its result available to its\n dependent task(s).\n* **Implicit parallelism**; Labtech resolves task dependencies and\n runs tasks in sub-processes with as much parallelism as possible.\n* **Implicit caching and loading of task results**; configurable and\n extensible options for how and where task results are cached.\n* **Integration with [mlflow](https://mlflow.org/)**; Automatically\n log task runs to mlflow with all of their parameters.\n\n\n## Installation\n\n```\npip install labtech\n```\n\n\n## Usage\n\n<!-- N.B. keep this code in-sync with tests/integration/readme/usage.py -->\n```python\nfrom time import sleep\n\nimport labtech\n\n# Decorate your task class with @labtech.task:\n@labtech.task\nclass Experiment:\n # Each Experiment task instance will take `base` and `power` parameters:\n base: int\n power: int\n\n def run(self) -> int:\n # Define the task's run() method to return the result of the experiment:\n labtech.logger.info(f'Raising {self.base} to the power of {self.power}')\n sleep(1)\n return self.base ** self.power\n\ndef main():\n # Configure Experiment parameter permutations\n experiments = [\n Experiment(\n base=base,\n power=power,\n )\n for base in range(5)\n for power in range(5)\n ]\n\n # Configure a Lab to run the experiments:\n lab = labtech.Lab(\n # Specify a directory to cache results in (running the experiments a second\n # time will just load results from the cache!):\n storage='demo_lab',\n # Control the degree of parallelism:\n max_workers=5,\n )\n\n # Run the experiments!\n results = lab.run_tasks(experiments)\n print([results[experiment] for experiment in experiments])\n\nif __name__ == '__main__':\n main()\n```\n\n\n\nLabtech can also produce graphical progress bars in\n[Jupyter](https://jupyter.org/) notebooks:\n\n\n\nTasks parameters can be any of the following types:\n\n* Simple scalar types: `str`, `bool`, `float`, `int`, `None`\n* Collections of any of these types: `list`, `tuple`, `dict`, `Enum`\n* Task types: A task parameter is a \"nested task\" that will be\n executed before its parent so that it may make use of the nested\n result.\n\nHere's an example of defining a single long-running task to produce a\nresult for a large number of dependent tasks:\n\n<!-- N.B. keep this code in-sync with tests/integration/readme/dependents_and_mermaid.py -->\n```python\nfrom time import sleep\n\nimport labtech\n\n@labtech.task\nclass SlowTask:\n base: int\n\n def run(self) -> int:\n sleep(5)\n return self.base ** 2\n\n@labtech.task\nclass DependentTask:\n slow_task: SlowTask\n multiplier: int\n\n def run(self) -> int:\n return self.multiplier * self.slow_task.result\n\ndef main():\n some_slow_task = SlowTask(base=42)\n dependent_tasks = [\n DependentTask(\n slow_task=some_slow_task,\n multiplier=multiplier,\n )\n for multiplier in range(10)\n ]\n\n lab = labtech.Lab(storage='demo_lab')\n results = lab.run_tasks(dependent_tasks)\n print([results[task] for task in dependent_tasks])\n\nif __name__ == '__main__':\n main()\n```\n\nLabtech can even generate a [Mermaid diagram](https://mermaid.js.org/syntax/classDiagram.html)\nto visualise your tasks:\n\n<!-- N.B. keep this code in-sync with tests/integration/readme/dependents_and_mermaid.py -->\n```python\nfrom labtech.diagram import display_task_diagram\n\nsome_slow_task = SlowTask(base=42)\ndependent_tasks = [\n DependentTask(\n slow_task=some_slow_task,\n multiplier=multiplier,\n )\n for multiplier in range(10)\n]\n\ndisplay_task_diagram(dependent_tasks)\n```\n\n```mermaid\nclassDiagram\n direction BT\n\n class DependentTask\n DependentTask : SlowTask slow_task\n DependentTask : int multiplier\n DependentTask : run() int\n\n class SlowTask\n SlowTask : int base\n SlowTask : run() int\n\n\n DependentTask <-- SlowTask: slow_task\n```\n\nTo learn more, dive into the following resources:\n\n* [The labtech tutorial](https://ben-denham.github.io/labtech/tutorial) ([as an interactive notebook](https://mybinder.org/v2/gh/ben-denham/labtech/main?filepath=examples/tutorial.ipynb))\n* [Cookbook of common patterns](https://ben-denham.github.io/labtech/cookbook) ([as an interactive notebook](https://mybinder.org/v2/gh/ben-denham/labtech/main?filepath=examples/cookbook.ipynb))\n* [API reference for Labs and Tasks](https://ben-denham.github.io/labtech/core)\n* [More options for cache formats and storage providers](https://ben-denham.github.io/labtech/caching)\n* [Diagramming tools](https://ben-denham.github.io/labtech/diagram)\n* [More examples](https://github.com/ben-denham/labtech/tree/main/examples)\n\n\n## Mypy Plugin\n\nFor [mypy](https://mypy-lang.org/) type-checking of classes decorated\nwith `labtech.task`, simply enable the labtech mypy plugin in your\n`mypy.ini` file:\n\n```INI\n[mypy]\nplugins = labtech.mypy_plugin\n```\n\n## Contributing\n\n* Install Poetry dependencies with `make deps`\n* Run linting, mypy, and tests with `make check`\n* Documentation:\n * Run local server: `make docs-serve`\n * Build docs: `make docs-build`\n * Deploy docs to GitHub Pages: `make docs-github`\n * Docstring style follows the [Google style guide](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)\n",
"bugtrack_url": null,
"license": "GPL-3.0-only",
"summary": "Easily run experiment permutations with multi-processing and caching.",
"version": "0.6.1",
"project_urls": {
"Homepage": "https://github.com/ben-denham/labtech",
"Repository": "https://github.com/ben-denham/labtech"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b2b9497b340b2fcd9f53064a3ed860ec00d361963108cf9f70acf90d9eb7c140",
"md5": "e347d23e811fca5c43ed325ca2cf1d28",
"sha256": "13ab2367e3d815a4ad03eea65b7a58b4773e4af2805f60549aad0ebb319908cf"
},
"downloads": -1,
"filename": "labtech-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e347d23e811fca5c43ed325ca2cf1d28",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 41979,
"upload_time": "2024-08-03T05:47:13",
"upload_time_iso_8601": "2024-08-03T05:47:13.960702Z",
"url": "https://files.pythonhosted.org/packages/b2/b9/497b340b2fcd9f53064a3ed860ec00d361963108cf9f70acf90d9eb7c140/labtech-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2690759fa1c085a73a1f0190c9d4467ff8c893b37168c2c6b24f620b05b4981b",
"md5": "ea27c6ebde836e4eb187c035969fc1f2",
"sha256": "a0ad26f5f7241bf540fabe25fd79e21b5a3a6fb815cc9502db00d42b0c0ad926"
},
"downloads": -1,
"filename": "labtech-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "ea27c6ebde836e4eb187c035969fc1f2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 39287,
"upload_time": "2024-08-03T05:47:16",
"upload_time_iso_8601": "2024-08-03T05:47:16.204040Z",
"url": "https://files.pythonhosted.org/packages/26/90/759fa1c085a73a1f0190c9d4467ff8c893b37168c2c6b24f620b05b4981b/labtech-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-03 05:47:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ben-denham",
"github_project": "labtech",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "labtech"
}
JSON
