# alkymi ⚗️
[](https://github.com/MathiasStokholm/alkymi/actions?query=workflow%3Abuild)
[](https://alkymi.readthedocs.io/en/latest/?badge=latest)
[](https://codecov.io/gh/MathiasStokholm/alkymi)
[](https://pypi.org/project/alkymi)
[](https://pypi.org/project/alkymi)
Alkymi is a pure Python (3.7+) library for describing and executing tasks and pipelines with built-in caching and
conditional evaluation based on checksums.
Alkymi is easy to install, simple to use, and has very few dependencies outside of Python's standard library. The code
is cross-platform, and allows you to write your pipelines once and deploy to multiple operating systems (tested on
Linux, Windows and Mac).
Documentation, including a quickstart guide, is provided [here](https://alkymi.readthedocs.io/en/latest/).
## Features
* Easily define complex data pipelines as decorated Python functions
* This allows you to run linting, type checking, etc. on your data pipelines
* Return values are automatically cached to disk, regardless of type
* Efficiently checks if pipeline is up-to-date
* Checks if external files have changed, bound functions have changed or if pipeline dependencies have changed
* No domain specific language (DSL) or CLI tool, just regular Python
* Supports caching and conditional evaluation in Jupyter Notebooks
* Cross-platform - works on Linux, Windows and Mac
* Expose recipes as a command-line interface (CLI) using alkymi's
[Lab](https://alkymi.readthedocs.io/en/latest/examples/command_line.html) type
## Sample Usage
For examples of how to use alkymi, see the
[quickstart guide](https://alkymi.readthedocs.io/en/latest/getting_started/quick_start.html).
Example code:
```python
import numpy as np
import alkymi as alk
@alk.recipe()
def long_running_task() -> np.ndarray:
# Perform expensive computation here ...
hard_to_compute_result = np.array([42])
# Return value will be automatically cached to disk
return hard_to_compute_result
result = long_running_task.brew() # == np.ndarray([42])
```
Or one of the examples, e.g. [MNIST](https://alkymi.readthedocs.io/en/latest/examples/mnist.html).
## Installation
Install via pip:
```shell script
pip install --user alkymi
```
Or see the [Installation page](https://alkymi.readthedocs.io/en/latest/getting_started/installation.html).
### Testing
After installing, you can run the test suite (use the `lint`, `coverage` and `type_check` recipes to perform those
actions):
```shell script
python3 labfile.py brew test
```
## License
alkymi is licensed under The MIT License as found in the LICENSE.md file
## Upcoming Features
The following features are being considered for future implementation:
* Type annotations propagated from bound functions to recipes
* Support for call/type checking all recipes (e.g. by adding a `check` command to `Lab`)
* Cache maintenance functionality
## Known Issues
* alkymi currently doesn't check custom objects for altered external files when computing cleanliness (e.g. `MyClass`
has a `self._some_path` that points to a file somewhere outside alkymi's internal cache)
* `alk.foreach()` currently only supports enumerable inputs of type `List` or `Dict`
* Recipes marked `transient` will always be dirty, and thus always require reevaluation. This functionality should be
replaced by a proper means of creating recipes that don't cache outputs, but only run when needed to provide inputs for
downstream recipes
Raw data
{
"_id": null,
"home_page": "https://github.com/MathiasStokholm/alkymi",
"name": "alkymi",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "automation,pipeline,validation,preprocessing,make,build,task",
"author": "Mathias B\u00f8gh Stokholm",
"author_email": "mathias.stokholm@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f0/bb/1dcdebc088fecbcc29b51fcb53486a37d1b4f5448be8b919b923037edf6e/alkymi-0.2.1.tar.gz",
"platform": null,
"description": "# alkymi \u2697\ufe0f\n\n[](https://github.com/MathiasStokholm/alkymi/actions?query=workflow%3Abuild)\n[](https://alkymi.readthedocs.io/en/latest/?badge=latest)\n[](https://codecov.io/gh/MathiasStokholm/alkymi)\n[](https://pypi.org/project/alkymi)\n[](https://pypi.org/project/alkymi)\n\nAlkymi is a pure Python (3.7+) library for describing and executing tasks and pipelines with built-in caching and\nconditional evaluation based on checksums.\n\nAlkymi is easy to install, simple to use, and has very few dependencies outside of Python's standard library. The code\nis cross-platform, and allows you to write your pipelines once and deploy to multiple operating systems (tested on\nLinux, Windows and Mac).\n\nDocumentation, including a quickstart guide, is provided [here](https://alkymi.readthedocs.io/en/latest/).\n\n## Features\n* Easily define complex data pipelines as decorated Python functions\n * This allows you to run linting, type checking, etc. on your data pipelines\n* Return values are automatically cached to disk, regardless of type\n* Efficiently checks if pipeline is up-to-date\n * Checks if external files have changed, bound functions have changed or if pipeline dependencies have changed\n* No domain specific language (DSL) or CLI tool, just regular Python\n * Supports caching and conditional evaluation in Jupyter Notebooks\n* Cross-platform - works on Linux, Windows and Mac\n* Expose recipes as a command-line interface (CLI) using alkymi's\n[Lab](https://alkymi.readthedocs.io/en/latest/examples/command_line.html) type\n\n## Sample Usage\nFor examples of how to use alkymi, see the\n[quickstart guide](https://alkymi.readthedocs.io/en/latest/getting_started/quick_start.html).\n\nExample code:\n```python\nimport numpy as np\nimport alkymi as alk\n\n@alk.recipe()\ndef long_running_task() -> np.ndarray:\n # Perform expensive computation here ...\n hard_to_compute_result = np.array([42])\n # Return value will be automatically cached to disk\n return hard_to_compute_result\n\nresult = long_running_task.brew() # == np.ndarray([42])\n```\n\nOr one of the examples, e.g. [MNIST](https://alkymi.readthedocs.io/en/latest/examples/mnist.html).\n\n## Installation\nInstall via pip:\n```shell script\npip install --user alkymi\n```\n\nOr see the [Installation page](https://alkymi.readthedocs.io/en/latest/getting_started/installation.html).\n\n### Testing\nAfter installing, you can run the test suite (use the `lint`, `coverage` and `type_check` recipes to perform those\nactions):\n```shell script\npython3 labfile.py brew test\n```\n\n## License\nalkymi is licensed under The MIT License as found in the LICENSE.md file\n\n## Upcoming Features\nThe following features are being considered for future implementation:\n* Type annotations propagated from bound functions to recipes\n* Support for call/type checking all recipes (e.g. by adding a `check` command to `Lab`)\n* Cache maintenance functionality\n\n## Known Issues\n* alkymi currently doesn't check custom objects for altered external files when computing cleanliness (e.g. `MyClass`\nhas a `self._some_path` that points to a file somewhere outside alkymi's internal cache)\n* `alk.foreach()` currently only supports enumerable inputs of type `List` or `Dict`\n* Recipes marked `transient` will always be dirty, and thus always require reevaluation. This functionality should be\nreplaced by a proper means of creating recipes that don't cache outputs, but only run when needed to provide inputs for\ndownstream recipes\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "alkymi - Pythonic task automation",
"version": "0.2.1",
"split_keywords": [
"automation",
"pipeline",
"validation",
"preprocessing",
"make",
"build",
"task"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "29fd261db7cee8cdb5a654b8eb26e8cd8d6ccd8f2925d66d0f7c26d891d1209b",
"md5": "65a7819905255ee17bea0763beddd398",
"sha256": "ab2a5c60a81bc567037e58c5984f552811c77cdd44eafe925d1c8c39005e0476"
},
"downloads": -1,
"filename": "alkymi-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "65a7819905255ee17bea0763beddd398",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 34025,
"upload_time": "2023-04-27T10:26:36",
"upload_time_iso_8601": "2023-04-27T10:26:36.437690Z",
"url": "https://files.pythonhosted.org/packages/29/fd/261db7cee8cdb5a654b8eb26e8cd8d6ccd8f2925d66d0f7c26d891d1209b/alkymi-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f0bb1dcdebc088fecbcc29b51fcb53486a37d1b4f5448be8b919b923037edf6e",
"md5": "01e683300ac7c2da3ac554c9a3a31055",
"sha256": "43ddc25ec08b343b45f569f6203a5356acf3aaf9eb9a6de8cc25ffedd39b913b"
},
"downloads": -1,
"filename": "alkymi-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "01e683300ac7c2da3ac554c9a3a31055",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 30372,
"upload_time": "2023-04-27T10:26:39",
"upload_time_iso_8601": "2023-04-27T10:26:39.024145Z",
"url": "https://files.pythonhosted.org/packages/f0/bb/1dcdebc088fecbcc29b51fcb53486a37d1b4f5448be8b919b923037edf6e/alkymi-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-27 10:26:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "MathiasStokholm",
"github_project": "alkymi",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "networkx",
"specs": [
[
">=",
"2.0"
]
]
},
{
"name": "rich",
"specs": [
[
">=",
"10.7"
]
]
}
],
"lcname": "alkymi"
}
JSON
