# goeieDAG
_/ɣu.jə.ˈdɑx/: hello, good day (Dutch greeting used during daytime)_
goeieDAG provides a unified Python API to Ninja and Make (_TODO_) build systems, aiming to
make it extremely easy to benefit from parallel processing in any graph-like workflow.
## Installation
pip install goeieDAG==0.0.2
## Usage
```python
from pathlib import Path
import goeiedag
from goeiedag import ALL_INPUTS, INPUT, OUTPUT
workdir = Path("output")
graph = goeiedag.Graph()
# Extract OS name from /etc/os-release
graph.add(["grep", "^NAME=", INPUT, ">", OUTPUT],
inputs=["/etc/os-release"],
outputs=["os-name.txt"])
# Get username
graph.add(["whoami", ">", OUTPUT],
inputs=[],
outputs=["username.txt"])
# Glue together to produce output
graph.add(["cat", ALL_INPUTS, ">", OUTPUT.result],
inputs=["os-name.txt", "username.txt"],
outputs=dict(result="result.txt")) # can also use a dictionary and refer to inputs/outputs by name
goeiedag.build_all(graph, workdir)
# Print output
print((workdir / "result.txt").read_text())
```
## Q&A
### Why use the _files and commands_ model rather than _Python objects and functions_?
- It is a tested and proven paradigm (`make` traces back to _1976_!)
- It provides an obvious way of evaluating which products need rebuilding (subject to an
accurate dependency graph)
- It naturally isolates and parallelizes individual build tasks
- It is agnostic as to how data objects are serialized (convenient for the library author...)
- Graph edges are implicitly defined by input/output file names
- A high-quality executor (Ninja) is available and installable via a Python package
### How is this different from using the Ninja package directly?
- Simpler mental model & usage: no need to separately define build rules or think about implicit/explicit inputs and
outputs
- API accepts Paths; no need to cast everything to `str`!
- Higher-level API in general (for example, the output directory is created automatically)
## Similar projects
- [Ninja](https://pypi.org/project/ninja/) (Python package) -- provides a lower-level API,
used by goeieDAG as back-end
- [TaskGraph](https://github.com/natcap/taskgraph/) -- similar project, but centered around
Python functions and in-process parallelism
- [Snakemake](https://snakemake.github.io/) -- similar goals, but a stand-alone tool rather
than a library
- [Dask](https://dask.org/) -- different execution model; caching of intermediate results
is left up to the user
- [doit](https://pydoit.org/)
Raw data
{
"_id": null,
"home_page": "https://github.com/mcejp/goeieDAG",
"name": "goeieDAG",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "dag, make, makefile, ninja",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/c8/70/1e6c7cb8bbe4d65178e70cdf23eda6a034d48c93a36212c929796e5cd54c/goeiedag-0.0.2.tar.gz",
"platform": null,
"description": "# goeieDAG\n\n_/\u0263u.j\u0259.\u02c8d\u0251x/: hello, good day (Dutch greeting used during daytime)_\n\ngoeieDAG provides a unified Python API to Ninja and Make (_TODO_) build systems, aiming to\nmake it extremely easy to benefit from parallel processing in any graph-like workflow.\n\n\n## Installation\n\n pip install goeieDAG==0.0.2\n\n## Usage\n\n```python\nfrom pathlib import Path\n\nimport goeiedag\nfrom goeiedag import ALL_INPUTS, INPUT, OUTPUT\n\nworkdir = Path(\"output\")\n\ngraph = goeiedag.Graph()\n\n# Extract OS name from /etc/os-release\ngraph.add([\"grep\", \"^NAME=\", INPUT, \">\", OUTPUT],\n inputs=[\"/etc/os-release\"],\n outputs=[\"os-name.txt\"])\n# Get username\ngraph.add([\"whoami\", \">\", OUTPUT],\n inputs=[],\n outputs=[\"username.txt\"])\n# Glue together to produce output\ngraph.add([\"cat\", ALL_INPUTS, \">\", OUTPUT.result],\n inputs=[\"os-name.txt\", \"username.txt\"],\n outputs=dict(result=\"result.txt\")) # can also use a dictionary and refer to inputs/outputs by name\n\ngoeiedag.build_all(graph, workdir)\n\n# Print output\nprint((workdir / \"result.txt\").read_text())\n```\n\n\n## Q&A\n\n### Why use the _files and commands_ model rather than _Python objects and functions_?\n\n- It is a tested and proven paradigm (`make` traces back to _1976_!)\n- It provides an obvious way of evaluating which products need rebuilding (subject to an\n accurate dependency graph)\n- It naturally isolates and parallelizes individual build tasks\n- It is agnostic as to how data objects are serialized (convenient for the library author...)\n- Graph edges are implicitly defined by input/output file names\n- A high-quality executor (Ninja) is available and installable via a Python package\n\n### How is this different from using the Ninja package directly?\n\n- Simpler mental model & usage: no need to separately define build rules or think about implicit/explicit inputs and\n outputs\n- API accepts Paths; no need to cast everything to `str`!\n- Higher-level API in general (for example, the output directory is created automatically)\n\n\n## Similar projects\n\n- [Ninja](https://pypi.org/project/ninja/) (Python package) -- provides a lower-level API,\n used by goeieDAG as back-end\n- [TaskGraph](https://github.com/natcap/taskgraph/) -- similar project, but centered around\n Python functions and in-process parallelism\n- [Snakemake](https://snakemake.github.io/) -- similar goals, but a stand-alone tool rather\n than a library\n- [Dask](https://dask.org/) -- different execution model; caching of intermediate results\n is left up to the user\n- [doit](https://pydoit.org/)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Task crunching with Ninja or Make",
"version": "0.0.2",
"project_urls": {
"Homepage": "https://github.com/mcejp/goeieDAG"
},
"split_keywords": [
"dag",
" make",
" makefile",
" ninja"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "96ddf3d6c1a772800ca6e53b3649337751a37d07d37d9c2c0636842f4a40cd9e",
"md5": "a45ab4a2c7160fd41eae51a79d2b4cf1",
"sha256": "33d12315959fc0fe78252d890eb05181111288a3541daa6166f78ca980a52baf"
},
"downloads": -1,
"filename": "goeieDAG-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a45ab4a2c7160fd41eae51a79d2b4cf1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 7901,
"upload_time": "2024-07-07T12:36:26",
"upload_time_iso_8601": "2024-07-07T12:36:26.881208Z",
"url": "https://files.pythonhosted.org/packages/96/dd/f3d6c1a772800ca6e53b3649337751a37d07d37d9c2c0636842f4a40cd9e/goeieDAG-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c8701e6c7cb8bbe4d65178e70cdf23eda6a034d48c93a36212c929796e5cd54c",
"md5": "fc25be42a755a0bda56166b910c0b970",
"sha256": "103a025a64d2ccc2946865be45304b104d3fb4efcd016693bdc1747b9b111bd9"
},
"downloads": -1,
"filename": "goeiedag-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "fc25be42a755a0bda56166b910c0b970",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 7129,
"upload_time": "2024-07-07T12:36:27",
"upload_time_iso_8601": "2024-07-07T12:36:27.957180Z",
"url": "https://files.pythonhosted.org/packages/c8/70/1e6c7cb8bbe4d65178e70cdf23eda6a034d48c93a36212c929796e5cd54c/goeiedag-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-07 12:36:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mcejp",
"github_project": "goeieDAG",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "goeiedag"
}
JSON
