# feedforward
This library makes it easy to run a linear DAG while extracting magical
parallelism if some steps don't make changes to some values.
## Comparison to dataflow
Sure, this is a dataflow-esque library that allows you to do transforms on
items in one direction towards a goal. Where it differs from other dataflow
models is that there is only `map` and the items can never never change type
(for a given key).
Additionally all steps and inputs need to be known up front, but that isn't a
restriction of the core algorithm, just in the name of readability.
In exchange for those restrictions, you get a lot of API simplicity, as well as
the ability to run future steps eagerly given sufficient slots and automatic
bundling into "batches" of items like xargs does to amortize child set-up times.
## Restrictions
* The steps need to be decided up front (although it's cheap to have steps that
maybe don't do anything). This includes the order that they will apply in.
* Steps ought to be deterministic and idempotent within a run (if they aren't,
you should mark individual steps `eager=False` or enable `deliberate=True` on
the `Run`, which only uses intra-step parallelism).
* Steps ought to have static relationships between the inputs and output keys
such as `%.py` input changes potentially affecting `%.java` outputs, using
the wildcard `%` you might know from Make. If you don't (say, files can
include other arbitrary files), then you might need to model this as *any*
input change invalidating *all* output keys which will tend to be inefficient.
* Steps ought to not change the type of a key's value (although they can create
new keys, or delete existing keys, so you can work around this by including
the type in the key and still get correctness). If you wanted to support
`str` <=> `int` transformations on the same key, this will only work if *all*
subsequent steps work with either.
* Your input values, as well as all intermediate output values, need to fit in
memory. Nothing keeps you from using filenames, urls, or CAS keys as the
value though.
# Version Compat
Usage of this library should work back to 3.8, but development (and mypy
compatibility) only on 3.10-3.12. Linting requires 3.12 for full fidelity.
# Versioning
This library follows [meanver](https://meanver.org/) which basically means
[semver](https://semver.org/) along with a promise to rename when the major
version changes.
# License
feedforward is copyright [Tim Hatch](https://timhatch.com/), and licensed under
the MIT license. See the `LICENSE` file for details.
Raw data
{
"_id": null,
"home_page": "https://github.com/advice-animal/feedforward/",
"name": "feedforward",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Tim Hatch",
"author_email": "tim@timhatch.com",
"download_url": "https://files.pythonhosted.org/packages/e6/db/29d82b9ae0d0cc517ebbc0efe81bf34998000d7dcb37550ca673cf0deaee/feedforward-0.8.0b2.tar.gz",
"platform": null,
"description": "# feedforward\n\nThis library makes it easy to run a linear DAG while extracting magical\nparallelism if some steps don't make changes to some values.\n\n## Comparison to dataflow\n\nSure, this is a dataflow-esque library that allows you to do transforms on\nitems in one direction towards a goal. Where it differs from other dataflow\nmodels is that there is only `map` and the items can never never change type\n(for a given key).\n\nAdditionally all steps and inputs need to be known up front, but that isn't a\nrestriction of the core algorithm, just in the name of readability.\n\nIn exchange for those restrictions, you get a lot of API simplicity, as well as\nthe ability to run future steps eagerly given sufficient slots and automatic\nbundling into \"batches\" of items like xargs does to amortize child set-up times.\n\n## Restrictions\n\n* The steps need to be decided up front (although it's cheap to have steps that\n maybe don't do anything). This includes the order that they will apply in.\n* Steps ought to be deterministic and idempotent within a run (if they aren't,\n you should mark individual steps `eager=False` or enable `deliberate=True` on\n the `Run`, which only uses intra-step parallelism).\n* Steps ought to have static relationships between the inputs and output keys\n such as `%.py` input changes potentially affecting `%.java` outputs, using\n the wildcard `%` you might know from Make. If you don't (say, files can\n include other arbitrary files), then you might need to model this as *any*\n input change invalidating *all* output keys which will tend to be inefficient.\n* Steps ought to not change the type of a key's value (although they can create\n new keys, or delete existing keys, so you can work around this by including\n the type in the key and still get correctness). If you wanted to support\n `str` <=> `int` transformations on the same key, this will only work if *all*\n subsequent steps work with either.\n* Your input values, as well as all intermediate output values, need to fit in\n memory. Nothing keeps you from using filenames, urls, or CAS keys as the\n value though.\n\n# Version Compat\n\nUsage of this library should work back to 3.8, but development (and mypy\ncompatibility) only on 3.10-3.12. Linting requires 3.12 for full fidelity.\n\n# Versioning\n\nThis library follows [meanver](https://meanver.org/) which basically means\n[semver](https://semver.org/) along with a promise to rename when the major\nversion changes.\n\n# License\n\nfeedforward is copyright [Tim Hatch](https://timhatch.com/), and licensed under\nthe MIT license. See the `LICENSE` file for details.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Simple local workflow runs",
"version": "0.8.0b2",
"project_urls": {
"Homepage": "https://github.com/advice-animal/feedforward/"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "c0d2e3aa56badce24796df1ebe0ab4d3097165d84818f213e91e4bf917553179",
"md5": "a21667c67b8f0ff68489f01cad04d651",
"sha256": "3d25da1ce7094968f659d6d4ecc1a3bb458602b2e6d8292e6a91908c6a71d0c9"
},
"downloads": -1,
"filename": "feedforward-0.8.0b2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a21667c67b8f0ff68489f01cad04d651",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11496,
"upload_time": "2025-08-02T01:44:32",
"upload_time_iso_8601": "2025-08-02T01:44:32.135664Z",
"url": "https://files.pythonhosted.org/packages/c0/d2/e3aa56badce24796df1ebe0ab4d3097165d84818f213e91e4bf917553179/feedforward-0.8.0b2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e6db29d82b9ae0d0cc517ebbc0efe81bf34998000d7dcb37550ca673cf0deaee",
"md5": "02c18e38c91255f659ffb29b7e030450",
"sha256": "453469d17327e59b7affc7bbef9f192ed17c5142cce28fb97e12f681930edf6b"
},
"downloads": -1,
"filename": "feedforward-0.8.0b2.tar.gz",
"has_sig": false,
"md5_digest": "02c18e38c91255f659ffb29b7e030450",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 133140,
"upload_time": "2025-08-02T01:44:33",
"upload_time_iso_8601": "2025-08-02T01:44:33.440556Z",
"url": "https://files.pythonhosted.org/packages/e6/db/29d82b9ae0d0cc517ebbc0efe81bf34998000d7dcb37550ca673cf0deaee/feedforward-0.8.0b2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-02 01:44:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "advice-animal",
"github_project": "feedforward",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "feedforward"
}
JSON
