# Substantial
Brokerless durable execution for Python.
> Substantial is part of the
> [Metatype ecosystem](https://github.com/metatypedev/metatype). Consider
> checking out how this component integrates with the whole ecosystem and browse
> the
> [documentation](https://metatype.dev?utm_source=github&utm_medium=readme&utm_campaign=substantial)
> to see more examples.
## What is durable execution?
Durable execution is a programming model where the state of a function is preserved across failures, restarts, or other (un)voluntary disruptions. This ensures that applications can continue execution from their last stable state without losing context or causing additional side effects. It is particularly well-suited for long-running workflows as it enables the management of complex sequences of steps, handling breaks, retries and recovery gracefully.
Substantial is designed around a replay mechanism that reconstructs the function state by re-executing historical events from stored logs. All the logic is embedded in a protocol and there is no centralized broker, allowing it to work with any backend (local files, cloud storage and databases). It aims to be an alternative for use cases that do not require the scale and complexity of [Temporal](https://github.com/temporalio/temporal) or [Durable Task](https://github.com/Azure/durabletask).
## Getting started
```
# pypi
pip install substantial
poetry add substantial
# remote master
pip install --upgrade git+https://github.com/zifeo/substantial.git
poetry add git+https://github.com/zifeo/substantial.git
# local repo/dev
poetry install
pre-commit install
protoc -I . --python_betterproto_out=. protocol/*
```
### Workflow
```py
def roulette():
if random.random() < 1/6:
raise Exception("shot!")
return 100
def multiply(a, b):
return a * b
@dataclass
class State:
cancelled: bool = False
def update(self):
self.cancelled = True
@workflow()
async def example(c: Context):
res = await c.save(roulette)
await c.sleep(timedelta(seconds=10))
res = await c.save(lambda: multiply(res, 0.5))
n = await c.receive("by")
res = await c.save(lambda: multiply(res, n))
s = State(is_cancelled=False)
c.handle("cancel", lambda: s.update())
await c.ensure(lambda: s.cancelled):
return res
```
### Worker
Brokerless means that there is no active broker and all the scheduling is solely organized around files/key values. Substantial currently supports Redis, local files and s3-compatible object storages.
```py
backend = FSBackend("./logs")
substantial = Conductor(backend)
substantial.register(example)
# run the agent in the background
agent = substantial.run()
# start the workflow
w = await substantial.start(example)
await asyncio.sleep(3)
print(await w.send("by", 2))
await asyncio.sleep(5)
print(await w.send("cancel"))
output = await w.result()
print("Final output", output) # 100
# stop the agent
agent.cancel()
await agent
```
## API
_Note_: not all features are implemented/completed yet.
### Primitives
`save(f: Callable, compensate_with: Optional[Callable]): Any` - memoize the result of a function to avoid re-execution on replay. Functions shall be idempotent as they may be called more than once in case of failure just before the value is persisted. The function can be compensated by providing its inverse effect and trigger later in the workflow with `revert`.
`handle(event_name: str, cb: Callable): None` - register a callback to be executed when a specific event is received. The callbacks are executed in the order they were received and whenever a primitive being called.
`ensure(f: Callable): True` - wait for the function to evaluate to true and schedule a new run when false.
### Higher-level
`sleep` - schedule a replay after a certain amount of time.
`receive` - wait for the value of an event to be received.
`log` - TODO
`datetime.now` - TODO
`random` - TODO
`uuid4` - TODO
### Advanced
`revert()` - execute the compensations and stop the workflow.
`continue_using(workflow, *args, **kwargs)` - stop the current workflow and pass the context to a new one.
`compact()` - a key can be defined on all the primitives to avoid the infinitely growing log issue. This function will keep only the last occurrence of each of the keys.
Raw data
{
"_id": null,
"home_page": null,
"name": "substantial",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "durable, execution, redis, s3, filesystem",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/13/b0/6c6e1987bc281e8334d01514335e30619d44ae5f0c8e71d11e36a83f427f/substantial-0.0.1.tar.gz",
"platform": null,
"description": "# Substantial\n\nBrokerless durable execution for Python.\n\n> Substantial is part of the\n> [Metatype ecosystem](https://github.com/metatypedev/metatype). Consider\n> checking out how this component integrates with the whole ecosystem and browse\n> the\n> [documentation](https://metatype.dev?utm_source=github&utm_medium=readme&utm_campaign=substantial)\n> to see more examples.\n\n## What is durable execution?\n\nDurable execution is a programming model where the state of a function is preserved across failures, restarts, or other (un)voluntary disruptions. This ensures that applications can continue execution from their last stable state without losing context or causing additional side effects. It is particularly well-suited for long-running workflows as it enables the management of complex sequences of steps, handling breaks, retries and recovery gracefully.\n\nSubstantial is designed around a replay mechanism that reconstructs the function state by re-executing historical events from stored logs. All the logic is embedded in a protocol and there is no centralized broker, allowing it to work with any backend (local files, cloud storage and databases). It aims to be an alternative for use cases that do not require the scale and complexity of [Temporal](https://github.com/temporalio/temporal) or [Durable Task](https://github.com/Azure/durabletask).\n\n## Getting started\n\n```\n# pypi\npip install substantial\npoetry add substantial\n\n# remote master\npip install --upgrade git+https://github.com/zifeo/substantial.git\npoetry add git+https://github.com/zifeo/substantial.git\n\n# local repo/dev\npoetry install\npre-commit install\nprotoc -I . --python_betterproto_out=. protocol/*\n```\n\n### Workflow\n\n```py\ndef roulette():\n if random.random() < 1/6:\n raise Exception(\"shot!\")\n return 100\n\ndef multiply(a, b):\n return a * b\n\n@dataclass\nclass State:\n cancelled: bool = False\n\n def update(self):\n self.cancelled = True\n\n\n@workflow()\nasync def example(c: Context):\n res = await c.save(roulette)\n\n await c.sleep(timedelta(seconds=10))\n res = await c.save(lambda: multiply(res, 0.5))\n\n n = await c.receive(\"by\")\n res = await c.save(lambda: multiply(res, n))\n\n s = State(is_cancelled=False)\n c.handle(\"cancel\", lambda: s.update())\n\n await c.ensure(lambda: s.cancelled):\n return res\n```\n\n### Worker\n\nBrokerless means that there is no active broker and all the scheduling is solely organized around files/key values. Substantial currently supports Redis, local files and s3-compatible object storages.\n\n```py\nbackend = FSBackend(\"./logs\")\nsubstantial = Conductor(backend)\nsubstantial.register(example)\n\n# run the agent in the background\nagent = substantial.run()\n\n# start the workflow\nw = await substantial.start(example)\n\nawait asyncio.sleep(3)\nprint(await w.send(\"by\", 2))\n\nawait asyncio.sleep(5)\nprint(await w.send(\"cancel\"))\n\noutput = await w.result()\nprint(\"Final output\", output) # 100\n\n# stop the agent\nagent.cancel()\nawait agent\n```\n\n## API\n\n_Note_: not all features are implemented/completed yet.\n\n### Primitives\n\n`save(f: Callable, compensate_with: Optional[Callable]): Any` - memoize the result of a function to avoid re-execution on replay. Functions shall be idempotent as they may be called more than once in case of failure just before the value is persisted. The function can be compensated by providing its inverse effect and trigger later in the workflow with `revert`.\n\n`handle(event_name: str, cb: Callable): None` - register a callback to be executed when a specific event is received. The callbacks are executed in the order they were received and whenever a primitive being called.\n\n`ensure(f: Callable): True` - wait for the function to evaluate to true and schedule a new run when false.\n\n### Higher-level\n\n`sleep` - schedule a replay after a certain amount of time.\n\n`receive` - wait for the value of an event to be received.\n\n`log` - TODO\n\n`datetime.now` - TODO\n\n`random` - TODO\n\n`uuid4` - TODO\n\n### Advanced\n\n`revert()` - execute the compensations and stop the workflow.\n\n`continue_using(workflow, *args, **kwargs)` - stop the current workflow and pass the context to a new one.\n\n`compact()` - a key can be defined on all the primitives to avoid the infinitely growing log issue. This function will keep only the last occurrence of each of the keys.\n",
"bugtrack_url": null,
"license": "MPL-2.0",
"summary": "Brokerless durable execution for Python.",
"version": "0.0.1",
"project_urls": null,
"split_keywords": [
"durable",
" execution",
" redis",
" s3",
" filesystem"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fe750b099086f1c8e3dee5cbc51495c81d69d97a623eaef4dbad6e7802dbb672",
"md5": "188cef82ebc627f3be7fe36364a99b0a",
"sha256": "908a621d6b9520bf3126ae94ff1a1cf967e764619055fc60102135b94e706220"
},
"downloads": -1,
"filename": "substantial-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "188cef82ebc627f3be7fe36364a99b0a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 22662,
"upload_time": "2024-10-25T20:36:11",
"upload_time_iso_8601": "2024-10-25T20:36:11.489462Z",
"url": "https://files.pythonhosted.org/packages/fe/75/0b099086f1c8e3dee5cbc51495c81d69d97a623eaef4dbad6e7802dbb672/substantial-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "13b06c6e1987bc281e8334d01514335e30619d44ae5f0c8e71d11e36a83f427f",
"md5": "bdd53ac0e604064afba5cf3690218075",
"sha256": "8274fb9bf08969e124214b43bc197a042d4dde346cbbe099cb467d3ae75536cc"
},
"downloads": -1,
"filename": "substantial-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "bdd53ac0e604064afba5cf3690218075",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 18609,
"upload_time": "2024-10-25T20:36:13",
"upload_time_iso_8601": "2024-10-25T20:36:13.046310Z",
"url": "https://files.pythonhosted.org/packages/13/b0/6c6e1987bc281e8334d01514335e30619d44ae5f0c8e71d11e36a83f427f/substantial-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-25 20:36:13",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "substantial"
}
JSON