# Django Side Effects
Django app for managing external side effects.
## Compatibility
**This project now supports Python 3.8+ and Django 3.2+ only on master.**
Legacy versions are tagged.
## Background
This project was created to try and bring some order to the use of
external side-effects within the YunoJuno platform. External
side-effects are (as defined by us) those actions that affect external
systems, and that are not part of the core application integrity. They
fall into two main categories within our application - *notifications*
and *updates*, and are best illustrated by example:
**Notifications**
* Slack messages
* SMS (via Twilio)
* Push notifications
* Email
**Updates**
* Base CRM (sales)
* Mailchimp CRM (marketing)
* Elasticsearch (full-text index)
There are some shared aspects of all of these side-effects:
1. They can all be processed asynchronously (queued)
2. They can all be replayed (and are idempotent)
3. They can be executed in any order
4. They are not time critical
5. They do not affect the state of the Django application
As we have continued to build out YunoJuno our use of these side-effects
has become ever more complex, and has in some areas left us with functions
that are 80% side-effects:
```python
def foo():
# do the thing the function is supposed to do
update_object(obj)
# spend the rest of the function working out which side-effects to fire
if settings.notify_account_handler:
send_notification(obj.account_handler)
if obj.has_changed_foo():
udpate_crm(obj)
```
This results in a codebase is:
* Hard to read
* Hard to test
* Hard to document^
^ Barely a week goes by without someone asking *"what happens when X does Y -
I thought they got email Z?"*
## Solution
This project aims to address all three of the issues above by:
* Removing all side-effects code from core functions
* Simplifying mocking / disabling of side-effects in tests
* Simplifying testing of side-effects only
* Automating documentation of side-effects
It does this with a combination of function decorators that can
be used to build up a global registry of side-effects.
The first decorator, `has_side_effects`, is used to mark a function as one
that has side effects:
```python
# mark this function as one that has side-effects. The label
# can be anything, and is used as a dict key for looking up
# associated side-effects functions
@side_effects.decorators.has_side_effects('update_profile')
def foo(*args, **kwargs):
pass
```
**Decorating view functions**
By default, the `has_side_effects` decorator will run so long as the inner
function does not raise an exception. View functions, however, are a paticular
case where the function may run, and return a perfectly valid `HttpResponse`
object, but you do **not** want the side effects to run, as the response object
has a `status_code` of 404, 500, etc. In this case, you want to inspect the
inner function return value before deciding whether to fire the side effects
functions. In order to support this, the `has_side_effects` decorator has
a kwarg `run_on_exit` which takes a function that takes a single parameter,
the return value from the inner function, and must return `True` or `False`
which determines whether to run the side effects.
The `decorators` module contains the default argument for this kwarg, a
function called `http_response_check`. This will return `False` if the
inner function return value is an `HttpResponse` object with a status
code in the 4xx-5xx range.
The second decorator, `is_side_effect_of`, is used to bind those functions
that implement the side effects to the origin function:
```python
# bind this function to the event 'update_profile'
@is_side_effect_of('update_profile')
def send_updates(*args, **kwargs):
"""Update CRM system."""
pass
# bind this function also to 'update_profile'
@is_side_effect_of('update_profile')
def send_notifications(*args, **kwargs):
"""Notify account managers."""
pass
```
In the above example, the updates and notifications have been separated
out from the origin function, which is now easier to understand as it is
only responsible for its own functionality. In this example we have two
side-effects bound to the same origin, however this is an implementation
detail - you could have a single function implementing all the
side-effects, or split them out further into the individual external
systems.
**Passing origin function return value to side-effects handlers**
By default, side-effects handling functions must have the same function
signature as the origin function. (Internally the `(*args, **kwargs)`
are just a straight pass-through to the handler.) However, in certain
cases it is very useful to have access to the origin function return
value. A common case is where the origin function creates a new object.
The framework handles this internally by introspecting the handler
function, and looking for `**kwargs`.
This is best illustrated with an example:
```python
@has_side_effects("foo")
def origin_func(arg1: int, arg2: int) -> int:
return arg1 + arg2
@is_side_effect_of("foo")
def handle_func1(arg1, arg2):
# this func will not receive the return_value, as
# no kwargs are specified
@is_side_effect_of("foo")
def handle_func1(arg1, arg2, **kwargs):
# this func will receive the return_value via **kwargs
assert "return_value" in kwargs
@is_side_effect_of("foo")
def handle_func1(arg1, arg2, return_value=None):
# this func will receive the return_value
@is_side_effect_of("foo")
def handle_func1(arg1, arg2, return_value):
# this func will receive the return_value, as it is a named arg,
# and there is no *args variable
@is_side_effect_of("foo")
def handle_func1(*args, return_value):
# this func will *NOT* receive the return_value
```
Internally, the app maintains a registry of side-effects functions bound
to origin functions using the text labels. The docstrings for all the
bound functions can be grouped using these labels, and then be printed
out using the management command `display_side_effects`:
```shell
$ ./manage.py display_side_effects
This command prints out the first line from the docstrings of all functions
registered using the @is_side_effect decorator, grouped by label.
update_profile:
- Update CRM system.
- Notify account managers.
close_account:
- Send confirmation email to user.
- Notify customer service.
```
If you have a lot of side-effects wired up, you can filter the list by the label:
```shell
$ ./manage.py display_side_effects --label update_profile
update_profile:
- Update CRM system.
- Notify account managers.
```
Or by a partial match on the event label:
```shell
$ ./manage.py display_side_effects --label-contains profile
update_profile:
- Update CRM system.
- Notify account managers.
```
If you want to enforce docstrings on side-effect functions, then you can use the
`--check-docstrings` option, which will exit with a non-zero exit code if any
docstrings are missing. This can be used as part of a CI process, failing any
build that does not have all its functions documented. (The exit code is the count
of functions without docstrings).
```shell
$ ./manage.py display_side_effects --check-docstrings
update_profile:
*** DOCSTRING MISSING: update_crm ***
- Notify account managers.
ERROR: InvocationError for command '...' (exited with code 1)
```
## Why not use signals?
The above solution probably looks extremely familiar - and it is very closely
related to the built-in Django signals implementation. You could easily
reproduce the output of this project using signals - this project is really
just a formalisation of the way in which a signal-like pattern could be used
to make your code clear and easy to document. The key differences are:
1. Explicit statement that a function has side-effects
2. A simpler binding mechanism (using text labels)
3. (TODO) Async processing of receiver functions
It may well be that this project merges back in to the signals pattern in
due course - at the moment we are still experimenting.
## Installation
The project is available through PyPI as `django-side-effects`:
```shell
$ pip install django-side-effects
```
And the main package itself is just `side_effects`:
```python
>>> from side_effects import decorators
```
## Tests
The project has pretty good test coverage (>90%) and the tests themselves run through `tox`.
```shell
$ pip install tox
$ tox
```
If you want to run the tests manually, make sure you install the requirements, and Django.
```shell
$ pip install django==2.0 # your version goes here
$ tox
```
If you are hacking on the project, please keep coverage up.
NB If you implement side-effects in your project, you will most likely want to be able to turn off the side-effects when testing your own code (so that you are not actually sending emails, updating systems), but you also probably want to know that the side-effects events that you are expecting are fired.
The following code snippet shows how to use the `disable_side_effects` context manager, which returns a list of all the side-effects events that are fired. There is a matching function decorator, which will append the events list as an arg to the decorated function, in the same manner that `unittest.mock.patch` does.
```python
from side_effects import decorators, registry
@decorators.has_side_effects('do_foo')
def foo():
pass
def test_foo():
# to disable side-effects temporarily, use decorator
with registry.disable_side_effects() as events:
foo()
assert events == ['do_foo']
foo()
assert events == ['do_foo', 'do_foo']
# events list is added to the test function as an arg
@decorators.disable_side_effects()
def test_foo_without_side_effects(events: list[str]):
foo()
assert events == ['do_foo']
```
In addition to these testing tools there is a universal 'kill-switch' which can be set using the env var `SIDE_EFFECTS_TEST_MODE=True`. This will completely disable all side-effects events. It is a useful tool when you are migrating a project over to the side_effects pattern - as it can highlight where existing tests are relying on side-effects from firing. Use with caution.
## Contributing
Standard GH rules apply: clone the repo to your own account, create a branch, make sure you update the tests, and submit a pull request.
## Status
We are using it at YunoJuno, but 'caveat emptor'. It does what we need it to do right now, and we will extend it as we evolve. If you need or want additional features, get involved :-).
Raw data
{
"_id": null,
"home_page": "https://github.com/yunojuno/django-side-effects",
"name": "django-side-effects",
"maintainer": "YunoJuno",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "code@yunojuno.com",
"keywords": "",
"author": "YunoJuno",
"author_email": "code@yunojuno.com",
"download_url": "https://files.pythonhosted.org/packages/b8/87/1ca1d44258894806c31574f2ccd6db0aaae7bedb3a3ffba50c8bc0d62acd/django_side_effects-3.0.1.tar.gz",
"platform": null,
"description": "# Django Side Effects\n\nDjango app for managing external side effects.\n\n## Compatibility\n\n**This project now supports Python 3.8+ and Django 3.2+ only on master.**\n\nLegacy versions are tagged.\n\n## Background\n\nThis project was created to try and bring some order to the use of\nexternal side-effects within the YunoJuno platform. External\nside-effects are (as defined by us) those actions that affect external\nsystems, and that are not part of the core application integrity. They\nfall into two main categories within our application - *notifications*\nand *updates*, and are best illustrated by example:\n\n**Notifications**\n\n* Slack messages\n* SMS (via Twilio)\n* Push notifications\n* Email\n\n**Updates**\n\n* Base CRM (sales)\n* Mailchimp CRM (marketing)\n* Elasticsearch (full-text index)\n\nThere are some shared aspects of all of these side-effects:\n\n1. They can all be processed asynchronously (queued)\n2. They can all be replayed (and are idempotent)\n3. They can be executed in any order\n4. They are not time critical\n5. They do not affect the state of the Django application\n\nAs we have continued to build out YunoJuno our use of these side-effects\nhas become ever more complex, and has in some areas left us with functions\nthat are 80% side-effects:\n\n```python\ndef foo():\n # do the thing the function is supposed to do\n update_object(obj)\n # spend the rest of the function working out which side-effects to fire\n if settings.notify_account_handler:\n send_notification(obj.account_handler)\n if obj.has_changed_foo():\n udpate_crm(obj)\n```\n\n\nThis results in a codebase is:\n\n* Hard to read\n* Hard to test\n* Hard to document^\n\n^ Barely a week goes by without someone asking *\"what happens when X does Y -\nI thought they got email Z?\"*\n\n## Solution\n\nThis project aims to address all three of the issues above by:\n\n* Removing all side-effects code from core functions\n* Simplifying mocking / disabling of side-effects in tests\n* Simplifying testing of side-effects only\n* Automating documentation of side-effects\n\nIt does this with a combination of function decorators that can\nbe used to build up a global registry of side-effects.\n\nThe first decorator, `has_side_effects`, is used to mark a function as one\nthat has side effects:\n\n```python\n# mark this function as one that has side-effects. The label\n# can be anything, and is used as a dict key for looking up\n# associated side-effects functions\n@side_effects.decorators.has_side_effects('update_profile')\ndef foo(*args, **kwargs):\n pass\n```\n\n**Decorating view functions**\n\nBy default, the `has_side_effects` decorator will run so long as the inner\nfunction does not raise an exception. View functions, however, are a paticular\ncase where the function may run, and return a perfectly valid `HttpResponse`\nobject, but you do **not** want the side effects to run, as the response object\nhas a `status_code` of 404, 500, etc. In this case, you want to inspect the\ninner function return value before deciding whether to fire the side effects\nfunctions. In order to support this, the `has_side_effects` decorator has\na kwarg `run_on_exit` which takes a function that takes a single parameter,\nthe return value from the inner function, and must return `True` or `False`\nwhich determines whether to run the side effects.\n\nThe `decorators` module contains the default argument for this kwarg, a\nfunction called `http_response_check`. This will return `False` if the\ninner function return value is an `HttpResponse` object with a status\ncode in the 4xx-5xx range.\n\n\nThe second decorator, `is_side_effect_of`, is used to bind those functions\nthat implement the side effects to the origin function:\n\n```python\n# bind this function to the event 'update_profile'\n@is_side_effect_of('update_profile')\ndef send_updates(*args, **kwargs):\n \"\"\"Update CRM system.\"\"\"\n pass\n\n# bind this function also to 'update_profile'\n@is_side_effect_of('update_profile')\ndef send_notifications(*args, **kwargs):\n \"\"\"Notify account managers.\"\"\"\n pass\n```\n\nIn the above example, the updates and notifications have been separated\nout from the origin function, which is now easier to understand as it is\nonly responsible for its own functionality. In this example we have two\nside-effects bound to the same origin, however this is an implementation\ndetail - you could have a single function implementing all the\nside-effects, or split them out further into the individual external\nsystems.\n\n**Passing origin function return value to side-effects handlers**\n\nBy default, side-effects handling functions must have the same function\nsignature as the origin function. (Internally the `(*args, **kwargs)`\nare just a straight pass-through to the handler.) However, in certain\ncases it is very useful to have access to the origin function return\nvalue. A common case is where the origin function creates a new object.\nThe framework handles this internally by introspecting the handler\nfunction, and looking for `**kwargs`.\n\nThis is best illustrated with an example:\n\n```python\n@has_side_effects(\"foo\")\ndef origin_func(arg1: int, arg2: int) -> int:\n return arg1 + arg2\n\n@is_side_effect_of(\"foo\")\ndef handle_func1(arg1, arg2):\n # this func will not receive the return_value, as\n # no kwargs are specified\n\n@is_side_effect_of(\"foo\")\ndef handle_func1(arg1, arg2, **kwargs):\n # this func will receive the return_value via **kwargs\n assert \"return_value\" in kwargs\n\n@is_side_effect_of(\"foo\")\ndef handle_func1(arg1, arg2, return_value=None):\n # this func will receive the return_value\n\n@is_side_effect_of(\"foo\")\ndef handle_func1(arg1, arg2, return_value):\n # this func will receive the return_value, as it is a named arg,\n # and there is no *args variable\n\n@is_side_effect_of(\"foo\")\ndef handle_func1(*args, return_value):\n # this func will *NOT* receive the return_value\n```\n\nInternally, the app maintains a registry of side-effects functions bound\nto origin functions using the text labels. The docstrings for all the\nbound functions can be grouped using these labels, and then be printed\nout using the management command `display_side_effects`:\n\n```shell\n$ ./manage.py display_side_effects\n\nThis command prints out the first line from the docstrings of all functions\nregistered using the @is_side_effect decorator, grouped by label.\n\nupdate_profile:\n\n - Update CRM system.\n - Notify account managers.\n\nclose_account:\n\n - Send confirmation email to user.\n - Notify customer service.\n```\n\nIf you have a lot of side-effects wired up, you can filter the list by the label:\n\n```shell\n$ ./manage.py display_side_effects --label update_profile\n\nupdate_profile:\n - Update CRM system.\n - Notify account managers.\n```\n\nOr by a partial match on the event label:\n\n```shell\n$ ./manage.py display_side_effects --label-contains profile\n\nupdate_profile:\n - Update CRM system.\n - Notify account managers.\n```\n\nIf you want to enforce docstrings on side-effect functions, then you can use the\n`--check-docstrings` option, which will exit with a non-zero exit code if any\ndocstrings are missing. This can be used as part of a CI process, failing any\nbuild that does not have all its functions documented. (The exit code is the count\nof functions without docstrings).\n\n```shell\n$ ./manage.py display_side_effects --check-docstrings\n\nupdate_profile:\n *** DOCSTRING MISSING: update_crm ***\n - Notify account managers.\n\nERROR: InvocationError for command '...' (exited with code 1)\n```\n\n## Why not use signals?\n\nThe above solution probably looks extremely familiar - and it is very closely\nrelated to the built-in Django signals implementation. You could easily\nreproduce the output of this project using signals - this project is really\njust a formalisation of the way in which a signal-like pattern could be used\nto make your code clear and easy to document. The key differences are:\n\n1. Explicit statement that a function has side-effects\n2. A simpler binding mechanism (using text labels)\n3. (TODO) Async processing of receiver functions\n\nIt may well be that this project merges back in to the signals pattern in\ndue course - at the moment we are still experimenting.\n\n\n## Installation\n\nThe project is available through PyPI as `django-side-effects`:\n\n```shell\n$ pip install django-side-effects\n```\n\nAnd the main package itself is just `side_effects`:\n\n```python\n>>> from side_effects import decorators\n```\n\n## Tests\n\nThe project has pretty good test coverage (>90%) and the tests themselves run through `tox`.\n\n```shell\n$ pip install tox\n$ tox\n```\n\nIf you want to run the tests manually, make sure you install the requirements, and Django.\n\n```shell\n$ pip install django==2.0 # your version goes here\n$ tox\n```\n\nIf you are hacking on the project, please keep coverage up.\n\nNB If you implement side-effects in your project, you will most likely want to be able to turn off the side-effects when testing your own code (so that you are not actually sending emails, updating systems), but you also probably want to know that the side-effects events that you are expecting are fired.\n\nThe following code snippet shows how to use the `disable_side_effects` context manager, which returns a list of all the side-effects events that are fired. There is a matching function decorator, which will append the events list as an arg to the decorated function, in the same manner that `unittest.mock.patch` does.\n\n```python\nfrom side_effects import decorators, registry\n\n@decorators.has_side_effects('do_foo')\ndef foo():\n pass\n\ndef test_foo():\n\n # to disable side-effects temporarily, use decorator\n with registry.disable_side_effects() as events:\n foo()\n assert events == ['do_foo']\n foo()\n assert events == ['do_foo', 'do_foo']\n\n# events list is added to the test function as an arg\n@decorators.disable_side_effects()\ndef test_foo_without_side_effects(events: list[str]):\n foo()\n assert events == ['do_foo']\n```\n\nIn addition to these testing tools there is a universal 'kill-switch' which can be set using the env var `SIDE_EFFECTS_TEST_MODE=True`. This will completely disable all side-effects events. It is a useful tool when you are migrating a project over to the side_effects pattern - as it can highlight where existing tests are relying on side-effects from firing. Use with caution.\n\n## Contributing\n\nStandard GH rules apply: clone the repo to your own account, create a branch, make sure you update the tests, and submit a pull request.\n\n## Status\n\nWe are using it at YunoJuno, but 'caveat emptor'. It does what we need it to do right now, and we will extend it as we evolve. If you need or want additional features, get involved :-).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Django app for managing external side effects.",
"version": "3.0.1",
"project_urls": {
"Homepage": "https://github.com/yunojuno/django-side-effects",
"Repository": "https://github.com/yunojuno/django-side-effects"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7ad7ab8429f5a57e4cf970fe6167b1455605d1be7ea1c35e393aed07b9218a9d",
"md5": "949a4e1992012b5e83260ff2692fd541",
"sha256": "26a1d92c06c3201c4cfc6b2498e27f1cead48322ca730d4d5657994a51db81bb"
},
"downloads": -1,
"filename": "django_side_effects-3.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "949a4e1992012b5e83260ff2692fd541",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 14207,
"upload_time": "2024-02-12T14:53:46",
"upload_time_iso_8601": "2024-02-12T14:53:46.129866Z",
"url": "https://files.pythonhosted.org/packages/7a/d7/ab8429f5a57e4cf970fe6167b1455605d1be7ea1c35e393aed07b9218a9d/django_side_effects-3.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b8871ca1d44258894806c31574f2ccd6db0aaae7bedb3a3ffba50c8bc0d62acd",
"md5": "4a1cdc6d248345c3d7b95877e75ca1af",
"sha256": "931591b498cc513efb317a4f421071625902e9bcaa457ede503ccc10b428e448"
},
"downloads": -1,
"filename": "django_side_effects-3.0.1.tar.gz",
"has_sig": false,
"md5_digest": "4a1cdc6d248345c3d7b95877e75ca1af",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 15010,
"upload_time": "2024-02-12T14:53:47",
"upload_time_iso_8601": "2024-02-12T14:53:47.832681Z",
"url": "https://files.pythonhosted.org/packages/b8/87/1ca1d44258894806c31574f2ccd6db0aaae7bedb3a3ffba50c8bc0d62acd/django_side_effects-3.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-12 14:53:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yunojuno",
"github_project": "django-side-effects",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "django-side-effects"
}
JSON
