:fire: Deep-Fried Marshmallow – Makes Marshmallow a Chicken Nugget
==================================================================
I need to be honest with you — I have no idea how to compare the speed of a
marshmallow and the speed of a chicken nugget. I really liked that headline,
though, so let's just assume that a nugget is indeed faster than a
marshmallow. So is this project, Deep-Fried Marshmallow, faster than
vanilla Marshmallow. Or, to be precise, it *makes* Marshmallow faster.
Deep-Fried Marshmallow implements a JIT for Marshmallow that speeds up dumping
objects 3-5x (depending on your schema). Deep-Fried Marshmallow allows you to
have the great API that
[Marshmallow](https://github.com/marshmallow-code/marshmallow) provides
without having to sacrifice performance.
```
Benchmark Result:
Original Dump Time: 220.50 usec/dump
Original Load Time: 536.51 usec/load
Optimized Dump Time: 58.67 usec/dump
Optimized Load Time: 118.44 usec/load
Speed up for dump: 3.76x
Speed up for load: 4.53x
```
Deep-Fried Marshmallow is a fork of the great
[Toasted Marshmallow](https://github.com/lyft/toasted-marshmallow) project that,
sadly, has been abandoned for years. Deep-Fried Marshmallow introduces many
changes that make it compatible with all latest versions of Marshmallow (3.13+).
It also changes the way the library interacts with Marshmallow, which means
that code of Marshmallow doesn't need to be forked and modified for the JIT
magic to work. That's a whole new level of magic!
## Installing Deep-Fried Marshmallow
```bash
pip install DeepFriedMarshmallow
# or, if your project uses Poetry:
poetry install DeepFriedMarshmallow
```
If your project doesn't have vanilla Marshmallow specified in requirements,
the latest version of it will be installed alongside Deep-Fried Marshmallow.
You are free to pin any version of it that you need, as long as it's
newer than v3.13.
## Enabling Deep-Fried Marshmallow
Enabling Deep-Fried Marshmallow on an existing schema is just one change of code. Change your schemas to inherit from the `JitSchema` class in the `deepfriedmarshmallow` package instead of `Schema` from `marshmallow`.
For example, this block of code:
```python
from marshmallow import Schema, fields
class ArtistSchema(Schema):
name = fields.Str()
class AlbumSchema(Schema):
title = fields.Str()
release_date = fields.Date()
artist = fields.Nested(ArtistSchema())
schema = AlbumSchema()
```
Should become this:
```python
from marshmallow import fields
from deepfriedmarshmallow import JitSchema
class ArtistSchema(JitSchema):
name = fields.Str()
class AlbumSchema(JitSchema):
title = fields.Str()
release_date = fields.Date()
artist = fields.Nested(ArtistSchema())
schema = AlbumSchema()
```
And that's it!
### Auto-patching all Marshmallow schemas
If you want to automatically patch all Marshmallow schemas in your project,
Deep-Fried Marshmallow provides a helper function for that. Just call
`deepfriedmarshmallow.deep_fry_marshmallow()` before you start using
Marshmallow schemas, and you're all set. The upmost ``__init__.py`` file of
your project is a good place to do that.
```python
# your_package/__init__.py
from deepfriedmarshmallow import deep_fry_marshmallow
deep_fry_marshmallow()
```
All imports of `marshmallow.Schema` will be automatically replaced with
`deepfriedmarshmallow.Schema` with no other changes to your code. Isn't that
~~sweet~~ extra crispy?
### Custom Schema classes
Deep-Fried Marshmallow also provides a mixin class that you can use to create
or extend custom Schema classes. To use it, just inherit from `JitSchemaMixin`.
Let's take a look at the following example:
```python
from marshmallow import fields
class ClockSchema(MyAwesomeBaseSchema):
time = fields.DateTime(data_key="Time")
```
If you want to make this schema JIT-compatible, and don't want to modify the
`MyAwesomeBaseSchema` class to inherit from `deepfriedmarshmallow.Schema`,
you can do the following:
```python
from marshmallow import fields
from deepfriedmarshmallow import JitSchemaMixin
class ClockSchema(JitSchemaMixin, MyAwesomeBaseSchema):
time = fields.DateTime(data_key="Time")
```
### Patcher functions
If all of the above wasn't enough, Deep-Fried Marshmallow also provides two
more ways to patch Marshmallow schemas. Both of them are functions that you
can call to patch either a Schema class, or a Schema instance. Let's take a
look at the following example:
```python
from marshmallow import Schema, fields
from deepfriedmarshmallow import deep_fry_schema
class ArtistSchema(Schema):
name = fields.Str()
deep_fry_schema(ArtistSchema)
schema = ArtistSchema()
```
The `deep_fry_schema` function will patch the `AlbumSchema` class, and all
instances of it will be JIT-compatible. If you want to patch a specific
instance of a schema, you can use the `deep_fry_schema_object` function:
```python
from marshmallow import Schema, fields
from deepfriedmarshmallow import deep_fry_schema_object
class ArtistSchema(Schema):
name = fields.Str()
schema = ArtistSchema()
deep_fry_schema_object(schema)
```
This function will patch the `schema` object, and all dumps and loads will
be JIT-compatible. This function is useful if you want to patch a schema
that you don't have control over, for example, a schema that is provided
by a third-party library.
## How it works
Deep-Fried Marshmallow works by generating code at runtime to optimize dumping
objects without going through layers and layers of reflection. The generated
code optimistically assumes the objects being passed in are schematically valid,
falling back to the original Marshmallow code on failure.
For example, taking `AlbumSchema` from above, Deep-Fried Marshmallow will
generate the following methods:
```python
def InstanceSerializer(obj):
res = {}
value = obj.title; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
value = obj.release_date; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
value = obj.artist; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
return res
def DictSerializer(obj):
res = {}
if "title" in obj:
value = obj["title"]; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
if "release_date" in obj:
value = obj["release_date"]; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
if "artist" in obj:
value = obj["artist"]; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
return res
def HybridSerializer(obj):
res = {}
try:
value = obj["title"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.title
value = value; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
try:
value = obj["release_date"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.release_date
value = value; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
try:
value = obj["artist"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.artist
value = value; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
return res
```
Deep-Fried Marshmallow will invoke the proper serializer based upon the input.
Since Deep-Fried Marshmallow generates code at runtime, it's critical you
re-use Schema objects. If you're creating a new Schema object every time you
serialize or deserialize an object, you're likely to experience much worse
performance.
## Special thanks to
* [@rowillia](https://github.com/rowillia)/[@lyft](https://github.com/lyft) — for creating Toasted Marshmallow
* [@taion](https://github.com/taion) — for a [PoC](https://github.com/lyft/toasted-marshmallow/pull/16) of injecting the JIT compiler by replacing the marshaller
* [@Kalepa](https://github.com/Kalepa) — for needing improved Marshmallow performance so that I could actually work on this project 😅
## License
See [LICENSE](/LICENSE) for details.
## Contributing
Contributions, issues and feature requests are welcome!
Feel free to check [existing issues](https://github.com/mLupine/DeepFriedMarshmallow/issues) before reporting a new one.
## Show your support
Give this repository a ⭐️ if this project helped you!
Raw data
{
"_id": null,
"home_page": "https://github.com/mLupine/DeepFriedMarshmallow",
"name": "DeepFriedMarshmallow",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0.0",
"maintainer_email": "",
"keywords": "serialization,rest,json,api,marshal,marshalling,deserialization,validation,schema",
"author": "Maciej Wilczynski",
"author_email": "oss@lupin.rocks",
"download_url": "https://files.pythonhosted.org/packages/b6/15/b5138f18ffe6f075dfc3b8acac29daae9b47a4173f4929840b0ffafd7efd/deepfriedmarshmallow-1.0.0b1.tar.gz",
"platform": null,
"description": ":fire: Deep-Fried Marshmallow \u2013 Makes Marshmallow a Chicken Nugget\n==================================================================\n\nI need to be honest with you \u2014 I have no idea how to compare the speed of a\nmarshmallow and the speed of a chicken nugget. I really liked that headline,\nthough, so let's just assume that a nugget is indeed faster than a \nmarshmallow. So is this project, Deep-Fried Marshmallow, faster than \nvanilla Marshmallow. Or, to be precise, it *makes* Marshmallow faster.\n\nDeep-Fried Marshmallow implements a JIT for Marshmallow that speeds up dumping\nobjects 3-5x (depending on your schema). Deep-Fried Marshmallow allows you to\nhave the great API that \n[Marshmallow](https://github.com/marshmallow-code/marshmallow) provides\nwithout having to sacrifice performance.\n```\n Benchmark Result:\n Original Dump Time: 220.50 usec/dump\n Original Load Time: 536.51 usec/load\n Optimized Dump Time: 58.67 usec/dump\n Optimized Load Time: 118.44 usec/load\n\n Speed up for dump: 3.76x\n Speed up for load: 4.53x\n```\n\nDeep-Fried Marshmallow is a fork of the great \n[Toasted Marshmallow](https://github.com/lyft/toasted-marshmallow) project that,\nsadly, has been abandoned for years. Deep-Fried Marshmallow introduces many\nchanges that make it compatible with all latest versions of Marshmallow (3.13+).\nIt also changes the way the library interacts with Marshmallow, which means\nthat code of Marshmallow doesn't need to be forked and modified for the JIT\nmagic to work. That's a whole new level of magic!\n\n\n\n## Installing Deep-Fried Marshmallow\n\n\n```bash\npip install DeepFriedMarshmallow\n# or, if your project uses Poetry:\npoetry install DeepFriedMarshmallow\n```\n\nIf your project doesn't have vanilla Marshmallow specified in requirements,\nthe latest version of it will be installed alongside Deep-Fried Marshmallow.\nYou are free to pin any version of it that you need, as long as it's\nnewer than v3.13.\n\n\n## Enabling Deep-Fried Marshmallow\n\nEnabling Deep-Fried Marshmallow on an existing schema is just one change of code. Change your schemas to inherit from the `JitSchema` class in the `deepfriedmarshmallow` package instead of `Schema` from `marshmallow`.\n\nFor example, this block of code:\n\n```python\nfrom marshmallow import Schema, fields\n\nclass ArtistSchema(Schema):\n name = fields.Str()\n\nclass AlbumSchema(Schema):\n title = fields.Str()\n release_date = fields.Date()\n artist = fields.Nested(ArtistSchema())\n\nschema = AlbumSchema()\n```\n\nShould become this:\n```python\nfrom marshmallow import fields\nfrom deepfriedmarshmallow import JitSchema\n\nclass ArtistSchema(JitSchema):\n name = fields.Str()\n\nclass AlbumSchema(JitSchema):\n title = fields.Str()\n release_date = fields.Date()\n artist = fields.Nested(ArtistSchema())\n\nschema = AlbumSchema()\n```\n\nAnd that's it!\n\n### Auto-patching all Marshmallow schemas\n\nIf you want to automatically patch all Marshmallow schemas in your project,\nDeep-Fried Marshmallow provides a helper function for that. Just call \n`deepfriedmarshmallow.deep_fry_marshmallow()` before you start using \nMarshmallow schemas, and you're all set. The upmost ``__init__.py`` file of\nyour project is a good place to do that.\n\n```python\n# your_package/__init__.py\nfrom deepfriedmarshmallow import deep_fry_marshmallow\n\ndeep_fry_marshmallow()\n```\n\nAll imports of `marshmallow.Schema` will be automatically replaced with \n`deepfriedmarshmallow.Schema` with no other changes to your code. Isn't that\n~~sweet~~ extra crispy?\n\n### Custom Schema classes\n\nDeep-Fried Marshmallow also provides a mixin class that you can use to create\nor extend custom Schema classes. To use it, just inherit from `JitSchemaMixin`.\nLet's take a look at the following example:\n\n```python\nfrom marshmallow import fields\n\nclass ClockSchema(MyAwesomeBaseSchema):\n time = fields.DateTime(data_key=\"Time\")\n```\n\nIf you want to make this schema JIT-compatible, and don't want to modify the\n`MyAwesomeBaseSchema` class to inherit from `deepfriedmarshmallow.Schema`, \nyou can do the following:\n\n```python\nfrom marshmallow import fields\nfrom deepfriedmarshmallow import JitSchemaMixin\n\nclass ClockSchema(JitSchemaMixin, MyAwesomeBaseSchema):\n time = fields.DateTime(data_key=\"Time\")\n```\n\n### Patcher functions\n\nIf all of the above wasn't enough, Deep-Fried Marshmallow also provides two\nmore ways to patch Marshmallow schemas. Both of them are functions that you\ncan call to patch either a Schema class, or a Schema instance. Let's take a\nlook at the following example:\n\n```python\nfrom marshmallow import Schema, fields\nfrom deepfriedmarshmallow import deep_fry_schema\n\nclass ArtistSchema(Schema):\n name = fields.Str()\n\ndeep_fry_schema(ArtistSchema)\nschema = ArtistSchema()\n```\n\nThe `deep_fry_schema` function will patch the `AlbumSchema` class, and all\ninstances of it will be JIT-compatible. If you want to patch a specific\ninstance of a schema, you can use the `deep_fry_schema_object` function:\n\n```python\nfrom marshmallow import Schema, fields\nfrom deepfriedmarshmallow import deep_fry_schema_object\n\nclass ArtistSchema(Schema):\n name = fields.Str()\n\nschema = ArtistSchema()\ndeep_fry_schema_object(schema)\n```\n\nThis function will patch the `schema` object, and all dumps and loads will\nbe JIT-compatible. This function is useful if you want to patch a schema\nthat you don't have control over, for example, a schema that is provided\nby a third-party library.\n\n## How it works\n\nDeep-Fried Marshmallow works by generating code at runtime to optimize dumping\nobjects without going through layers and layers of reflection. The generated\ncode optimistically assumes the objects being passed in are schematically valid,\nfalling back to the original Marshmallow code on failure.\n\nFor example, taking `AlbumSchema` from above, Deep-Fried Marshmallow will\ngenerate the following methods:\n\n```python\ndef InstanceSerializer(obj):\n res = {}\n value = obj.title; value = value() if callable(value) else value; value = str(value) if value is not None else None; res[\"title\"] = value\n value = obj.release_date; value = value() if callable(value) else value; res[\"release_date\"] = _field_release_date__serialize(value, \"release_date\", obj)\n value = obj.artist; value = value() if callable(value) else value; res[\"artist\"] = _field_artist__serialize(value, \"artist\", obj)\n return res\n\ndef DictSerializer(obj):\n res = {}\n if \"title\" in obj:\n value = obj[\"title\"]; value = value() if callable(value) else value; value = str(value) if value is not None else None; res[\"title\"] = value\n if \"release_date\" in obj:\n value = obj[\"release_date\"]; value = value() if callable(value) else value; res[\"release_date\"] = _field_release_date__serialize(value, \"release_date\", obj)\n if \"artist\" in obj:\n value = obj[\"artist\"]; value = value() if callable(value) else value; res[\"artist\"] = _field_artist__serialize(value, \"artist\", obj)\n return res\n\ndef HybridSerializer(obj):\n res = {}\n try:\n value = obj[\"title\"]\n except (KeyError, AttributeError, IndexError, TypeError):\n value = obj.title\n value = value; value = value() if callable(value) else value; value = str(value) if value is not None else None; res[\"title\"] = value\n try:\n value = obj[\"release_date\"]\n except (KeyError, AttributeError, IndexError, TypeError):\n value = obj.release_date\n value = value; value = value() if callable(value) else value; res[\"release_date\"] = _field_release_date__serialize(value, \"release_date\", obj)\n try:\n value = obj[\"artist\"]\n except (KeyError, AttributeError, IndexError, TypeError):\n value = obj.artist\n value = value; value = value() if callable(value) else value; res[\"artist\"] = _field_artist__serialize(value, \"artist\", obj)\n return res\n```\n\nDeep-Fried Marshmallow will invoke the proper serializer based upon the input.\n\nSince Deep-Fried Marshmallow generates code at runtime, it's critical you\nre-use Schema objects. If you're creating a new Schema object every time you\nserialize or deserialize an object, you're likely to experience much worse \nperformance.\n\n## Special thanks to\n * [@rowillia](https://github.com/rowillia)/[@lyft](https://github.com/lyft) \u2014 for creating Toasted Marshmallow\n * [@taion](https://github.com/taion) \u2014 for a [PoC](https://github.com/lyft/toasted-marshmallow/pull/16) of injecting the JIT compiler by replacing the marshaller\n * [@Kalepa](https://github.com/Kalepa) \u2014 for needing improved Marshmallow performance so that I could actually work on this project \ud83d\ude05\n\n## License\nSee [LICENSE](/LICENSE) for details.\n\n## Contributing\n\nContributions, issues and feature requests are welcome!\n\nFeel free to check [existing issues](https://github.com/mLupine/DeepFriedMarshmallow/issues) before reporting a new one.\n\n## Show your support\nGive this repository a \u2b50\ufe0f if this project helped you!\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A plug-and-play JIT implementation for Marshmallow to speed up data serialization and deserialization",
"version": "1.0.0b1",
"project_urls": {
"Homepage": "https://github.com/mLupine/DeepFriedMarshmallow"
},
"split_keywords": [
"serialization",
"rest",
"json",
"api",
"marshal",
"marshalling",
"deserialization",
"validation",
"schema"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f3abb28b9c391382ad581887220ef8a32d6abc6245dd6f1ceb841e5d8b2b9271",
"md5": "6bfcc4e9199af3aa39be796ab39d8751",
"sha256": "5bb814b2adecc5ae5bbfd7f736abda6ab974448b0311c46568f7b3558153634e"
},
"downloads": -1,
"filename": "deepfriedmarshmallow-1.0.0b1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6bfcc4e9199af3aa39be796ab39d8751",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0.0",
"size": 19813,
"upload_time": "2023-10-06T09:42:45",
"upload_time_iso_8601": "2023-10-06T09:42:45.213037Z",
"url": "https://files.pythonhosted.org/packages/f3/ab/b28b9c391382ad581887220ef8a32d6abc6245dd6f1ceb841e5d8b2b9271/deepfriedmarshmallow-1.0.0b1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b615b5138f18ffe6f075dfc3b8acac29daae9b47a4173f4929840b0ffafd7efd",
"md5": "5618d0fb80001a74a5525771bfb078a2",
"sha256": "b16f686ca501ec36284f42ba777583d3d32805740e1d3d5c6597d254b6e23e2f"
},
"downloads": -1,
"filename": "deepfriedmarshmallow-1.0.0b1.tar.gz",
"has_sig": false,
"md5_digest": "5618d0fb80001a74a5525771bfb078a2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0.0",
"size": 19908,
"upload_time": "2023-10-06T09:42:47",
"upload_time_iso_8601": "2023-10-06T09:42:47.119706Z",
"url": "https://files.pythonhosted.org/packages/b6/15/b5138f18ffe6f075dfc3b8acac29daae9b47a4173f4929840b0ffafd7efd/deepfriedmarshmallow-1.0.0b1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-06 09:42:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mLupine",
"github_project": "DeepFriedMarshmallow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "deepfriedmarshmallow"
}
JSON
