# PY To Proto
This library holds utilities for converting in-memory data schema representations to [Protobuf](https://developers.google.com/protocol-buffers). The intent is to allow python libraries to leverage the power of `protobuf` while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
## Why?
The `protobuf` langauge is a powerful tool for defining language-agnostic, composable datastructures. `Protobuf` also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that `protobuf` requires_a static built step to perform this `proto` -> `X` conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
- `Protobuf`:
- **Advantages**
- Compact serialization
- Auto-generated [`grpc`](https://grpc.io/) client and service libraries
- Client libraries can be used from different programming languages
- **Disadvantages**
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Python schemas:
- **Advantages**
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- **Disadvantages**
- Generally, no standard serialization beyond `json`
- No automated service implementations
- No/manual mechanism for usage in other programming languages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create [`google.protobuf.Descriptor`](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/descriptor.py#L245) objects to allow for `protobuf` serialization and deserialization
- Reverse render a `.proto` file from the generated `Descriptor` so that stubs can be generated in other languages
- No static compiliation needed!
## Supported Python Schema Types
Currently, objects can be declared using either [python `dataclasses`](https://docs.python.org/3/library/dataclasses.html) or [Json TypeDef (JTD)](https://jsontypedef.com/). Additional schemas can be added by [subclassing `ConverterBase`](py_to_proto/converter_base.py).
### Dataclass To Proto
The following example illustrates how `dataclasses` and `enums` can be converted to proto:
```py
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
```
### JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
```py
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
```
## Similar Projects
There are a number of similar projects in this space that offer slightly different value:
- [`jtd-codegen`](https://jsontypedef.com/docs/jtd-codegen/): This project focuses on statically generating language-native code (including `python`) to represent the JTD schema.
- [`py-json-to-proto`](https://pypi.org/project/py-json-to-proto/): This project aims to deduce a schema from an instance of a `json` object.
- [`pure-protobuf`](https://pypi.org/project/pure-protobuf/): This project has a very similar aim to `py-to-proto`, but it skips the intermediate `descriptor` representation and thus is not able to produce native `message.Message` classes.
Raw data
{
"_id": null,
"home_page": "https://github.com/IBM/py-to-proto",
"name": "py-to-proto",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "json,json typedef,jtd,protobuf,proto,dataclass",
"author": "Gabe Goodhart",
"author_email": "gabe.l.hart@gmail.com",
"download_url": "",
"platform": null,
"description": "# PY To Proto\n\nThis library holds utilities for converting in-memory data schema representations to [Protobuf](https://developers.google.com/protocol-buffers). The intent is to allow python libraries to leverage the power of `protobuf` while maintaining the source-of-truth for their data in pure python and avoiding static build steps.\n\n## Why?\n\nThe `protobuf` langauge is a powerful tool for defining language-agnostic, composable datastructures. `Protobuf` also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that `protobuf` requires_a static built step to perform this `proto` -> `X` conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:\n\n- `Protobuf`:\n - **Advantages**\n - Compact serialization\n - Auto-generated [`grpc`](https://grpc.io/) client and service libraries\n - Client libraries can be used from different programming languages\n - **Disadvantages**\n - Learning curve to understand the full ecosystem\n - Not a familiar tool outside of service engineering\n - Static compilation step required to use in code\n- Python schemas:\n - **Advantages**\n - Can be learned quickly using pure-python documentation\n - Can be written inline in pure python\n - **Disadvantages**\n - Generally, no standard serialization beyond `json`\n - No automated service implementations\n - No/manual mechanism for usage in other programming languages\n\nThis project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:\n\n- Define your structures in pure python for simplicity\n- Dynamically create [`google.protobuf.Descriptor`](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/descriptor.py#L245) objects to allow for `protobuf` serialization and deserialization\n- Reverse render a `.proto` file from the generated `Descriptor` so that stubs can be generated in other languages\n- No static compiliation needed!\n\n## Supported Python Schema Types\n\nCurrently, objects can be declared using either [python `dataclasses`](https://docs.python.org/3/library/dataclasses.html) or [Json TypeDef (JTD)](https://jsontypedef.com/). Additional schemas can be added by [subclassing `ConverterBase`](py_to_proto/converter_base.py).\n\n### Dataclass To Proto\n\nThe following example illustrates how `dataclasses` and `enums` can be converted to proto:\n\n```py\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom typing import Annotated, Dict, List, Enum\nimport py_to_proto\n\n# Define the Foo structure as a python dataclass, including a nested enum\n@dataclass\nclass Foo:\n\n class BarEnum(Enum):\n EXAM: 0\n JOKE_SETTING: 1\n\n foo: bool\n bar: List[BarEnum]\n\n# Define the Foo protobuf message class\nFooProto = py_to_proto.descriptor_to_message_class(\n py_to_proto.dataclass_to_proto(\n package=\"foobar\",\n dataclass_=Foo,\n )\n)\n\n# Declare the Bar structure as a python dataclass with a reference to the\n# FooProto type\n@dataclass\nclass Bar:\n baz: FooProto\n\n# Define the Bar protobuf message class\nBarProto = py_to_proto.descriptor_to_message_class(\n py_to_proto.dataclass_to_proto(\n package=\"foobar\",\n dataclass_=Bar,\n )\n)\n\n# Instantiate a BarProto\nprint(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))\n\ndef write_protos(proto_dir: str):\n \"\"\"Write out the .proto files for FooProto and BarProto to the given\n directory\n \"\"\"\n FooProto.write_proto_file(proto_dir)\n BarProto.write_proto_file(proto_dir)\n```\n\n### JTD To Proto\n\nThe following example illustrates how JTD schemas can be converted to proto:\n\n```py\nimport py_to_proto\n\n# Declare the Foo protobuf message class\nFoo = py_to_proto.descriptor_to_message_class(\n py_to_proto.jtd_to_proto(\n name=\"Foo\",\n package=\"foobar\",\n jtd_def={\n \"properties\": {\n # Bool field\n \"foo\": {\n \"type\": \"boolean\",\n },\n # Array of nested enum values\n \"bar\": {\n \"elements\": {\n \"enum\": [\"EXAM\", \"JOKE_SETTING\"],\n }\n }\n }\n },\n )\n)\n\n# Declare an object that references Foo as the type for a field\nBar = py_to_proto.descriptor_to_message_class(\n py_to_proto.jtd_to_proto(\n name=\"Bar\",\n package=\"foobar\",\n jtd_def={\n \"properties\": {\n \"baz\": {\n \"type\": Foo.DESCRIPTOR,\n },\n },\n },\n ),\n)\n\ndef write_protos(proto_dir: str):\n \"\"\"Write out the .proto files for Foo and Bar to the given directory\"\"\"\n Foo.write_proto_file(proto_dir)\n Bar.write_proto_file(proto_dir)\n```\n\n## Similar Projects\n\nThere are a number of similar projects in this space that offer slightly different value:\n\n- [`jtd-codegen`](https://jsontypedef.com/docs/jtd-codegen/): This project focuses on statically generating language-native code (including `python`) to represent the JTD schema.\n- [`py-json-to-proto`](https://pypi.org/project/py-json-to-proto/): This project aims to deduce a schema from an instance of a `json` object.\n- [`pure-protobuf`](https://pypi.org/project/pure-protobuf/): This project has a very similar aim to `py-to-proto`, but it skips the intermediate `descriptor` representation and thus is not able to produce native `message.Message` classes.\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool to dynamically create protobuf message classes from python data schemas",
"version": "0.5.2",
"project_urls": {
"Homepage": "https://github.com/IBM/py-to-proto"
},
"split_keywords": [
"json",
"json typedef",
"jtd",
"protobuf",
"proto",
"dataclass"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "eb298b2b5f79f3842406f06b68c4ab74ac43bd9c3ee04c97d93c196544679d7a",
"md5": "5da8f1217a1861d9eb1251881639db11",
"sha256": "9b5d633bb47a96ea8fd61b5ae4993deb62a104c0027831e1b5a226aed6b7fce7"
},
"downloads": -1,
"filename": "py_to_proto-0.5.2-py310-none-any.whl",
"has_sig": false,
"md5_digest": "5da8f1217a1861d9eb1251881639db11",
"packagetype": "bdist_wheel",
"python_version": "py310",
"requires_python": null,
"size": 32866,
"upload_time": "2023-10-12T23:07:52",
"upload_time_iso_8601": "2023-10-12T23:07:52.537104Z",
"url": "https://files.pythonhosted.org/packages/eb/29/8b2b5f79f3842406f06b68c4ab74ac43bd9c3ee04c97d93c196544679d7a/py_to_proto-0.5.2-py310-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7edf1ec98e3a764efe4005bd99da372de179d130c927467ae25cd8b86f9b8aaa",
"md5": "b8c271d8e42d2ce954acbcf859495936",
"sha256": "38c08b63837596b285174288255d28757e1ade55c444391485eb55845f295ba9"
},
"downloads": -1,
"filename": "py_to_proto-0.5.2-py311-none-any.whl",
"has_sig": false,
"md5_digest": "b8c271d8e42d2ce954acbcf859495936",
"packagetype": "bdist_wheel",
"python_version": "py311",
"requires_python": null,
"size": 32866,
"upload_time": "2023-10-12T23:07:59",
"upload_time_iso_8601": "2023-10-12T23:07:59.114245Z",
"url": "https://files.pythonhosted.org/packages/7e/df/1ec98e3a764efe4005bd99da372de179d130c927467ae25cd8b86f9b8aaa/py_to_proto-0.5.2-py311-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "760555f99fc5c4ba9a6cb048377955dbf1bc5b1ca8081e869ae00d48cb624365",
"md5": "bd93d00f678382ca5a725b4ea133f65b",
"sha256": "22021b46f216afa899751102d78ac090f23a757ea97635c77aabeff0106d6dad"
},
"downloads": -1,
"filename": "py_to_proto-0.5.2-py37-none-any.whl",
"has_sig": false,
"md5_digest": "bd93d00f678382ca5a725b4ea133f65b",
"packagetype": "bdist_wheel",
"python_version": "py37",
"requires_python": null,
"size": 32880,
"upload_time": "2023-10-12T23:07:58",
"upload_time_iso_8601": "2023-10-12T23:07:58.649199Z",
"url": "https://files.pythonhosted.org/packages/76/05/55f99fc5c4ba9a6cb048377955dbf1bc5b1ca8081e869ae00d48cb624365/py_to_proto-0.5.2-py37-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "711f47f90ad339e1f3d0f72b58dd39e57786c1422dd32d37363a3bd5ebec91d1",
"md5": "645a49a61a6033efedf3f532eab5408e",
"sha256": "91f48866c132fbd5901c065e73161195c108b3ce73edf549083c79fed5d44fb8"
},
"downloads": -1,
"filename": "py_to_proto-0.5.2-py38-none-any.whl",
"has_sig": false,
"md5_digest": "645a49a61a6033efedf3f532eab5408e",
"packagetype": "bdist_wheel",
"python_version": "py38",
"requires_python": null,
"size": 32880,
"upload_time": "2023-10-12T23:08:11",
"upload_time_iso_8601": "2023-10-12T23:08:11.957673Z",
"url": "https://files.pythonhosted.org/packages/71/1f/47f90ad339e1f3d0f72b58dd39e57786c1422dd32d37363a3bd5ebec91d1/py_to_proto-0.5.2-py38-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "09d3c95198744626cad2e209784cf3acf763e9299b5dd378b7393cfe49966f87",
"md5": "d6f3326bc021391abc0c8c29a1b5abae",
"sha256": "a1a3553b6b0ce87fed08567615c444f1eeafacb3a124920a2ea02b91f62ac0e1"
},
"downloads": -1,
"filename": "py_to_proto-0.5.2-py39-none-any.whl",
"has_sig": false,
"md5_digest": "d6f3326bc021391abc0c8c29a1b5abae",
"packagetype": "bdist_wheel",
"python_version": "py39",
"requires_python": null,
"size": 32881,
"upload_time": "2023-10-12T23:07:49",
"upload_time_iso_8601": "2023-10-12T23:07:49.416228Z",
"url": "https://files.pythonhosted.org/packages/09/d3/c95198744626cad2e209784cf3acf763e9299b5dd378b7393cfe49966f87/py_to_proto-0.5.2-py39-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-12 23:07:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "IBM",
"github_project": "py-to-proto",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "py-to-proto"
}
JSON
