# DEPRECATED
This project has been renamed to `py-to-proto` to reflect its expansion to include other input schema formats. Please see https://pypi.org/project/py-to-proto/
# PY To Proto
This library holds utilities for converting in-memory data schema representations to [Protobuf](https://developers.google.com/protocol-buffers). The intent is to allow python libraries to leverage the power of `protobuf` while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
## Why?
The `protobuf` langauge is a powerful tool for defining language-agnostic, composable datastructures. `Protobuf` also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that `protobuf` requires_a static built step to perform this `proto` -> `X` conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
- `Protobuf`:
- **Advantages**
- Compact serialization
- Auto-generated [`grpc`](https://grpc.io/) client and service libraries
- Client libraries can be used from different programming languages
- **Disadvantages**
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Python schemas:
- **Advantages**
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- **Disadvantages**
- Generally, no standard serialization beyond `json`
- No automated service implementations
- No/manual mechanism for usage in other programming languages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create [`google.protobuf.Descriptor`](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/descriptor.py#L245) objects to allow for `protobuf` serialization and deserialization
- Reverse render a `.proto` file from the generated `Descriptor` so that stubs can be generated in other languages
- No static compiliation needed!
## Supported Python Schema Types
Currently, objects can be declared using either [python `dataclasses`](https://docs.python.org/3/library/dataclasses.html) or [Json TypeDef (JTD)](https://jsontypedef.com/). Additional schemas can be added by [subclassing `ConverterBase`](py_to_proto/converter_base.py).
### Dataclass To Proto
The following example illustrates how `dataclasses` and `enums` can be converted to proto:
```py
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
```
### JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
```py
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
```
## Similar Projects
There are a number of similar projects in this space that offer slightly different value:
- [`jtd-codegen`](https://jsontypedef.com/docs/jtd-codegen/): This project focuses on statically generating language-native code (including `python`) to represent the JTD schema.
- [`py-json-to-proto`](https://pypi.org/project/py-json-to-proto/): This project aims to deduce a schema from an instance of a `json` object.
- [`pure-protobuf`](https://pypi.org/project/pure-protobuf/): This project has a very similar aim to `py-to-proto`, but it skips the intermediate `descriptor` representation and thus is not able to produce native `message.Message` classes.
Raw data
{
"_id": null,
"home_page": "https://github.com/IBM/py-to-proto",
"name": "jtd-to-proto",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "json,json typedef,jtd,protobuf,proto,dataclass",
"author": "Gabe Goodhart",
"author_email": "gabe.l.hart@gmail.com",
"download_url": "",
"platform": null,
"description": "\n# DEPRECATED\n\nThis project has been renamed to `py-to-proto` to reflect its expansion to include other input schema formats. Please see https://pypi.org/project/py-to-proto/\n\n# PY To Proto\n\nThis library holds utilities for converting in-memory data schema representations to [Protobuf](https://developers.google.com/protocol-buffers). The intent is to allow python libraries to leverage the power of `protobuf` while maintaining the source-of-truth for their data in pure python and avoiding static build steps.\n\n## Why?\n\nThe `protobuf` langauge is a powerful tool for defining language-agnostic, composable datastructures. `Protobuf` also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that `protobuf` requires_a static built step to perform this `proto` -> `X` conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:\n\n- `Protobuf`:\n - **Advantages**\n - Compact serialization\n - Auto-generated [`grpc`](https://grpc.io/) client and service libraries\n - Client libraries can be used from different programming languages\n - **Disadvantages**\n - Learning curve to understand the full ecosystem\n - Not a familiar tool outside of service engineering\n - Static compilation step required to use in code\n- Python schemas:\n - **Advantages**\n - Can be learned quickly using pure-python documentation\n - Can be written inline in pure python\n - **Disadvantages**\n - Generally, no standard serialization beyond `json`\n - No automated service implementations\n - No/manual mechanism for usage in other programming languages\n\nThis project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:\n\n- Define your structures in pure python for simplicity\n- Dynamically create [`google.protobuf.Descriptor`](https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/descriptor.py#L245) objects to allow for `protobuf` serialization and deserialization\n- Reverse render a `.proto` file from the generated `Descriptor` so that stubs can be generated in other languages\n- No static compiliation needed!\n\n## Supported Python Schema Types\n\nCurrently, objects can be declared using either [python `dataclasses`](https://docs.python.org/3/library/dataclasses.html) or [Json TypeDef (JTD)](https://jsontypedef.com/). Additional schemas can be added by [subclassing `ConverterBase`](py_to_proto/converter_base.py).\n\n### Dataclass To Proto\n\nThe following example illustrates how `dataclasses` and `enums` can be converted to proto:\n\n```py\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom typing import Annotated, Dict, List, Enum\nimport py_to_proto\n\n# Define the Foo structure as a python dataclass, including a nested enum\n@dataclass\nclass Foo:\n\n class BarEnum(Enum):\n EXAM: 0\n JOKE_SETTING: 1\n\n foo: bool\n bar: List[BarEnum]\n\n# Define the Foo protobuf message class\nFooProto = py_to_proto.descriptor_to_message_class(\n py_to_proto.dataclass_to_proto(\n package=\"foobar\",\n dataclass_=Foo,\n )\n)\n\n# Declare the Bar structure as a python dataclass with a reference to the\n# FooProto type\n@dataclass\nclass Bar:\n baz: FooProto\n\n# Define the Bar protobuf message class\nBarProto = py_to_proto.descriptor_to_message_class(\n py_to_proto.dataclass_to_proto(\n package=\"foobar\",\n dataclass_=Bar,\n )\n)\n\n# Instantiate a BarProto\nprint(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))\n\ndef write_protos(proto_dir: str):\n \"\"\"Write out the .proto files for FooProto and BarProto to the given\n directory\n \"\"\"\n FooProto.write_proto_file(proto_dir)\n BarProto.write_proto_file(proto_dir)\n```\n\n### JTD To Proto\n\nThe following example illustrates how JTD schemas can be converted to proto:\n\n```py\nimport py_to_proto\n\n# Declare the Foo protobuf message class\nFoo = py_to_proto.descriptor_to_message_class(\n py_to_proto.jtd_to_proto(\n name=\"Foo\",\n package=\"foobar\",\n jtd_def={\n \"properties\": {\n # Bool field\n \"foo\": {\n \"type\": \"boolean\",\n },\n # Array of nested enum values\n \"bar\": {\n \"elements\": {\n \"enum\": [\"EXAM\", \"JOKE_SETTING\"],\n }\n }\n }\n },\n )\n)\n\n# Declare an object that references Foo as the type for a field\nBar = py_to_proto.descriptor_to_message_class(\n py_to_proto.jtd_to_proto(\n name=\"Bar\",\n package=\"foobar\",\n jtd_def={\n \"properties\": {\n \"baz\": {\n \"type\": Foo.DESCRIPTOR,\n },\n },\n },\n ),\n)\n\ndef write_protos(proto_dir: str):\n \"\"\"Write out the .proto files for Foo and Bar to the given directory\"\"\"\n Foo.write_proto_file(proto_dir)\n Bar.write_proto_file(proto_dir)\n```\n\n## Similar Projects\n\nThere are a number of similar projects in this space that offer slightly different value:\n\n- [`jtd-codegen`](https://jsontypedef.com/docs/jtd-codegen/): This project focuses on statically generating language-native code (including `python`) to represent the JTD schema.\n- [`py-json-to-proto`](https://pypi.org/project/py-json-to-proto/): This project aims to deduce a schema from an instance of a `json` object.\n- [`pure-protobuf`](https://pypi.org/project/pure-protobuf/): This project has a very similar aim to `py-to-proto`, but it skips the intermediate `descriptor` representation and thus is not able to produce native `message.Message` classes.\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "DEPRECATED: Please see py-to-proto: https://pypi.org/project/py-to-proto/",
"version": "0.13.0",
"split_keywords": [
"json",
"json typedef",
"jtd",
"protobuf",
"proto",
"dataclass"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ad2cb59eb40d43bf902084c9fcf1434b5951e5d86599438fd57e028ba05ccd92",
"md5": "642d0b69f87b8bbeec51eaf9417fc781",
"sha256": "42461ff004ee869e4261a3fc42a77b8510b73fd456b0eb069eacd36e057ae3d7"
},
"downloads": -1,
"filename": "jtd_to_proto-0.13.0-py39-none-any.whl",
"has_sig": false,
"md5_digest": "642d0b69f87b8bbeec51eaf9417fc781",
"packagetype": "bdist_wheel",
"python_version": "py39",
"requires_python": null,
"size": 31454,
"upload_time": "2023-04-26T22:36:13",
"upload_time_iso_8601": "2023-04-26T22:36:13.133755Z",
"url": "https://files.pythonhosted.org/packages/ad/2c/b59eb40d43bf902084c9fcf1434b5951e5d86599438fd57e028ba05ccd92/jtd_to_proto-0.13.0-py39-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-26 22:36:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "IBM",
"github_project": "py-to-proto",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "protobuf",
"specs": [
[
"<",
"5"
],
[
">=",
"3.19.0"
]
]
},
{
"name": "alchemy-logging",
"specs": [
[
">=",
"1.0.3"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"<",
"5"
],
[
">=",
"4.5.0"
]
]
}
],
"lcname": "jtd-to-proto"
}
JSON
