# `protogen`
[](https://badge.fury.io/py/protogen)
[](https://protogen.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/fischor/protogen-python/actions/workflows/test.yaml)
Package `protogen` makes writing `protoc` plugins easier.
Working with the raw protobuf descriptor messages can be cumbersome.
`protogen` resolves and links the dependencies and references between the raw Protobuf descriptors and turns them into their corresponding `protogen` classes that are easier to work with.
It also provides mechanisms that are espacially useful to generate Python code like dealing with Python imports.
## Installation
Package `protogen` is available via `pip`. To install run:
```
pip install protogen
```
## API
Most classes in `protogen` are simply replacements of their corresponding Protobuf descriptors: `protogen.File` represents a FileDescriptor, `protogen.Message` a Descriptor, `protogen.Field` a FieldDescriptor and so on. They should be self explanatory. You can [read the docstrings](https://pypi.org/project/protogen/) for more information about them.
The classes `protogen.Options`, `protogen.Plugin` and `protogen.GeneratedFile` make up a framework to generate files.
You can see these in action in the following example plugin:
```python
#!/usr/bin/env python
"""An example plugin."""
import protogen
def generate(gen: protogen.Plugin):
for f in gen.files_to_generate:
g = gen.new_generated_file(
f.proto.name.replace(".proto", ".py"),
f.py_import_path,
)
g.P("# Generated code ahead.")
g.P()
g.print_imports()
g.P()
for m in f.message:
g.P("class ", m.py_ident, ":")
for ff in m.fields:
# ...
for s in f.services:
g.P("class ", s.py_ident, ":")
for m in f.methods:
g.P(" def ", m.py_name, "(request):")
g.P(" pass")
if __name__ == "__main__":
opts = protogen.Options()
opts.run(generate)
```
## class `protogen.Options`
The `protogen.Options` class can be used to specify options for the resolution process (resolution from plain proto descriptors to `protogen` classes).
`Option.run(f: func(Plugin))` waits for `protoc` to write the CodeGeneratorRequest to `stdin`, resolves the descriptors contained in it to their corresponding `protogen` classes and initializes a new `Plugin` with the resolved classes.
`f` is then called with the `Plugin` as argument.
Once `f` returns, `Options` will collect the CodeGeneratorResponse from the `Plugin` that contains the all created `GeneratedFile`s and write it to `stdout` for `protoc` to pick it up.
`protoc` writes the generated files to disk.
## class `protogen.Plugin`
The `Plugin` class holds the files code generation is requested for in the `Plugin.files_to_generate` attribute. These are the files that were provided as command line arguments to `protoc`.
Any options/parameters passed to the plugin via the `protoc --plugin_opt=<param>` command line flag are accessible via `Plugin.parameter`.
With `Plugin.new_generated_file` a new `GeneratedFile` gets created that is automatically added to the CodeGeneratorResponse of the plugin.
Typically, but not necessarily, one file for each file in `Plugin.files_to_generate` is created.
## class `protogen.GeneratedFile`
The `GeneratedFile` is just a buffer you can add lines to using the `g.P` (print) method.
A `GeneratedFile` is created with `Plugin.new_generated_file(filename, py_import_path)`.
The `filename` is obviously the name of the file to be created.
The `py_import_path` is used for *import resolution*.
Note that the following assumes the plugin generates Python code. For other kinds of plugins, the following is not relevant:
It is often necessary to import Python identifiers that are defined in different Python modules.
For example, a Protobuf messages might reference `google.protobuf.Timestamp` in one of its fields.
The corresponding Python class `google.protobuf.timestamp_pb2.Timestamp` needs to be imported before its mentioned in the generated code.
The `protogen.PyImportPath` class represent a Python import path. Is just a wrapper around an import path (for example `"google.protobuf.timestamp_pb2"`).
The `PyIdent` class represent a Python identifier. It holds a `PyImportPath` together with a name (e.g. a class name like `"Timestamp"`).
The `protogen.GeneratedFile` provides mechanisms to handle Python imports.
Internally it maintains a list of `PyImportPath`s that it needs to import.
`PyImportPaths` might be added to this list implictly when calling `GeneratedFile.P(*args)` or rather explicitly when calling `GeneratedFile.qualified_py_ident(PyIdent)`.
When any of the arguments to `GeneratedFile.P` is a `protogen.PyIdent`, the `py_import_path` of the `GeneratedFile` gets compared to the arguments `PyIdent.py_import_path`.
If they are from different Python modules, the arguments import path will be added to the list of imports and the fully qualified name of the `PyIdent` will be printed.
If both files are from the same `PyImportPath`, then the import path is not added to the list of imports. In that case it is sufficient to reference the `PyIdent` by its simple name (e.g. `Timestamp`), thus only the `PyIdent.py_name` will be printed.
To place the import statements in the buffer of the `GeneratedFile` use `GeneratedFile.print_imports`. This will put a line `"import <path>"` for each `PyImportPath` that the generated file needs to import (e.g `"import google.protobuf.timestamp_pb2"`) in the buffer.
The following example shows how the `GeneratedFile.P` function behaves for different `PyImportPaths`::
```python
# g is of type protogen.GeneratedFile
# message_a and message_b are of type protogen.Message
>>> g.py_import_path
{ "mypackage.mymodule" }
>>> message_a.py_ident
{ py_import_path: "google.protobuf.timestamp_pb2", py_name: "Timestamp" }
>>> g.P("hello ", message_a.py_ident)
# adds "hello google.protobuf.timestamp_pb2.Timestamp" to g's line buffer and "google.protobuf.timestamp_pb2" to the imports
>>> message_b.py_ident
{ py_import_path: "mypackage.mymodule", py_name: "MyMessage" }
>>> g.P("hello ", message_b.py_ident)
# adds "hello MyMessage" to g's line buffer (and nothing to the imports)
```
Note that you can provide a custom `py_import_func` in the `Options` constructor.
This function is used in the resolution process to calculate the `PyImportPath` for `protogen.File`s.
`protogen.Message`s, `protogen.Service`s and `protogen.Enum`s inherit the `PyImportPath` (that is part of their `PyIdent`) from the file they are defined in.
By default the `protogen.default_py_import_func` is used.
It is compatible with the style of the offical Python `protoc` plugin that generates for each input file `path/to/file.proto` a corresponding `path/to/file_pb2.py` file.
For example, assume you know that code generation for proto definitions that are part of the `mypackage.**` proto package happens with a `protoc` plugin that generates one `.py` file per proto package.
That `plugin` also omits the `_pb2` suffix.
For the proto package `mypackage.api.a`, that might contain any number of files, it creates a `mypackage/api/a.py` file.
For the proto package `mypackage.api.b`, a `mypackage/api/b.py` file.
A `py_import_func` describing this would be:
```python
def py_import_func(
proto_filename: str,
proto_package:str,
) -> protogen.PyImportPath:
if proto_package.split(".")[0] == "mypackage":
# Python import path is simply the package name.
return protogen.PyImportPath(proto_package)
# For every other package, assume its generated with the offical Python plugin.
return protogen.default_py_import_func(proto_filename, proto_package)
```
# Misc
## What is a protoc plugin anyway?
`protoc`, the **Proto**buf **c**ompiler, is used to generate code derived from Protobuf definitions (`.proto` files).
Under the hood, `protoc`'s job is to read and parse the definitions into their *Descriptor* types (see [google/protobuf/descriptor.proto](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/descriptor.proto)).
When `protoc` is run (with a plugin) it creates a CodeGeneratorRequest (see [google/protobuf/compiler/plugin.proto#L68](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/compiler/plugin.proto#L68)) that contains the descriptors for the files to generate and everything they import and passes it to the plugin via `stdin`.
A *protoc plugin* is an executable. It reads the CodeGeneratorRequest from `stdin` and returns a CodeGeneratorResponse (see [google/protobuf/compiler/plugin.proto#L99](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/compiler/plugin.proto#L99)) via `stdout`.
The plugin can use the descriptors from the CodeGeneratorRequest to create output files (in memory).
It returns these output files (consisting of name and content as string) in the CodeGeneratorResponse to `protoc`.
`protoc` then writes these files to disk.
## Run `protoc` with your plugin
Assume you have an executable plugin under `path/to/plugin/main.py`.
You can invoke it via:
```
protoc
--plugin=protoc-gen-myplugin=path/to/plugin/main.py \
--myplugin_out=./output_root \
myproto.proto myproto2.proto
```
Caveats:
- you must use the `--plugin=protoc-gen-<plugin_name>` prefix, otherwise `protoc` fails with "plugin not executable"
- specify the output path of the plugin with `--<plugin_name>-out` flag where `<plugin_name>` is the same as used in the `--plugin` flag
- your plugin must be executable (`chmod +x path/to/plugin/main.py` and put a `#!/usr/bin/env python` at the top of the file)
# See also
- if you want to write protoc plugins with JavaScript/TypeScript: [github.com/fischor/protogen-javascript](https://github.com/fischor/protogen-javascript)
- if you want to write protoc plugins with Golang: [google.golang.org/protobuf/compiler/protogen](https://google.golang.org/protobuf/compiler/protogen)
# Credits
This package is inspired by the [google.golang.org/protobuf/compiler/protogen Golang](https://pkg.go.dev/google.golang.org/protobuf@v1.27.1/compiler/protogen) package.
Raw data
{
"_id": null,
"home_page": "https://github.com/fischor/protogen-python",
"name": "protogen",
"maintainer": "fischor",
"docs_url": null,
"requires_python": ">=3.7,<4.0",
"maintainer_email": "fischor.sh@gmail.com",
"keywords": "proto,protoc,Protobuf,Protocol buffers,Code generation",
"author": "fischor",
"author_email": "fischor.sh@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/24/fc/2e68784a06e46fe799dd375b732c13f99559a2c3b2164100607ec8b5cccf/protogen-0.3.1.tar.gz",
"platform": null,
"description": "# `protogen`\n\n[](https://badge.fury.io/py/protogen)\n[](https://protogen.readthedocs.io/en/latest/?badge=latest)\n[](https://github.com/fischor/protogen-python/actions/workflows/test.yaml)\n\nPackage `protogen` makes writing `protoc` plugins easier.\nWorking with the raw protobuf descriptor messages can be cumbersome.\n`protogen` resolves and links the dependencies and references between the raw Protobuf descriptors and turns them into their corresponding `protogen` classes that are easier to work with.\nIt also provides mechanisms that are espacially useful to generate Python code like dealing with Python imports.\n\n## Installation\n\nPackage `protogen` is available via `pip`. To install run:\n\n```\npip install protogen\n```\n\n## API\n\nMost classes in `protogen` are simply replacements of their corresponding Protobuf descriptors: `protogen.File` represents a FileDescriptor, `protogen.Message` a Descriptor, `protogen.Field` a FieldDescriptor and so on. They should be self explanatory. You can [read the docstrings](https://pypi.org/project/protogen/) for more information about them.\n\nThe classes `protogen.Options`, `protogen.Plugin` and `protogen.GeneratedFile` make up a framework to generate files.\nYou can see these in action in the following example plugin:\n\n```python\n#!/usr/bin/env python\n\"\"\"An example plugin.\"\"\"\n\nimport protogen\n\ndef generate(gen: protogen.Plugin):\n for f in gen.files_to_generate:\n g = gen.new_generated_file(\n f.proto.name.replace(\".proto\", \".py\"), \n f.py_import_path,\n )\n g.P(\"# Generated code ahead.\")\n g.P()\n g.print_imports()\n g.P()\n for m in f.message:\n g.P(\"class \", m.py_ident, \":\")\n for ff in m.fields:\n # ...\n for s in f.services:\n g.P(\"class \", s.py_ident, \":\")\n for m in f.methods:\n g.P(\" def \", m.py_name, \"(request):\")\n g.P(\" pass\")\n\nif __name__ == \"__main__\":\n opts = protogen.Options()\n opts.run(generate)\n```\n\n## class `protogen.Options`\n\nThe `protogen.Options` class can be used to specify options for the resolution process (resolution from plain proto descriptors to `protogen` classes).\n`Option.run(f: func(Plugin))` waits for `protoc` to write the CodeGeneratorRequest to `stdin`, resolves the descriptors contained in it to their corresponding `protogen` classes and initializes a new `Plugin` with the resolved classes. \n`f` is then called with the `Plugin` as argument.\n\nOnce `f` returns, `Options` will collect the CodeGeneratorResponse from the `Plugin` that contains the all created `GeneratedFile`s and write it to `stdout` for `protoc` to pick it up.\n`protoc` writes the generated files to disk.\n\n## class `protogen.Plugin`\n\nThe `Plugin` class holds the files code generation is requested for in the `Plugin.files_to_generate` attribute. These are the files that were provided as command line arguments to `protoc`.\nAny options/parameters passed to the plugin via the `protoc --plugin_opt=<param>` command line flag are accessible via `Plugin.parameter`.\nWith `Plugin.new_generated_file` a new `GeneratedFile` gets created that is automatically added to the CodeGeneratorResponse of the plugin.\nTypically, but not necessarily, one file for each file in `Plugin.files_to_generate` is created.\n\n## class `protogen.GeneratedFile`\n\nThe `GeneratedFile` is just a buffer you can add lines to using the `g.P` (print) method.\nA `GeneratedFile` is created with `Plugin.new_generated_file(filename, py_import_path)`.\nThe `filename` is obviously the name of the file to be created.\nThe `py_import_path` is used for *import resolution*.\n\nNote that the following assumes the plugin generates Python code. For other kinds of plugins, the following is not relevant:\n\nIt is often necessary to import Python identifiers that are defined in different Python modules.\nFor example, a Protobuf messages might reference `google.protobuf.Timestamp` in one of its fields.\nThe corresponding Python class `google.protobuf.timestamp_pb2.Timestamp` needs to be imported before its mentioned in the generated code.\n\nThe `protogen.PyImportPath` class represent a Python import path. Is just a wrapper around an import path (for example `\"google.protobuf.timestamp_pb2\"`).\nThe `PyIdent` class represent a Python identifier. It holds a `PyImportPath` together with a name (e.g. a class name like `\"Timestamp\"`).\n\nThe `protogen.GeneratedFile` provides mechanisms to handle Python imports.\nInternally it maintains a list of `PyImportPath`s that it needs to import.\n`PyImportPaths` might be added to this list implictly when calling `GeneratedFile.P(*args)` or rather explicitly when calling `GeneratedFile.qualified_py_ident(PyIdent)`.\nWhen any of the arguments to `GeneratedFile.P` is a `protogen.PyIdent`, the `py_import_path` of the `GeneratedFile` gets compared to the arguments `PyIdent.py_import_path`. \n\nIf they are from different Python modules, the arguments import path will be added to the list of imports and the fully qualified name of the `PyIdent` will be printed. \n\nIf both files are from the same `PyImportPath`, then the import path is not added to the list of imports. In that case it is sufficient to reference the `PyIdent` by its simple name (e.g. `Timestamp`), thus only the `PyIdent.py_name` will be printed.\n\nTo place the import statements in the buffer of the `GeneratedFile` use `GeneratedFile.print_imports`. This will put a line `\"import <path>\"` for each `PyImportPath` that the generated file needs to import (e.g `\"import google.protobuf.timestamp_pb2\"`) in the buffer.\n\nThe following example shows how the `GeneratedFile.P` function behaves for different `PyImportPaths`::\n\n```python\n# g is of type protogen.GeneratedFile\n# message_a and message_b are of type protogen.Message\n\n>>> g.py_import_path\n{ \"mypackage.mymodule\" }\n\n>>> message_a.py_ident\n{ py_import_path: \"google.protobuf.timestamp_pb2\", py_name: \"Timestamp\" }\n>>> g.P(\"hello \", message_a.py_ident) \n# adds \"hello google.protobuf.timestamp_pb2.Timestamp\" to g's line buffer and \"google.protobuf.timestamp_pb2\" to the imports\n\n>>> message_b.py_ident\n{ py_import_path: \"mypackage.mymodule\", py_name: \"MyMessage\" }\n>>> g.P(\"hello \", message_b.py_ident) \n# adds \"hello MyMessage\" to g's line buffer (and nothing to the imports)\n```\n\nNote that you can provide a custom `py_import_func` in the `Options` constructor.\nThis function is used in the resolution process to calculate the `PyImportPath` for `protogen.File`s.\n`protogen.Message`s, `protogen.Service`s and `protogen.Enum`s inherit the `PyImportPath` (that is part of their `PyIdent`) from the file they are defined in.\nBy default the `protogen.default_py_import_func` is used. \nIt is compatible with the style of the offical Python `protoc` plugin that generates for each input file `path/to/file.proto` a corresponding `path/to/file_pb2.py` file.\n\nFor example, assume you know that code generation for proto definitions that are part of the `mypackage.**` proto package happens with a `protoc` plugin that generates one `.py` file per proto package. \nThat `plugin` also omits the `_pb2` suffix.\nFor the proto package `mypackage.api.a`, that might contain any number of files, it creates a `mypackage/api/a.py` file.\nFor the proto package `mypackage.api.b`, a `mypackage/api/b.py` file.\n\nA `py_import_func` describing this would be:\n\n```python\ndef py_import_func(\n proto_filename: str, \n proto_package:str,\n) -> protogen.PyImportPath:\n if proto_package.split(\".\")[0] == \"mypackage\":\n # Python import path is simply the package name.\n return protogen.PyImportPath(proto_package) \n # For every other package, assume its generated with the offical Python plugin.\n return protogen.default_py_import_func(proto_filename, proto_package)\n```\n\n# Misc\n\n## What is a protoc plugin anyway?\n\n`protoc`, the **Proto**buf **c**ompiler, is used to generate code derived from Protobuf definitions (`.proto` files).\nUnder the hood, `protoc`'s job is to read and parse the definitions into their *Descriptor* types (see [google/protobuf/descriptor.proto](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/descriptor.proto)).\nWhen `protoc` is run (with a plugin) it creates a CodeGeneratorRequest (see [google/protobuf/compiler/plugin.proto#L68](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/compiler/plugin.proto#L68)) that contains the descriptors for the files to generate and everything they import and passes it to the plugin via `stdin`.\n\nA *protoc plugin* is an executable. It reads the CodeGeneratorRequest from `stdin` and returns a CodeGeneratorResponse (see [google/protobuf/compiler/plugin.proto#L99](https://github.com/protocolbuffers/protobuf/blob/4f49062a95f18a6c7e21ba17715a2b0a4608151a/src/google/protobuf/compiler/plugin.proto#L99)) via `stdout`.\nThe plugin can use the descriptors from the CodeGeneratorRequest to create output files (in memory).\nIt returns these output files (consisting of name and content as string) in the CodeGeneratorResponse to `protoc`.\n\n`protoc` then writes these files to disk.\n\n## Run `protoc` with your plugin\n\nAssume you have an executable plugin under `path/to/plugin/main.py`.\nYou can invoke it via:\n\n```\nprotoc \n --plugin=protoc-gen-myplugin=path/to/plugin/main.py \\\n --myplugin_out=./output_root \\\n myproto.proto myproto2.proto\n```\n\nCaveats:\n- you must use the `--plugin=protoc-gen-<plugin_name>` prefix, otherwise `protoc` fails with \"plugin not executable\"\n- specify the output path of the plugin with `--<plugin_name>-out` flag where `<plugin_name>` is the same as used in the `--plugin` flag \n- your plugin must be executable (`chmod +x path/to/plugin/main.py` and put a `#!/usr/bin/env python` at the top of the file)\n\n# See also\n\n- if you want to write protoc plugins with JavaScript/TypeScript: [github.com/fischor/protogen-javascript](https://github.com/fischor/protogen-javascript)\n- if you want to write protoc plugins with Golang: [google.golang.org/protobuf/compiler/protogen](https://google.golang.org/protobuf/compiler/protogen)\n\n# Credits\n\nThis package is inspired by the [google.golang.org/protobuf/compiler/protogen Golang](https://pkg.go.dev/google.golang.org/protobuf@v1.27.1/compiler/protogen) package.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "protogen makes writing protoc plugins easy.",
"version": "0.3.1",
"project_urls": {
"Homepage": "https://github.com/fischor/protogen-python",
"Repository": "https://github.com/fischor/protogen-python"
},
"split_keywords": [
"proto",
"protoc",
"protobuf",
"protocol buffers",
"code generation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8da0c3f3a2e2fa866547d82190ec5c0cd55580bc29c7894221bd793003a578a1",
"md5": "f0af0100a4860a7fd73b6e451d17eccd",
"sha256": "65b60b284d20ee4899d515b1959882d8c7504b271552de36f4ebfe77f6b07331"
},
"downloads": -1,
"filename": "protogen-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f0af0100a4860a7fd73b6e451d17eccd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7,<4.0",
"size": 21425,
"upload_time": "2023-11-20T15:34:45",
"upload_time_iso_8601": "2023-11-20T15:34:45.958772Z",
"url": "https://files.pythonhosted.org/packages/8d/a0/c3f3a2e2fa866547d82190ec5c0cd55580bc29c7894221bd793003a578a1/protogen-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "24fc2e68784a06e46fe799dd375b732c13f99559a2c3b2164100607ec8b5cccf",
"md5": "e5e808402445bc37c8d789e348734f0f",
"sha256": "1e55405f6c94476c45c400b069dbdb0274f065e3109fee28122e96dbba075dcd"
},
"downloads": -1,
"filename": "protogen-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "e5e808402445bc37c8d789e348734f0f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7,<4.0",
"size": 23018,
"upload_time": "2023-11-20T15:34:48",
"upload_time_iso_8601": "2023-11-20T15:34:48.288479Z",
"url": "https://files.pythonhosted.org/packages/24/fc/2e68784a06e46fe799dd375b732c13f99559a2c3b2164100607ec8b5cccf/protogen-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-20 15:34:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fischor",
"github_project": "protogen-python",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "protogen"
}
JSON
