|nala|_
About
=====
`Google Protocol Buffers`_ tools in Python 3.6+.
- `C` source code generator.
- `proto3`_ language parser.
Known limitations:
- Options, services (gRPC) and reserved fields are ignored.
- Public imports are not implemented.
Project homepage: https://github.com/eerimoq/pbtools
Documentation: https://pbtools.readthedocs.io
Installation
============
.. code-block:: python
pip install pbtools
C source code design
====================
The C source code is designed with the following in mind:
- Clean and easy to use API.
- No malloc/free. Uses a workspace/arena for memory allocations.
- Fast encoding and decoding.
- Small memory footprint.
- Thread safety.
Known limitations:
- ``char`` must be 8 bits.
ToDo:
- Make ``map`` easier to use. Only one allocation should be needed
before encoding, not one per sub-message item.
Memory management
-----------------
A workspace, or arena, is used to allocate memory when encoding and
decoding messages. For simplicity, allocated memory can't be freed,
which puts restrictions on how a message can be modified between
encodings (if one want to do that). Scalar value type fields (ints,
strings, bytes, etc.) can be modified, but the length of repeated
fields can't.
Scalar Value Types
------------------
Protobuf scalar value types are mapped to C types as shown in the
table below.
+---------------+--------------------------------------------+
| Protubuf Type | C Type |
+===============+============================================+
| ``double`` | ``double`` |
+---------------+--------------------------------------------+
| ``float`` | ``float`` |
+---------------+--------------------------------------------+
| ``int32`` | ``int32_t`` |
+---------------+--------------------------------------------+
| ``int64`` | ``int64_t`` |
+---------------+--------------------------------------------+
| ``uint32`` | ``uint32_t`` |
+---------------+--------------------------------------------+
| ``uint64`` | ``uint64_t`` |
+---------------+--------------------------------------------+
| ``sint32`` | ``int32_t`` |
+---------------+--------------------------------------------+
| ``sint64`` | ``int64_t`` |
+---------------+--------------------------------------------+
| ``fixed32`` | ``int32_t`` |
+---------------+--------------------------------------------+
| ``fixed64`` | ``int64_t`` |
+---------------+--------------------------------------------+
| ``sfixed32`` | ``int32_t`` |
+---------------+--------------------------------------------+
| ``sfixed64`` | ``int64_t`` |
+---------------+--------------------------------------------+
| ``bool`` | ``bool`` |
+---------------+--------------------------------------------+
| ``string`` | ``char *`` |
+---------------+--------------------------------------------+
| ``bytes`` | ``struct { uint8_t *buf_p, size_t size }`` |
+---------------+--------------------------------------------+
Message
-------
A message is a struct in C.
For example, let's create a protocol specification.
.. code-block:: proto
syntax = "proto3";
package foo;
message Bar {
bool v1 = 1;
}
message Fie {
int32 v2 = 1;
Bar v3 = 2;
}
One struct is generated per message.
.. code-block:: c
struct foo_bar_t {
bool v1;
};
struct foo_fie_t {
int32_t v2;
struct foo_bar_t *v3_p;
};
The sub-message ``v3`` has to be allocated before encoding and checked
if ``NULL`` after decoding.
.. code-block:: c
struct foo_fie_t *fie_p;
/* Encode. */
fie_p = foo_fie_new(...);
fie_p->v2 = 5;
foo_fie_v3_alloc(fie_p);
fie_p->v3_p->v1 = true;
foo_fie_encode(fie_p, ...);
/* Decode. */
fie_p = foo_fie_new(...);
foo_fie_decode(fie_p, ...);
printf("%d\n", fie_p->v2);
if (fie_p->v3_p != NULL) {
printf("%d\n", fie_p->v3_p->v1);
}
Oneof
-----
A oneof is an enum (the choice) and a union in C.
For example, let's create a protocol specification.
.. code-block:: proto
syntax = "proto3";
package foo;
message Bar {
oneof fie {
int32 v1 = 1;
bool v2 = 2;
};
}
One enum and one struct is generated per oneof.
.. code-block:: c
enum foo_bar_fie_e {
foo_bar_fie_none_e = 0,
foo_bar_fie_v1_e = 1,
foo_bar_fie_v2_e = 2
};
struct foo_bar_t {
enum foo_bar_fie_choice_e fie;
union {
int32_t v1;
bool v2;
};
};
The generated code can encode and decode messages. Call
``_<field>_init()`` or ``_<field>_alloc()`` to select which oneof
field to encode. Use the ``enum`` to check which oneof field was
decoded (if any).
.. code-block:: c
struct foo_bar_t *bar_p;
/* Encode with choice v1. */
bar_p = foo_bar_new(...);
foo_bar_v1_init(bar_p);
bar_p->v1 = -2;
foo_bar_encode(bar_p, ...);
/* Decode. */
bar_p = foo_bar_new(...);
foo_bar_decode(bar_p, ...);
switch (bar_p->fie) {
case foo_bar_fie_none_e:
printf("Not present.\n");
break;
case foo_bar_fie_v1_e:
printf("%d\n", bar_p->v1);
break;
case foo_bar_fie_v2_e:
printf("%d\n", bar_p->v2);
break;
default:
printf("Can not happen.\n");
break;
}
Benchmark
---------
See `benchmark`_ for a benchmark of a few C/C++ protobuf libraries.
Example usage
=============
C source code
-------------
In this example we use the simple proto-file `hello_world.proto`_.
.. code-block:: proto
syntax = "proto3";
package hello_world;
message Foo {
int32 bar = 1;
}
Generate C source code from the proto-file.
.. code-block:: text
$ pbtools generate_c_source examples/hello_world/hello_world.proto
See `hello_world.h`_ and `hello_world.c`_ for the contents of the
generated files.
We'll use the generated types and functions below.
.. code-block:: c
struct hello_world_foo_t {
struct pbtools_message_base_t base;
int32_t bar;
};
struct hello_world_foo_t *hello_world_foo_new(
void *workspace_p,
size_t size);
int hello_world_foo_encode(
struct hello_world_foo_t *self_p,
void *encoded_p,
size_t size);
int hello_world_foo_decode(
struct hello_world_foo_t *self_p,
const uint8_t *encoded_p,
size_t size);
Encode and decode the Foo-message in `main.c`_.
.. code-block:: c
#include <stdio.h>
#include "hello_world.h"
int main(int argc, const char *argv[])
{
int size;
uint8_t workspace[64];
uint8_t encoded[16];
struct hello_world_foo_t *foo_p;
/* Encode. */
foo_p = hello_world_foo_new(&workspace[0], sizeof(workspace));
if (foo_p == NULL) {
return (1);
}
foo_p->bar = 78;
size = hello_world_foo_encode(foo_p, &encoded[0], sizeof(encoded));
if (size < 0) {
return (2);
}
printf("Successfully encoded Foo into %d bytes.\n", size);
/* Decode. */
foo_p = hello_world_foo_new(&workspace[0], sizeof(workspace));
if (foo_p == NULL) {
return (3);
}
size = hello_world_foo_decode(foo_p, &encoded[0], size);
if (size < 0) {
return (4);
}
printf("Successfully decoded %d bytes into Foo.\n", size);
printf("Foo.bar: %d\n", foo_p->bar);
return (0);
}
Build and run the program.
.. code-block:: text
$ gcc -I lib/include main.c hello_world.c lib/src/pbtools.c -o main
$ ./main
Successfully encoded Foo into 2 bytes.
Successfully decoded 2 bytes into Foo.
Foo.bar: 78
See `examples/hello_world`_ for all files used in this example.
Command line tool
-----------------
The generate C source subcommand
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Below is an example of how to generate C source code from a
proto-file.
.. code-block:: text
$ pbtools generate_c_source examples/address_book/address_book.proto
See `address_book.h`_ and `address_book.c`_ for the contents of the
generated files.
.. |nala| image:: https://img.shields.io/badge/nala-test-blue.svg
.. _nala: https://github.com/eerimoq/nala
.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers
.. _proto3: https://developers.google.com/protocol-buffers/docs/proto3
.. _address_book.h: https://github.com/eerimoq/pbtools/blob/master/examples/address_book/generated/address_book.h
.. _address_book.c: https://github.com/eerimoq/pbtools/blob/master/examples/address_book/generated/address_book.c
.. _hello_world.proto: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/hello_world.proto
.. _hello_world.h: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/generated/hello_world.h
.. _hello_world.c: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/generated/hello_world.c
.. _main.c: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/main.c
.. _examples/hello_world: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world
.. _benchmark: https://github.com/eerimoq/pbtools/blob/master/benchmark
Raw data
{
"_id": null,
"home_page": "https://github.com/eerimoq/pbtools",
"name": "pbtools",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "protobuf,proto,protocol buffers",
"author": "Erik Moqvist",
"author_email": "erik.moqvist@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e5/9f/ace031604d09571e5c8a2e3521c83f50a074b4e0c2abe99323f8855adb7c/pbtools-0.47.0.tar.gz",
"platform": null,
"description": "|nala|_\n\nAbout\n=====\n\n`Google Protocol Buffers`_ tools in Python 3.6+.\n\n- `C` source code generator.\n\n- `proto3`_ language parser.\n\nKnown limitations:\n\n- Options, services (gRPC) and reserved fields are ignored.\n\n- Public imports are not implemented.\n\nProject homepage: https://github.com/eerimoq/pbtools\n\nDocumentation: https://pbtools.readthedocs.io\n\nInstallation\n============\n\n.. code-block:: python\n\n pip install pbtools\n\nC source code design\n====================\n\nThe C source code is designed with the following in mind:\n\n- Clean and easy to use API.\n\n- No malloc/free. Uses a workspace/arena for memory allocations.\n\n- Fast encoding and decoding.\n\n- Small memory footprint.\n\n- Thread safety.\n\nKnown limitations:\n\n- ``char`` must be 8 bits.\n\nToDo:\n\n- Make ``map`` easier to use. Only one allocation should be needed\n before encoding, not one per sub-message item.\n\nMemory management\n-----------------\n\nA workspace, or arena, is used to allocate memory when encoding and\ndecoding messages. For simplicity, allocated memory can't be freed,\nwhich puts restrictions on how a message can be modified between\nencodings (if one want to do that). Scalar value type fields (ints,\nstrings, bytes, etc.) can be modified, but the length of repeated\nfields can't.\n\nScalar Value Types\n------------------\n\nProtobuf scalar value types are mapped to C types as shown in the\ntable below.\n\n+---------------+--------------------------------------------+\n| Protubuf Type | C Type |\n+===============+============================================+\n| ``double`` | ``double`` |\n+---------------+--------------------------------------------+\n| ``float`` | ``float`` |\n+---------------+--------------------------------------------+\n| ``int32`` | ``int32_t`` |\n+---------------+--------------------------------------------+\n| ``int64`` | ``int64_t`` |\n+---------------+--------------------------------------------+\n| ``uint32`` | ``uint32_t`` |\n+---------------+--------------------------------------------+\n| ``uint64`` | ``uint64_t`` |\n+---------------+--------------------------------------------+\n| ``sint32`` | ``int32_t`` |\n+---------------+--------------------------------------------+\n| ``sint64`` | ``int64_t`` |\n+---------------+--------------------------------------------+\n| ``fixed32`` | ``int32_t`` |\n+---------------+--------------------------------------------+\n| ``fixed64`` | ``int64_t`` |\n+---------------+--------------------------------------------+\n| ``sfixed32`` | ``int32_t`` |\n+---------------+--------------------------------------------+\n| ``sfixed64`` | ``int64_t`` |\n+---------------+--------------------------------------------+\n| ``bool`` | ``bool`` |\n+---------------+--------------------------------------------+\n| ``string`` | ``char *`` |\n+---------------+--------------------------------------------+\n| ``bytes`` | ``struct { uint8_t *buf_p, size_t size }`` |\n+---------------+--------------------------------------------+\n\nMessage\n-------\n\nA message is a struct in C.\n\nFor example, let's create a protocol specification.\n\n.. code-block:: proto\n\n syntax = \"proto3\";\n\n package foo;\n\n message Bar {\n bool v1 = 1;\n }\n\n message Fie {\n int32 v2 = 1;\n Bar v3 = 2;\n }\n\nOne struct is generated per message.\n\n.. code-block:: c\n\n struct foo_bar_t {\n bool v1;\n };\n\n struct foo_fie_t {\n int32_t v2;\n struct foo_bar_t *v3_p;\n };\n\nThe sub-message ``v3`` has to be allocated before encoding and checked\nif ``NULL`` after decoding.\n\n.. code-block:: c\n\n struct foo_fie_t *fie_p;\n\n /* Encode. */\n fie_p = foo_fie_new(...);\n fie_p->v2 = 5;\n foo_fie_v3_alloc(fie_p);\n fie_p->v3_p->v1 = true;\n foo_fie_encode(fie_p, ...);\n\n /* Decode. */\n fie_p = foo_fie_new(...);\n foo_fie_decode(fie_p, ...);\n\n printf(\"%d\\n\", fie_p->v2);\n\n if (fie_p->v3_p != NULL) {\n printf(\"%d\\n\", fie_p->v3_p->v1);\n }\n\nOneof\n-----\n\nA oneof is an enum (the choice) and a union in C.\n\nFor example, let's create a protocol specification.\n\n.. code-block:: proto\n\n syntax = \"proto3\";\n\n package foo;\n\n message Bar {\n oneof fie {\n int32 v1 = 1;\n bool v2 = 2;\n };\n }\n\nOne enum and one struct is generated per oneof.\n\n.. code-block:: c\n\n enum foo_bar_fie_e {\n foo_bar_fie_none_e = 0,\n foo_bar_fie_v1_e = 1,\n foo_bar_fie_v2_e = 2\n };\n\n struct foo_bar_t {\n enum foo_bar_fie_choice_e fie;\n union {\n int32_t v1;\n bool v2;\n };\n };\n\nThe generated code can encode and decode messages. Call\n``_<field>_init()`` or ``_<field>_alloc()`` to select which oneof\nfield to encode. Use the ``enum`` to check which oneof field was\ndecoded (if any).\n\n.. code-block:: c\n\n struct foo_bar_t *bar_p;\n\n /* Encode with choice v1. */\n bar_p = foo_bar_new(...);\n foo_bar_v1_init(bar_p);\n bar_p->v1 = -2;\n foo_bar_encode(bar_p, ...);\n\n /* Decode. */\n bar_p = foo_bar_new(...);\n foo_bar_decode(bar_p, ...);\n\n switch (bar_p->fie) {\n\n case foo_bar_fie_none_e:\n printf(\"Not present.\\n\");\n break;\n\n case foo_bar_fie_v1_e:\n printf(\"%d\\n\", bar_p->v1);\n break;\n\n case foo_bar_fie_v2_e:\n printf(\"%d\\n\", bar_p->v2);\n break;\n\n default:\n printf(\"Can not happen.\\n\");\n break;\n }\n\nBenchmark\n---------\n\nSee `benchmark`_ for a benchmark of a few C/C++ protobuf libraries.\n\nExample usage\n=============\n\nC source code\n-------------\n\nIn this example we use the simple proto-file `hello_world.proto`_.\n\n.. code-block:: proto\n\n syntax = \"proto3\";\n\n package hello_world;\n\n message Foo {\n int32 bar = 1;\n }\n\nGenerate C source code from the proto-file.\n\n.. code-block:: text\n\n $ pbtools generate_c_source examples/hello_world/hello_world.proto\n\nSee `hello_world.h`_ and `hello_world.c`_ for the contents of the\ngenerated files.\n\nWe'll use the generated types and functions below.\n\n.. code-block:: c\n\n struct hello_world_foo_t {\n struct pbtools_message_base_t base;\n int32_t bar;\n };\n\n struct hello_world_foo_t *hello_world_foo_new(\n void *workspace_p,\n size_t size);\n\n int hello_world_foo_encode(\n struct hello_world_foo_t *self_p,\n void *encoded_p,\n size_t size);\n\n int hello_world_foo_decode(\n struct hello_world_foo_t *self_p,\n const uint8_t *encoded_p,\n size_t size);\n\nEncode and decode the Foo-message in `main.c`_.\n\n.. code-block:: c\n\n #include <stdio.h>\n #include \"hello_world.h\"\n\n int main(int argc, const char *argv[])\n {\n int size;\n uint8_t workspace[64];\n uint8_t encoded[16];\n struct hello_world_foo_t *foo_p;\n\n /* Encode. */\n foo_p = hello_world_foo_new(&workspace[0], sizeof(workspace));\n\n if (foo_p == NULL) {\n return (1);\n }\n\n foo_p->bar = 78;\n size = hello_world_foo_encode(foo_p, &encoded[0], sizeof(encoded));\n\n if (size < 0) {\n return (2);\n }\n\n printf(\"Successfully encoded Foo into %d bytes.\\n\", size);\n\n /* Decode. */\n foo_p = hello_world_foo_new(&workspace[0], sizeof(workspace));\n\n if (foo_p == NULL) {\n return (3);\n }\n\n size = hello_world_foo_decode(foo_p, &encoded[0], size);\n\n if (size < 0) {\n return (4);\n }\n\n printf(\"Successfully decoded %d bytes into Foo.\\n\", size);\n printf(\"Foo.bar: %d\\n\", foo_p->bar);\n\n return (0);\n }\n\nBuild and run the program.\n\n.. code-block:: text\n\n $ gcc -I lib/include main.c hello_world.c lib/src/pbtools.c -o main\n $ ./main\n Successfully encoded Foo into 2 bytes.\n Successfully decoded 2 bytes into Foo.\n Foo.bar: 78\n\nSee `examples/hello_world`_ for all files used in this example.\n\nCommand line tool\n-----------------\n\nThe generate C source subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBelow is an example of how to generate C source code from a\nproto-file.\n\n.. code-block:: text\n\n $ pbtools generate_c_source examples/address_book/address_book.proto\n\nSee `address_book.h`_ and `address_book.c`_ for the contents of the\ngenerated files.\n\n.. |nala| image:: https://img.shields.io/badge/nala-test-blue.svg\n.. _nala: https://github.com/eerimoq/nala\n\n.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers\n\n.. _proto3: https://developers.google.com/protocol-buffers/docs/proto3\n\n.. _address_book.h: https://github.com/eerimoq/pbtools/blob/master/examples/address_book/generated/address_book.h\n\n.. _address_book.c: https://github.com/eerimoq/pbtools/blob/master/examples/address_book/generated/address_book.c\n\n.. _hello_world.proto: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/hello_world.proto\n\n.. _hello_world.h: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/generated/hello_world.h\n\n.. _hello_world.c: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/generated/hello_world.c\n\n.. _main.c: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world/main.c\n\n.. _examples/hello_world: https://github.com/eerimoq/pbtools/blob/master/examples/hello_world\n\n.. _benchmark: https://github.com/eerimoq/pbtools/blob/master/benchmark\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Google Protocol Buffers tools.",
"version": "0.47.0",
"project_urls": {
"Homepage": "https://github.com/eerimoq/pbtools"
},
"split_keywords": [
"protobuf",
"proto",
"protocol buffers"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e59face031604d09571e5c8a2e3521c83f50a074b4e0c2abe99323f8855adb7c",
"md5": "b2089992ea00645be0a47e837b3c02ff",
"sha256": "55646c433c67119c386e14348a834087e31893cd90cb3db59ab845e6cbd571a1"
},
"downloads": -1,
"filename": "pbtools-0.47.0.tar.gz",
"has_sig": false,
"md5_digest": "b2089992ea00645be0a47e837b3c02ff",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 126816,
"upload_time": "2023-06-04T05:47:46",
"upload_time_iso_8601": "2023-06-04T05:47:46.167925Z",
"url": "https://files.pythonhosted.org/packages/e5/9f/ace031604d09571e5c8a2e3521c83f50a074b4e0c2abe99323f8855adb7c/pbtools-0.47.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-04 05:47:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eerimoq",
"github_project": "pbtools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "pbtools"
}
JSON
