# BlackBox Protobuf Library
## Description
Blackbox protobuf library is a Python module for decoding and re-encoding protobuf
messages without access to the source protobuf descriptor file. This library
provides a simple Python interface to encode/decode messages that can be
integrated into other tools.
This library is targeted towards use in penetration testing where being able to
modify messages is critical and a protocol buffer definition may not be readily
available.
## Background
Protocol Buffers (protobufs) are a standard published by Google with
accompanying libraries for binary serialization of data. Protocol buffers are
defined by a `.proto` file known to both the sender and the receiver. The actual
binary message does not contain information such as field names or most type
information.
For each field, the serialized protocol buffer includes two pieces of metadata,
a field number and the wire type. The wire type tells a parser how to parse the
length of the field, so that it can be skipped if it is not known (one protocol
buffer design goal is being able to handle messages with unknown fields). A
single wire-type generally encompasses multiple protocol buffer types, for
example the length delimited wire-type can be used for string, bytestring,
inner message or packed repeated fields. See
<https://developers.google.com/protocol-buffers/docs/encoding#structure> for
the breakdown of wire types.
The protocol buffer compiler (`protoc`) does support a similar method of
decoding protocol buffers without the definition with the `--decode_raw`
option. However, it does not provide any functionality to re-encode the decoded
message.
## How it works
The library makes a best effort guess of the type based on the provided wire type (and
occasionally field content) and builds a type definition that can be used to
re-encode the data. In general, most fields of interest are likely to be parsed
into a usable form. Users can optionally pass in custom type definitions that
override the guessed type. Custom type definitions also allow naming of fields to
improve user friendliness.
# Usage
## Installation
This library depends on internal functions of Google's protobuf Python library
to do some encoding/decoding of individual fields and six for python 2
compatibility.
The package can be installed from source with:
```
poetry install
```
BlackBox Protobuf is also available on PyPi at <https://pypi.org/project/bbpb>.
It can be installed with:
```
pip install bbpb
```
## Interface
The main `blackboxprotobuf` module defines an API with the core encode/decode
message functions, along with several convenience functions to make it easier
to use blackboxprotobuf with a user interface, such as encoding/decoding
directly to JSON and validating modified type definitions.
### Decode
Decoding functions takes a protobuf bytestring, and optionally
either a type definition or a known message name mapped to a type definition
(in `blackboxprotobuf.known_messages`). If a type definition isn't provided, an
empty message type is assumed and all types are derived from the protobuf
binary.
The decoder returns a tuple containing a dictionary with the decoded data and a
dictionary containing the generated type definition. If the input type
definition does not include types for all fields in the message, the output
type definitions will include type guesses for those fields.
Example use:
```python
import blackboxprotobuf
import base64
data = base64.b64decode('KglNb2RpZnkgTWU=')
message,typedef = blackboxprotobuf.protobuf_to_json(data)
print(message)
```
### Encode
The encoding functions takes a Python dictionary containing the data and a type
definition. Unlike decoding, the type definition is required and will fail if
any fields are not defined. Generally, the type definition should be the output
from the decoding function or a modified version thereof.
Example use:
```python
import blackboxprotobuf
import base64
data = base64.b64decode('KglNb2RpZnkgTWU=')
message,typedef = blackboxprotobuf.decode_message(data)
message[5] = 'Modified Me'
data = blackboxprotobuf.encode_message(message,typedef)
print(data)
```
### Type definition structure
The type definition object is a Python dictionary representing the type
structure of a message, it includes a type for each field and optionally a
name. Each entry in the dictionary represents a field in the message. The key
should be the field number and the value is a dictionary containing attributes.
At the minimum the dictionary should contain the 'type' entry which contains a
string identifier for the type. Valid type identifiers can be found in
`blackboxprotobuf/lib/types/type_maps.py`.
Message fields will also contain one of two entries, 'message_typedef' or
'message_type_name'. 'message_typedef' should contain a second type definition
structure for the inner message. 'message_type_name' should contain the string
identifier for a message type previously stored in
`blackboxprotobuf.known_messages`. If both are specified, the 'message_type_name'
will be ignored.
### JSON Encode/Decode
The `protobuf_to_json` and `protobuf_from_json` functions are convenience
functions for encoding/decoding messages to JSON instead of a python
dictionary. These functions are designed for user-facing input/output and will
also automatically sort the output, try to encode bytestrings for better
printing and annotate example values onto the type definition structure.
### Export/import protofile
The `export_protofile` and `import_protofile` will attempt to convert a
protobuffer `.proto` file into the blackboxprotobuf type definition and vice
versa. These functions provide a higher level interface to
`blackboxprotobuf.lib.protofile` which only takes a filename. The protofile
functions do not implement a full proper parser and may break on some types.
One common case to be aware of is the "import" statements in ".proto" files,
which are not supported. Any imported files must be manually imported with
`import_protofile` and saved in `blackboxprotobuf.known_messages` first.
### Validate Typedef
The `validate_typedef` function is designed to sanity check modified type
definitions and make sure they are internally consistent and consistent with
the previous type definition (if provided). This should help catch issues such
as changing a field to an incompatible type or duplicate field names.
### Output Helper Functions
The `json_safe_transform` is a helper function to help create more readable
JSON output of bytes. It will encode/decode bytes types as `latin1` based on
the type in the type definition.
The `sort_output` is a helper function which sorts the output message based on
the field numbers from the typedef. This helps makes the JSON output more
consistent and predictable.
The `sort_typedef` function sorts the fields of the typedef in order to make
the output more readable. The message fields are sorted by their number and
type fields (eg. name, type, inner message typedef) are sorted to prioritize
important short fields at the top and especially to keep the name and type
fields from getting buried underneath a long inner typedef.
### Config
Many of the functions accept a `config` keyword argument of the
`blackboxprotobuf.lib.config.Config` class. The config object allows modifying
some of the encoding/decoding functionality and storing some state. This
replaces some variables that were global before.
At the moment this includes:
* `known_types` - Mapping of message type names to typedef (previously
`blackboxprotobuf.known_messages)
* `default_binary_type` - Change the default type choice for binary fields when
decoding previously unknown fields. Defaults to `bytes` but can be set to
`bytes_hex` to return a hex encoded string instead. `bytes_base64` might be
another option in the future. The type can always be changed for an
individual field by changing the `type` in the typedef.
* `default_types` - Change the default type choice for any wiretype when
decoding a previously unknown field. For example, to default to unsigned
integers for all varints, set `default_types[WIRETYPE_VARINT] =
'uint'`.
The `api` functions like `blackboxprotobuf.decode_message` will default to
using the global `blackboxprotobuf.lib.config.default` object if one is not
specified.
## Type Breakdown
The following is a quick breakdown of wire types and default values. See
<https://developers.google.com/protocol-buffers/docs/encoding> for more detailed
information from Google.
### Variable Length Integers (varint)
The `varint` wire type represents integers with multiple bytes where one bit of
each is dedicated to indicating if it is the last byte. This can be used to
represent integers (signed/unsigned), boolean values or enums. Integers can be
encoded using three variations:
- `uint`: Varint encoding with no representation of negative numbers.
- `int`: Standard encoding but inefficient for negative numbers (always 10 bytes).
- `sint`: Uses ZigZag encoding to efficiently represent negative numbers by
mapping negative numbers into the integer space. For example -1 is converted
to 1, 1 to 2, -2 to 3, and so on. This can result in drastically different
numbers if a type is misinterpreted and either the original or incorrect type
is `sint`.
The default is currently `int` with no ZigZag encoding.
### Fixed32/64
The fixed length wire types have an implicit size based on the wire type. These
support either fixed size integers (signed/unsigned) or fixed size floating
point numbers (float/double). The default type for these is the floating point
type as most integers are more likely to be represented by a varint.
### Length Delimited
Length delimited wire types are prefixed with a `varint` indicating the length.
This is used for strings, bytestrings, inner messages and packed repeated
fields. Messages can generally be identified by validating if it is a valid
protobuf binary. If it is not a message, the default type is a string/byte
which are relatively interchangeable in Python. A different default type (such
as `bytes_hex` can be specified by changing
`blackboxprotobuf.lib.types.default_binary_type`.
Packed repeated fields are arrays of either `varints` or a fixed length wire
type. Non-packed repeated fields use a separate tag (wire type + field number)
for each element, allowing them to be easily identified and parsed. However,
packed repeated fields only have the initial length delimited wire type tag.
The parser is assumed to know the full type already for parsing out the
individual elements. This makes this field type difficult to differentiate from
an arbitrary byte string and will require user intervention to identify. In
protobuf version 2, repeated fields had to be explicitly declared packed in the
definition. In protobuf version 3, repeated fields are packed by default and
are likely to become more common.
Raw data
{
"_id": null,
"home_page": "https://github.com/nccgroup/blackboxprotobuf",
"name": "bbpb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "protobuf",
"author": "Ryan Winkelmaier",
"author_email": "ryan.winkelmaier@nccgroup.com",
"download_url": "https://files.pythonhosted.org/packages/3a/73/74cd6f199336f289b5eec38986f34c6843869a5ec2560f6eb09bf63f312f/bbpb-1.1.2.tar.gz",
"platform": null,
"description": "# BlackBox Protobuf Library\n\n## Description\nBlackbox protobuf library is a Python module for decoding and re-encoding protobuf\nmessages without access to the source protobuf descriptor file. This library\nprovides a simple Python interface to encode/decode messages that can be\nintegrated into other tools.\n\nThis library is targeted towards use in penetration testing where being able to\nmodify messages is critical and a protocol buffer definition may not be readily\navailable.\n\n## Background\nProtocol Buffers (protobufs) are a standard published by Google with\naccompanying libraries for binary serialization of data. Protocol buffers are\ndefined by a `.proto` file known to both the sender and the receiver. The actual\nbinary message does not contain information such as field names or most type\ninformation.\n\nFor each field, the serialized protocol buffer includes two pieces of metadata,\na field number and the wire type. The wire type tells a parser how to parse the\nlength of the field, so that it can be skipped if it is not known (one protocol\nbuffer design goal is being able to handle messages with unknown fields). A\nsingle wire-type generally encompasses multiple protocol buffer types, for\nexample the length delimited wire-type can be used for string, bytestring,\ninner message or packed repeated fields. See\n<https://developers.google.com/protocol-buffers/docs/encoding#structure> for\nthe breakdown of wire types.\n\nThe protocol buffer compiler (`protoc`) does support a similar method of\ndecoding protocol buffers without the definition with the `--decode_raw`\noption. However, it does not provide any functionality to re-encode the decoded\nmessage.\n\n## How it works\nThe library makes a best effort guess of the type based on the provided wire type (and\noccasionally field content) and builds a type definition that can be used to\nre-encode the data. In general, most fields of interest are likely to be parsed\ninto a usable form. Users can optionally pass in custom type definitions that\noverride the guessed type. Custom type definitions also allow naming of fields to\nimprove user friendliness.\n\n# Usage\n## Installation \nThis library depends on internal functions of Google's protobuf Python library\nto do some encoding/decoding of individual fields and six for python 2\ncompatibility.\n\nThe package can be installed from source with:\n\n```\npoetry install\n```\n\nBlackBox Protobuf is also available on PyPi at <https://pypi.org/project/bbpb>.\nIt can be installed with:\n\n```\npip install bbpb\n```\n\n## Interface\nThe main `blackboxprotobuf` module defines an API with the core encode/decode\nmessage functions, along with several convenience functions to make it easier\nto use blackboxprotobuf with a user interface, such as encoding/decoding\ndirectly to JSON and validating modified type definitions.\n\n### Decode \nDecoding functions takes a protobuf bytestring, and optionally\neither a type definition or a known message name mapped to a type definition\n(in `blackboxprotobuf.known_messages`). If a type definition isn't provided, an\nempty message type is assumed and all types are derived from the protobuf\nbinary.\n\nThe decoder returns a tuple containing a dictionary with the decoded data and a\ndictionary containing the generated type definition. If the input type\ndefinition does not include types for all fields in the message, the output\ntype definitions will include type guesses for those fields.\n\nExample use:\n```python\nimport blackboxprotobuf\nimport base64\n\ndata = base64.b64decode('KglNb2RpZnkgTWU=')\nmessage,typedef = blackboxprotobuf.protobuf_to_json(data)\nprint(message)\n```\n\n### Encode\nThe encoding functions takes a Python dictionary containing the data and a type\ndefinition. Unlike decoding, the type definition is required and will fail if\nany fields are not defined. Generally, the type definition should be the output\nfrom the decoding function or a modified version thereof.\n\nExample use:\n```python\nimport blackboxprotobuf\nimport base64\n\ndata = base64.b64decode('KglNb2RpZnkgTWU=')\nmessage,typedef = blackboxprotobuf.decode_message(data)\n\nmessage[5] = 'Modified Me'\n\ndata = blackboxprotobuf.encode_message(message,typedef)\nprint(data)\n```\n\n### Type definition structure\nThe type definition object is a Python dictionary representing the type\nstructure of a message, it includes a type for each field and optionally a\nname. Each entry in the dictionary represents a field in the message. The key\nshould be the field number and the value is a dictionary containing attributes.\n\nAt the minimum the dictionary should contain the 'type' entry which contains a\nstring identifier for the type. Valid type identifiers can be found in\n`blackboxprotobuf/lib/types/type_maps.py`.\n\nMessage fields will also contain one of two entries, 'message_typedef' or\n'message_type_name'. 'message_typedef' should contain a second type definition\nstructure for the inner message. 'message_type_name' should contain the string\nidentifier for a message type previously stored in\n`blackboxprotobuf.known_messages`. If both are specified, the 'message_type_name'\nwill be ignored.\n\n### JSON Encode/Decode\n\nThe `protobuf_to_json` and `protobuf_from_json` functions are convenience\nfunctions for encoding/decoding messages to JSON instead of a python\ndictionary. These functions are designed for user-facing input/output and will\nalso automatically sort the output, try to encode bytestrings for better\nprinting and annotate example values onto the type definition structure.\n\n### Export/import protofile\n\nThe `export_protofile` and `import_protofile` will attempt to convert a\nprotobuffer `.proto` file into the blackboxprotobuf type definition and vice\nversa. These functions provide a higher level interface to\n`blackboxprotobuf.lib.protofile` which only takes a filename. The protofile\nfunctions do not implement a full proper parser and may break on some types.\nOne common case to be aware of is the \"import\" statements in \".proto\" files,\nwhich are not supported. Any imported files must be manually imported with\n`import_protofile` and saved in `blackboxprotobuf.known_messages` first.\n\n\n### Validate Typedef\n\nThe `validate_typedef` function is designed to sanity check modified type\ndefinitions and make sure they are internally consistent and consistent with\nthe previous type definition (if provided). This should help catch issues such\nas changing a field to an incompatible type or duplicate field names.\n\n### Output Helper Functions\n\nThe `json_safe_transform` is a helper function to help create more readable\nJSON output of bytes. It will encode/decode bytes types as `latin1` based on\nthe type in the type definition.\n\nThe `sort_output` is a helper function which sorts the output message based on\nthe field numbers from the typedef. This helps makes the JSON output more\nconsistent and predictable.\n\nThe `sort_typedef` function sorts the fields of the typedef in order to make\nthe output more readable. The message fields are sorted by their number and\ntype fields (eg. name, type, inner message typedef) are sorted to prioritize\nimportant short fields at the top and especially to keep the name and type\nfields from getting buried underneath a long inner typedef.\n\n### Config\n\nMany of the functions accept a `config` keyword argument of the\n`blackboxprotobuf.lib.config.Config` class. The config object allows modifying\nsome of the encoding/decoding functionality and storing some state. This\nreplaces some variables that were global before.\n\nAt the moment this includes:\n\n* `known_types` - Mapping of message type names to typedef (previously\n `blackboxprotobuf.known_messages)\n\n* `default_binary_type` - Change the default type choice for binary fields when\n decoding previously unknown fields. Defaults to `bytes` but can be set to\n `bytes_hex` to return a hex encoded string instead. `bytes_base64` might be\n another option in the future. The type can always be changed for an\n individual field by changing the `type` in the typedef.\n\n* `default_types` - Change the default type choice for any wiretype when\n decoding a previously unknown field. For example, to default to unsigned\n integers for all varints, set `default_types[WIRETYPE_VARINT] =\n 'uint'`.\n\nThe `api` functions like `blackboxprotobuf.decode_message` will default to\nusing the global `blackboxprotobuf.lib.config.default` object if one is not\nspecified.\n\n## Type Breakdown\nThe following is a quick breakdown of wire types and default values. See\n<https://developers.google.com/protocol-buffers/docs/encoding> for more detailed\ninformation from Google.\n\n### Variable Length Integers (varint)\nThe `varint` wire type represents integers with multiple bytes where one bit of\neach is dedicated to indicating if it is the last byte. This can be used to\nrepresent integers (signed/unsigned), boolean values or enums. Integers can be\nencoded using three variations:\n\n- `uint`: Varint encoding with no representation of negative numbers.\n- `int`: Standard encoding but inefficient for negative numbers (always 10 bytes).\n- `sint`: Uses ZigZag encoding to efficiently represent negative numbers by\n mapping negative numbers into the integer space. For example -1 is converted\n to 1, 1 to 2, -2 to 3, and so on. This can result in drastically different\n numbers if a type is misinterpreted and either the original or incorrect type\n is `sint`.\n\nThe default is currently `int` with no ZigZag encoding.\n\n### Fixed32/64\nThe fixed length wire types have an implicit size based on the wire type. These\nsupport either fixed size integers (signed/unsigned) or fixed size floating\npoint numbers (float/double). The default type for these is the floating point\ntype as most integers are more likely to be represented by a varint.\n\n### Length Delimited\nLength delimited wire types are prefixed with a `varint` indicating the length.\nThis is used for strings, bytestrings, inner messages and packed repeated\nfields. Messages can generally be identified by validating if it is a valid\nprotobuf binary. If it is not a message, the default type is a string/byte\nwhich are relatively interchangeable in Python. A different default type (such\nas `bytes_hex` can be specified by changing\n`blackboxprotobuf.lib.types.default_binary_type`.\n\nPacked repeated fields are arrays of either `varints` or a fixed length wire\ntype. Non-packed repeated fields use a separate tag (wire type + field number)\nfor each element, allowing them to be easily identified and parsed. However,\npacked repeated fields only have the initial length delimited wire type tag.\nThe parser is assumed to know the full type already for parsing out the\nindividual elements. This makes this field type difficult to differentiate from\nan arbitrary byte string and will require user intervention to identify. In\nprotobuf version 2, repeated fields had to be explicitly declared packed in the\ndefinition. In protobuf version 3, repeated fields are packed by default and\nare likely to become more common.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Library for working with protobuf messages without a protobuf type definition.",
"version": "1.1.2",
"project_urls": {
"Homepage": "https://github.com/nccgroup/blackboxprotobuf",
"Repository": "https://github.com/nccgroup/blackboxprotobuf"
},
"split_keywords": [
"protobuf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c71993c6883addca71dd183d28ea2442dc3ce36766877474ac1e1c3bcdeda736",
"md5": "7e838ea37c026326079d25f3113c4312",
"sha256": "9662f83b4e941351add0bc6f131dafd3e156eef65e0e00dcbd5b9ac2d55b6123"
},
"downloads": -1,
"filename": "bbpb-1.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7e838ea37c026326079d25f3113c4312",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 35424,
"upload_time": "2024-03-11T14:53:03",
"upload_time_iso_8601": "2024-03-11T14:53:03.634724Z",
"url": "https://files.pythonhosted.org/packages/c7/19/93c6883addca71dd183d28ea2442dc3ce36766877474ac1e1c3bcdeda736/bbpb-1.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3a7374cd6f199336f289b5eec38986f34c6843869a5ec2560f6eb09bf63f312f",
"md5": "ca9e52e4f97c35e8a6eaee400cd02e60",
"sha256": "1140592ffd19cabab7eccd2f9266b77640b0156a24e22fc5de83b64b57bbe164"
},
"downloads": -1,
"filename": "bbpb-1.1.2.tar.gz",
"has_sig": false,
"md5_digest": "ca9e52e4f97c35e8a6eaee400cd02e60",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 28059,
"upload_time": "2024-03-11T14:53:05",
"upload_time_iso_8601": "2024-03-11T14:53:05.468495Z",
"url": "https://files.pythonhosted.org/packages/3a/73/74cd6f199336f289b5eec38986f34c6843869a5ec2560f6eb09bf63f312f/bbpb-1.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-11 14:53:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nccgroup",
"github_project": "blackboxprotobuf",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "bbpb"
}
JSON
