| Name | protodeep JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | |
| Summary | A tool to help reversing protobuf. |
| upload_time | 2023-03-25 15:15:15 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | MIT |
| keywords |
protobuf
reverse
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<br>
# 🏄♂️ Description
ProtoDeep is an easy to use tool that allows you to decode and analyze protobuf data.
It is **heavily based** on the well-established [Blackbox Protobuf](https://github.com/nccgroup/blackboxprotobuf) package, and inspired by [Protobuf Inspector](https://github.com/mildsunrise/protobuf-inspector) for the custom definitions feature.\
This project was originally intended to be integrated into the [GHunt RDTK](https://github.com/mxrch/ghunt-rdtk), but the script grew a lot and ended up becoming a tool that can help many people.
Features :
- CLI usage
- Python library usage
- Make your own definitions
- Easily match / filter data
- Export and compile on the fly
<br>
Example of using ProtoDeep on the Google's Play Store `searchList` endpoint, with custom definitions :
# ✔️ Requirements
- Python >= 3.10
# ⚙️ Installation
```bash
$ pip3 install pipx
$ pipx ensurepath
$ pipx install protodeep
```
It will automatically use venvs to avoid dependency conflicts with other projects.
# 💃 Usage
## Help menu
```
Usage: main.py [-h] -t TYPE [-d DEFINITIONS] [-na] [-s] [-b] [-bi NUMBER] [-he] [-np] [-m MASK] [-mk MASK] [-mv MASK]
[-f MASK] [-fk MASK] [-fv MASK] [-epf [PROTOFILE_FILENAME]] [-epd [PROTODEEP_FILENAME]]
[-c [PYTHON_FILENAME]] [-n SCHEMA_NAME]
[proto_file]
Positional Arguments:
proto_file
Options:
-h, --help show this help message and exit
-t, --type TYPE Either protobuf (raw protobuf content), or protodeep (a ProtoDeep file).
-d, --definitions DEFINITIONS
The file containing the custom protobuf definitions.
-na, --no-autodetect Don't try to autodetect if it's a raw HTTP request.
-s, --stdin Parse from stdin.
-b, --base64 If this is a base64 input, so it automatically decodes it.
-bi, --bruteforce-index NUMBER
The index up to which to try bruteforce to find Protobuf content. Default : 20
-he, --hide-empty Hide the empty values.
-np, --no-print Don't print the decoded protobuf.
-m, --match MASK Match anything with the given string. You can use '?' and '*' to wildcard match.Ex : "*token*"
-mk, --match-keychain MASK
Match keychains with the given string.
-mv, --match-value MASK
Match values with the given string.
-f, --filter MASK Filter anything with the given string. You can use '?' and '*' to wildcard match.
-fk, --filter-keychain MASK
Filter keychains with the given string.
-fv, --filter-value MASK
Filter values with the given string.
-epf, --export-protofile [PROTOFILE_FILENAME]
Export the proto file with the definitions.
-epd, --export-protodeep [PROTODEEP_FILENAME]
Export a protodeep file, to reuse in ProtoDeep.
-c, --compile [PYTHON_FILENAME]
Compile protobuf into a Python file.
-n, --name SCHEMA_NAME
Name of the schema when exporting into a proto file.
```
## Concepts
Here are the main concepts to know when using ProtoDeep :
### Output example
- Keychains : Since the protobuf is made of nested keys and values, keychains are a way to precisely identify a value in the decoded protobuf. It's the key sequence used to access the value.
- Pretty Keychains : This is the same as keychains, except that the keys are replaced by the names defined in the custom definitions.
- Type : Type of the value. Supported types are listed in this [blackboxprotobuf's file](https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/blackboxprotobuf/lib/types/type_maps.py).
- Value : The value found in the protobuf data.
- Iterator : ProtoDeep will try to autodetect repeated messages, and will print elements of these arrays with the `i<position>` key, so you can know the position of the element in the list.
### Custom definitions file
- Definitions : It is a JSON file, containg a dict with the keychains as keys, and names as values. You can specify the type of a value by adding `:<type>` next to it. By doing so, ProtoDeep will detect it, and relaunch the decoding of the protobuf data with this new type. *Note that it will only work when using protobuf data, not a protodeep file, since data has already been decoded.*
**Have fun 🥰💞**
# 🧑💻 Developers
To use ProtoDeep as a lib, you can't use pipx because it uses a venv.\
So you should install ProtoDeep with pip :
```bash
$ pip3 install protodeep
```
And now, you should be able to `import protodeep` in your projects like this :
```python
from protodeep.lib import guess_schema
with open('protobuf_data.bin', 'rb') as f:
raw = f.read()
protodeep_schema = guess_schema(data=raw)
protodeep_schema.pretty_print(hide_empty=True, filter_any=["*term_to_filter*"])
protodeep_schema.export_protodeep("obj.pdeep")
```
## Testing
Thanks to [learn-more](https://github.com/learn-more), tests are now available, to test the CLI and lib usage !\
You can launch the tests by doing :
```bash
$ pip3 install -r requirements-dev.txt
$ pytest
```
*Tests are run automatically through [GitHub Actions](https://github.com/mxrch/ProtoDeep/actions).*
# 📕 Cheatsheet
*Some examples so you know how to use protodeep :*
Reading a protobuf file:
```bash
$ protodeep protobuf_data.bin -t protobuf
```
Read a protobuf file, provide a custom definitions file, hide the output, export to protofile & protodeep, and compile a Python file called "final.py" :
```bash
$ protodeep protobuf_data.bin -t protobuf -d search_ps_defs.json -np -epf -epd -c final.py
```
*Names for the arguments `--export-protofile` / `--export-protodeep` / `--compile` are optional. If they aren't set, a default name will be used.*
Read protobuf from stdin, provide a custom definitions file, match the keychain "11,1,1,2", hide the empty values, and filter lines where the word "access" and "denied" are present, and lines where the word "tiktok" is present:
```bash
$ curl -s <protobuf_endpoint> | protodeep --stdin -t protobuf -mk "11,1,1,2" -he -f "*access*denied*" -f "*tiktok*"
```
*Matching / filtering arguments can be used as many times as you like.*
## Thanks
- The [NCC Group](https://github.com/nccgroup) for the super useful [blackboxprotobuf](https://github.com/nccgroup/blackboxprotobuf) project
- [mildsunrise](https://github.com/mildsunrise) for [protobuf-inspector](https://github.com/mildsunrise/protobuf-inspector)
- The HideAndSec team 💜 (blog : https://hideandsec.sh)
## Sponsors
Thanks to these awesome people for supporting me !
<!-- sponsors --><a href="https://github.com/BlWasp"><img src="https://github.com/BlWasp.png" width="50px" alt="BlWasp" /></a> <a href="https://github.com/pl4nty"><img src="https://github.com/pl4nty.png" width="50px" alt="pl4nty" /></a> <a href="https://github.com/0xN0x"><img src="https://github.com/0xN0x.png" width="50px" alt="0xN0x" /></a> <a href="https://github.com/C3n7ral051nt4g3ncy"><img src="https://github.com/C3n7ral051nt4g3ncy.png" width="50px" alt="C3n7ral051nt4g3ncy" /></a> <a href="https://github.com/rayanlecat"><img src="https://github.com/rayanlecat.png" width="50px" alt="rayanlecat" /></a> <a href="https://github.com/ajmeese7"><img src="https://github.com/ajmeese7.png" width="50px" alt="ajmeese7" /></a> <a href="https://github.com/im-hanzou"><img src="https://github.com/im-hanzou.png" width="50px" alt="im-hanzou" /></a> <a href="https://github.com/gingeleski"><img src="https://github.com/gingeleski.png" width="50px" alt="gingeleski" /></a> <!-- sponsors -->
\
You like my work ?\
[Sponsor me](https://github.com/sponsors/mxrch) on GitHub ! 🤗
Raw data
{
"_id": null,
"home_page": "",
"name": "protodeep",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "protobuf,reverse",
"author": "",
"author_email": "mxrch <mxrch.dev@pm.me>",
"download_url": "",
"platform": null,
"description": "\n\n<br>\n\n\n\n\n# \ud83c\udfc4\u200d\u2642\ufe0f Description\n\nProtoDeep is an easy to use tool that allows you to decode and analyze protobuf data.\n\nIt is **heavily based** on the well-established [Blackbox Protobuf](https://github.com/nccgroup/blackboxprotobuf) package, and inspired by [Protobuf Inspector](https://github.com/mildsunrise/protobuf-inspector) for the custom definitions feature.\\\nThis project was originally intended to be integrated into the [GHunt RDTK](https://github.com/mxrch/ghunt-rdtk), but the script grew a lot and ended up becoming a tool that can help many people.\n\nFeatures :\n- CLI usage\n- Python library usage\n- Make your own definitions\n- Easily match / filter data\n- Export and compile on the fly\n\n<br>\n\nExample of using ProtoDeep on the Google's Play Store `searchList` endpoint, with custom definitions :\n\n\n\n# \u2714\ufe0f Requirements\n- Python >= 3.10\n\n# \u2699\ufe0f Installation\n\n```bash\n$ pip3 install pipx\n$ pipx ensurepath\n$ pipx install protodeep\n```\nIt will automatically use venvs to avoid dependency conflicts with other projects.\n\n# \ud83d\udc83 Usage\n\n## Help menu\n\n```\nUsage: main.py [-h] -t TYPE [-d DEFINITIONS] [-na] [-s] [-b] [-bi NUMBER] [-he] [-np] [-m MASK] [-mk MASK] [-mv MASK]\n [-f MASK] [-fk MASK] [-fv MASK] [-epf [PROTOFILE_FILENAME]] [-epd [PROTODEEP_FILENAME]]\n [-c [PYTHON_FILENAME]] [-n SCHEMA_NAME]\n [proto_file]\n\nPositional Arguments:\n proto_file\n\nOptions:\n -h, --help show this help message and exit\n -t, --type TYPE Either protobuf (raw protobuf content), or protodeep (a ProtoDeep file).\n -d, --definitions DEFINITIONS\n The file containing the custom protobuf definitions.\n -na, --no-autodetect Don't try to autodetect if it's a raw HTTP request.\n -s, --stdin Parse from stdin.\n -b, --base64 If this is a base64 input, so it automatically decodes it.\n -bi, --bruteforce-index NUMBER\n The index up to which to try bruteforce to find Protobuf content. Default : 20\n -he, --hide-empty Hide the empty values.\n -np, --no-print Don't print the decoded protobuf.\n -m, --match MASK Match anything with the given string. You can use '?' and '*' to wildcard match.Ex : \"*token*\"\n -mk, --match-keychain MASK\n Match keychains with the given string.\n -mv, --match-value MASK\n Match values with the given string.\n -f, --filter MASK Filter anything with the given string. You can use '?' and '*' to wildcard match.\n -fk, --filter-keychain MASK\n Filter keychains with the given string.\n -fv, --filter-value MASK\n Filter values with the given string.\n -epf, --export-protofile [PROTOFILE_FILENAME]\n Export the proto file with the definitions.\n -epd, --export-protodeep [PROTODEEP_FILENAME]\n Export a protodeep file, to reuse in ProtoDeep.\n -c, --compile [PYTHON_FILENAME]\n Compile protobuf into a Python file.\n -n, --name SCHEMA_NAME\n Name of the schema when exporting into a proto file.\n```\n\n## Concepts\n\nHere are the main concepts to know when using ProtoDeep :\n\n### Output example\n\n\n- Keychains : Since the protobuf is made of nested keys and values, keychains are a way to precisely identify a value in the decoded protobuf. It's the key sequence used to access the value.\n- Pretty Keychains : This is the same as keychains, except that the keys are replaced by the names defined in the custom definitions.\n- Type : Type of the value. Supported types are listed in this [blackboxprotobuf's file](https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/blackboxprotobuf/lib/types/type_maps.py).\n- Value : The value found in the protobuf data.\n- Iterator : ProtoDeep will try to autodetect repeated messages, and will print elements of these arrays with the `i<position>` key, so you can know the position of the element in the list.\n\n### Custom definitions file\n\n\n\n- Definitions : It is a JSON file, containg a dict with the keychains as keys, and names as values. You can specify the type of a value by adding `:<type>` next to it. By doing so, ProtoDeep will detect it, and relaunch the decoding of the protobuf data with this new type. *Note that it will only work when using protobuf data, not a protodeep file, since data has already been decoded.*\n\n**Have fun \ud83e\udd70\ud83d\udc9e**\n\n# \ud83e\uddd1\u200d\ud83d\udcbb Developers\n\nTo use ProtoDeep as a lib, you can't use pipx because it uses a venv.\\\nSo you should install ProtoDeep with pip :\n```bash\n$ pip3 install protodeep\n```\n\nAnd now, you should be able to `import protodeep` in your projects like this :\n\n```python\nfrom protodeep.lib import guess_schema\n\nwith open('protobuf_data.bin', 'rb') as f:\n raw = f.read()\n\nprotodeep_schema = guess_schema(data=raw)\nprotodeep_schema.pretty_print(hide_empty=True, filter_any=[\"*term_to_filter*\"])\nprotodeep_schema.export_protodeep(\"obj.pdeep\")\n```\n\n## Testing\n\nThanks to [learn-more](https://github.com/learn-more), tests are now available, to test the CLI and lib usage !\\\nYou can launch the tests by doing :\n```bash\n$ pip3 install -r requirements-dev.txt\n$ pytest\n```\n\n*Tests are run automatically through [GitHub Actions](https://github.com/mxrch/ProtoDeep/actions).*\n\n# \ud83d\udcd5 Cheatsheet\n\n*Some examples so you know how to use protodeep :*\n\nReading a protobuf file:\n```bash\n$ protodeep protobuf_data.bin -t protobuf\n```\n\nRead a protobuf file, provide a custom definitions file, hide the output, export to protofile & protodeep, and compile a Python file called \"final.py\" :\n```bash\n$ protodeep protobuf_data.bin -t protobuf -d search_ps_defs.json -np -epf -epd -c final.py\n```\n\n*Names for the arguments `--export-protofile` / `--export-protodeep` / `--compile` are optional. If they aren't set, a default name will be used.*\n\nRead protobuf from stdin, provide a custom definitions file, match the keychain \"11,1,1,2\", hide the empty values, and filter lines where the word \"access\" and \"denied\" are present, and lines where the word \"tiktok\" is present:\n```bash\n$ curl -s <protobuf_endpoint> | protodeep --stdin -t protobuf -mk \"11,1,1,2\" -he -f \"*access*denied*\" -f \"*tiktok*\"\n```\n\n*Matching / filtering arguments can be used as many times as you like.*\n\n## Thanks\n\n- The [NCC Group](https://github.com/nccgroup) for the super useful [blackboxprotobuf](https://github.com/nccgroup/blackboxprotobuf) project\n- [mildsunrise](https://github.com/mildsunrise) for [protobuf-inspector](https://github.com/mildsunrise/protobuf-inspector)\n- The HideAndSec team \ud83d\udc9c (blog : https://hideandsec.sh)\n\n## Sponsors\n\nThanks to these awesome people for supporting me !\n\n<!-- sponsors --><a href=\"https://github.com/BlWasp\"><img src=\"https://github.com/BlWasp.png\" width=\"50px\" alt=\"BlWasp\" /></a> <a href=\"https://github.com/pl4nty\"><img src=\"https://github.com/pl4nty.png\" width=\"50px\" alt=\"pl4nty\" /></a> <a href=\"https://github.com/0xN0x\"><img src=\"https://github.com/0xN0x.png\" width=\"50px\" alt=\"0xN0x\" /></a> <a href=\"https://github.com/C3n7ral051nt4g3ncy\"><img src=\"https://github.com/C3n7ral051nt4g3ncy.png\" width=\"50px\" alt=\"C3n7ral051nt4g3ncy\" /></a> <a href=\"https://github.com/rayanlecat\"><img src=\"https://github.com/rayanlecat.png\" width=\"50px\" alt=\"rayanlecat\" /></a> <a href=\"https://github.com/ajmeese7\"><img src=\"https://github.com/ajmeese7.png\" width=\"50px\" alt=\"ajmeese7\" /></a> <a href=\"https://github.com/im-hanzou\"><img src=\"https://github.com/im-hanzou.png\" width=\"50px\" alt=\"im-hanzou\" /></a> <a href=\"https://github.com/gingeleski\"><img src=\"https://github.com/gingeleski.png\" width=\"50px\" alt=\"gingeleski\" /></a> <!-- sponsors -->\n\n\\\nYou like my work ?\\\n[Sponsor me](https://github.com/sponsors/mxrch) on GitHub ! \ud83e\udd17\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool to help reversing protobuf.",
"version": "1.1.0",
"split_keywords": [
"protobuf",
"reverse"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4e2de637c19ea96c7a863e7305fc3199150103b4fe79c746dc23d331309aa5ca",
"md5": "21121f0ec5a1035358fa2baac5c730e4",
"sha256": "0a22f9b24a7f6bcb477abc163754e942bc36763dfb44138676400f993a8b90ad"
},
"downloads": -1,
"filename": "protodeep-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "21121f0ec5a1035358fa2baac5c730e4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 47382,
"upload_time": "2023-03-25T15:15:15",
"upload_time_iso_8601": "2023-03-25T15:15:15.650990Z",
"url": "https://files.pythonhosted.org/packages/4e/2d/e637c19ea96c7a863e7305fc3199150103b4fe79c746dc23d331309aa5ca/protodeep-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-25 15:15:15",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "protodeep"
}
