Corpus Replicator
=================
[](https://community-tc.services.mozilla.com/api/github/v1/repository/MozillaSecurity/corpus-replicator/main/latest)
[].num_joined_members&suffix=%20users&url=https%3A%2F%2Fmozilla.modular.im%2F_matrix%2Fclient%2Fr0%2FpublicRooms&style=flat&logo=matrix)](https://riot.im/app/#/room/#fuzzing:mozilla.org)
[](https://pypi.org/project/corpus-replicator)
Corpus Replicator is a corpus generation tool that enables the creation of multiple
unique output files based on templates. The primary intended use case is the
creation of a seed corpus that can be used by fuzzers. Support for additional output
formats can be added via the creation of `Recipes`. If a desired format is unsupported,
support can be added via the creation of a `CorpusGenerator`.
The goal is to create an efficient corpus that maximizes code coverage and minimizes
file size. Small unique files that execute quickly are preferred.
Currently four media types can be generated `animation`, `audio`, `image` and
`video`.
Requirements
------------
Corpus Replicator relies on [FFmpeg](https://ffmpeg.org/).
Installation
------------
```
pip install corpus-replicator
```
Example
-------
This is an example `recipe` file.
```yaml
# "base" contains required entries and default flags
base:
codec: "h264" # name of the codec
container: "mp4" # container/file extension
library: "libx264" # name of library
medium: "video" # supported medium
tool: "ffmpeg" # name of supported tool
default_flags:
encoder: # "encoder" flag group
["-c:v", "libx264"]
resolution: # "resolution" flag group
["-s", "320x240"]
# variations allow flags to be added and overwritten
# one file will be generated for each entry in a flag group
variation:
resolution: # flag group - overwrites default flag group in "base"
- ["-s", "640x480"]
- ["-s", "32x18"]
- ["-s", "64x64"]
monochrome: # flag group - adds new flag group
- ["-vf", "hue=s=0"]
```
Running the recipe will generate a corpus:
```
$ corpus-replicator example.yml video -t test
Generating templates...
1 recipe(s) will be used with 1 template(s) to create 4 file(s).
Generating 4 'video/libx264/h264/mp4' file(s) using template 'test'...
Optimizing corpus, checking for duplicates...
Done.
```
Resulting corpus:
```
$ ls generated-corpus/
video-h264-libx264-test-monochrome-00.mp4
video-h264-libx264-test-resolution-01.mp4
video-h264-libx264-test-resolution-00.mp4
video-h264-libx264-test-resolution-02.mp4
```
A more complex corpus can be generated by using multiple `Recipes` and `Templates` at
once.
Recipes are stored in [src/corpus_replicator/recipes](/src/corpus_replicator/recipes/).
Raw data
{
"_id": null,
"home_page": "https://github.com/MozillaSecurity/corpus-replicator",
"name": "corpus-replicator",
"maintainer": "Mozilla Fuzzing Team",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "fuzzing@mozilla.com",
"keywords": "automation corpus fuzz fuzzing security test testing",
"author": "Tyson Smith",
"author_email": "twsmith@mozilla.com",
"download_url": "https://files.pythonhosted.org/packages/17/f3/42dc7ebee750dc32ecad2e52d7da631ca066b5f0e8de55048385d4a4ce2b/corpus-replicator-1.1.2.tar.gz",
"platform": null,
"description": "Corpus Replicator\n=================\n[](https://community-tc.services.mozilla.com/api/github/v1/repository/MozillaSecurity/corpus-replicator/main/latest)\n[].num_joined_members&suffix=%20users&url=https%3A%2F%2Fmozilla.modular.im%2F_matrix%2Fclient%2Fr0%2FpublicRooms&style=flat&logo=matrix)](https://riot.im/app/#/room/#fuzzing:mozilla.org)\n[](https://pypi.org/project/corpus-replicator)\n\nCorpus Replicator is a corpus generation tool that enables the creation of multiple\nunique output files based on templates. The primary intended use case is the\ncreation of a seed corpus that can be used by fuzzers. Support for additional output\nformats can be added via the creation of `Recipes`. If a desired format is unsupported,\nsupport can be added via the creation of a `CorpusGenerator`.\n\nThe goal is to create an efficient corpus that maximizes code coverage and minimizes\nfile size. Small unique files that execute quickly are preferred.\n\nCurrently four media types can be generated `animation`, `audio`, `image` and\n`video`.\n\nRequirements\n------------\n\nCorpus Replicator relies on [FFmpeg](https://ffmpeg.org/).\n\nInstallation\n------------\n```\npip install corpus-replicator\n```\n\nExample\n-------\n\nThis is an example `recipe` file.\n\n```yaml\n# \"base\" contains required entries and default flags\nbase:\n codec: \"h264\" # name of the codec\n container: \"mp4\" # container/file extension\n library: \"libx264\" # name of library\n medium: \"video\" # supported medium\n tool: \"ffmpeg\" # name of supported tool\n default_flags:\n encoder: # \"encoder\" flag group\n [\"-c:v\", \"libx264\"]\n resolution: # \"resolution\" flag group\n [\"-s\", \"320x240\"]\n\n# variations allow flags to be added and overwritten\n# one file will be generated for each entry in a flag group\nvariation:\n resolution: # flag group - overwrites default flag group in \"base\"\n - [\"-s\", \"640x480\"]\n - [\"-s\", \"32x18\"]\n - [\"-s\", \"64x64\"]\n monochrome: # flag group - adds new flag group\n - [\"-vf\", \"hue=s=0\"]\n```\n\nRunning the recipe will generate a corpus:\n```\n$ corpus-replicator example.yml video -t test\nGenerating templates...\n1 recipe(s) will be used with 1 template(s) to create 4 file(s).\nGenerating 4 'video/libx264/h264/mp4' file(s) using template 'test'...\nOptimizing corpus, checking for duplicates...\nDone.\n```\n\nResulting corpus:\n```\n$ ls generated-corpus/\nvideo-h264-libx264-test-monochrome-00.mp4\nvideo-h264-libx264-test-resolution-01.mp4\nvideo-h264-libx264-test-resolution-00.mp4\nvideo-h264-libx264-test-resolution-02.mp4\n```\n\nA more complex corpus can be generated by using multiple `Recipes` and `Templates` at\nonce.\n\nRecipes are stored in [src/corpus_replicator/recipes](/src/corpus_replicator/recipes/).\n",
"bugtrack_url": null,
"license": "MPL 2.0",
"summary": "",
"version": "1.1.2",
"project_urls": {
"Homepage": "https://github.com/MozillaSecurity/corpus-replicator"
},
"split_keywords": [
"automation",
"corpus",
"fuzz",
"fuzzing",
"security",
"test",
"testing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4f6c5e73b0839e88392bc7264374a3bdb16e25110369b8c3c26dcaf72728f705",
"md5": "ee93f3d56c3e0804ce2488d6b0d392f8",
"sha256": "966c028f82ca7ae754e5a0a28a0381fa920b8168c3440603f76c5328df503260"
},
"downloads": -1,
"filename": "corpus_replicator-1.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ee93f3d56c3e0804ce2488d6b0d392f8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 40260,
"upload_time": "2023-11-09T23:51:38",
"upload_time_iso_8601": "2023-11-09T23:51:38.625642Z",
"url": "https://files.pythonhosted.org/packages/4f/6c/5e73b0839e88392bc7264374a3bdb16e25110369b8c3c26dcaf72728f705/corpus_replicator-1.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "17f342dc7ebee750dc32ecad2e52d7da631ca066b5f0e8de55048385d4a4ce2b",
"md5": "300289b7f3025444c0097a85f9154515",
"sha256": "f75c72b4e5f7337cc1e0b6c92ab3749b5ec6864ef56195a22679b86340cc64c4"
},
"downloads": -1,
"filename": "corpus-replicator-1.1.2.tar.gz",
"has_sig": false,
"md5_digest": "300289b7f3025444c0097a85f9154515",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 27671,
"upload_time": "2023-11-09T23:51:39",
"upload_time_iso_8601": "2023-11-09T23:51:39.676731Z",
"url": "https://files.pythonhosted.org/packages/17/f3/42dc7ebee750dc32ecad2e52d7da631ca066b5f0e8de55048385d4a4ce2b/corpus-replicator-1.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-09 23:51:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MozillaSecurity",
"github_project": "corpus-replicator",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "corpus-replicator"
}
JSON
