# OWASP CRS Rules parser
Incomplete parser model and sample application for parsing [Core Rule Set](https://github.com/coreruleset/coreruleset/) written in the ModSecurity DSL SecRule language. It uses the python library [textX](http://www.igordejanovic.net/textX/) for parsing.
## How to use it (CLI):
1. Install dependencies
Dependencies can be installed system-wide, or just for your user (using `--user`).
System-wide:
```shell
sudo pip install secrules-parsing
```
User:
```shell
pip install --user secrules-parsing
```
2. Execute `secrules-parser` specifying the location of the files you want to scan using the -f/--files argument. This takes wildcards or individual files.
`$ secrules-parser -c -f /owasp-crs/rules/*.conf`
3. Add flags to accomplish needed tasks:
- -h, --help:
* *Description:* show the help message and exit
* *Example:* `$ secrules-parser -h`
- -r, --regex:
* *Description:* Extract regular expressions from rules file
* *Example:*
```
$ secrules-parser --regex -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
{"/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf": [{"920100": ["^(?i:(?:[a-z]{3,10}\\s+(?:\\w{3,7}?://[\\w\\-\\./]*(?::\\d+)?)?/[^?#]*(?:\\?[^#\\s]*)?(?:#[\\S]*)?|connect (?:\\d{1,3}\\.){3}\\d{1,3}\\.?(?::\\d+)?|options \\*)\\s+[\\w\\./]+|get /[^?#]*(?:\\?[^#\\s]*)?(?:#[\\S]*)?)$"]}, {"920120": ["(?<!&(?:[aAoOuUyY]uml)|&(?:[aAeEiIoOuU]circ)|&(?:[eEiIoOuUyY]acute)|&(?:[aAeEiIoOuU]grave)|&(?:[cC]cedil)|&(?:[aAnNoO]tilde)|&(?:amp)|&(?:apos));|['\\\"=]"]}, {"920160": ["^\\d+$"]}, {"920170": ["^(?:GET|HEAD)$"]}, {"920171": ["^(?:GET|HEAD)$"]}, {"920180": ["^POST$"]}, {"920190": ["(\\d+)\\-(\\d+)\\,"]}, {"920210": ["\\b(?:keep-alive|close),\\s?(?:keep-alive|close)\\b"]}, {"920220": ["\\%(?:(?!$|\\W)|[0-9a-fA-F]{2}|u[0-9a-fA-F]{4})"]}, {"920240": ["^(?:application\\/x-www-form-urlencoded|text\\/xml)(?:;(?:\\s?charset\\s?=\\s?[\\w\\d\\-]{1,18})?)??$"]}, {"920260": ["\\%u[fF]{2}[0-9a-fA-F]{2}"]}, {"920290": ["^$"]}, {"920310": ["^$"]}, {"920311": ["^$"]}, {"920330": ["^$"]}, {"920340": ["^0$"]}, {"920350": ["^[\\d.:]+$"]}, {"920420": ["^(?:GET|HEAD|PROPFIND|OPTIONS)$"]}, {"920440": ["\\.(.*)$"]}, {"920450": ["^.*$"]}, {"920200": ["^bytes=(?:(?:\\d+)?\\-(?:\\d+)?\\s*,?\\s*){6}"]}, {"920230": ["\\%((?!$|\\W)|[0-9a-fA-F]{2}|u[0-9a-fA-F]{4})"]}, {"920121": ["['\\\";=]"]}, {"920460": ["(?<!\\Q\\\\\\E)\\Q\\\\\\E[cdeghijklmpqwxyz123456789]"]}]}
```
* -c, --correctness:
* *Description:* Check the validity of the syntax
* *Example:*
```
$ secrules-parser -c -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
Syntax OK: ../../../rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
```
* -v, --verbose
* *Description:* Print verbose messages
* *Example:*
```
$ secrules-parser -c -v -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
...
```
* -o FILE, --output FILE
* *Description:* Output results to file
* *Example:*
```
$ secrules-parser -c -o out.json -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
```
* --output-type github | plain
* *Description:* Desired output format. Useful if running from Github Actions and you want annotated output
* *Example:*
```
$ secrules-parser -c --output-type github -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
```
## How to use it (API):
### process_rules(list files)
Takes a list of file path's and returns models
```python
import glob
import os
from secrules_parsing import parser
# Extract all of our pathing
files = glob.glob("../../rules/*.conf")
# Pass absolute paths because of module location
files = [os.path.abspath(path) for path in files]
models = parser.process_rules(files)
```
### get_correctness(list files, list models)
```python
import glob
import os
from secrules_parsing import parser
# Extract all of our pathing
files = glob.glob("../../rules/*.conf")
# Pass absolute paths because of module location
files = [os.path.abspath(path) for path in files]
models = parser.process_rules(files)
parser.get_correctness(files, models)
```
## Development
If you want to modify this module, follow these steps:
1. Clone this repository: `git clone git@github.com:coreruleset/secrules_parsing.git`
1. Do not forget to install dependencies using [poetry](https://python-poetry.org/docs/): `poetry install` first!
1. Edit and change the files you want.
1. Write tests! Tests are in the `tests` subdirectory
1. Create a PR [here](https://github.com/coreruleset/secrules_parsing/compare), and ask for review!
## Misc
To visualize the syntax tree, use:
```
textx visualize secrules.tx
dot -Tpng -O secrules.tx.dot
```
Then review the generated PNG modsec.tx.dot.png!
Please file an [issue](https://github.com/coreruleset/secrules_parsing/issues) if you find a bug or you want some feature added.
Raw data
{
"_id": null,
"home_page": "https://github.com/coreruleset/secrules_parsing",
"name": "secrules-parsing",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.7",
"maintainer_email": null,
"keywords": "secrule, modsecurity, parser, textX",
"author": "Felipe Zipitria",
"author_email": "felipe.zipitria@owasp.org",
"download_url": "https://files.pythonhosted.org/packages/f5/af/070d33f04428fd91a2103d1adeafcfff95fd41d8d9689862394c33ba6d75/secrules_parsing-0.2.10.tar.gz",
"platform": null,
"description": "# OWASP CRS Rules parser\n\nIncomplete parser model and sample application for parsing [Core Rule Set](https://github.com/coreruleset/coreruleset/) written in the ModSecurity DSL SecRule language. It uses the python library [textX](http://www.igordejanovic.net/textX/) for parsing.\n\n## How to use it (CLI):\n\n1. Install dependencies\n Dependencies can be installed system-wide, or just for your user (using `--user`).\n\n System-wide:\n ```shell\n sudo pip install secrules-parsing\n ```\n User:\n ```shell\n pip install --user secrules-parsing\n ```\n\n2. Execute `secrules-parser` specifying the location of the files you want to scan using the -f/--files argument. This takes wildcards or individual files.\n `$ secrules-parser -c -f /owasp-crs/rules/*.conf`\n\n3. Add flags to accomplish needed tasks:\n\n - -h, --help:\n * *Description:* show the help message and exit\n * *Example:* `$ secrules-parser -h`\n\n - -r, --regex:\n * *Description:* Extract regular expressions from rules file\n * *Example:*\n ```\n $ secrules-parser --regex -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\n {\"/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\": [{\"920100\": [\"^(?i:(?:[a-z]{3,10}\\\\s+(?:\\\\w{3,7}?://[\\\\w\\\\-\\\\./]*(?::\\\\d+)?)?/[^?#]*(?:\\\\?[^#\\\\s]*)?(?:#[\\\\S]*)?|connect (?:\\\\d{1,3}\\\\.){3}\\\\d{1,3}\\\\.?(?::\\\\d+)?|options \\\\*)\\\\s+[\\\\w\\\\./]+|get /[^?#]*(?:\\\\?[^#\\\\s]*)?(?:#[\\\\S]*)?)$\"]}, {\"920120\": [\"(?<!&(?:[aAoOuUyY]uml)|&(?:[aAeEiIoOuU]circ)|&(?:[eEiIoOuUyY]acute)|&(?:[aAeEiIoOuU]grave)|&(?:[cC]cedil)|&(?:[aAnNoO]tilde)|&(?:amp)|&(?:apos));|['\\\\\\\"=]\"]}, {\"920160\": [\"^\\\\d+$\"]}, {\"920170\": [\"^(?:GET|HEAD)$\"]}, {\"920171\": [\"^(?:GET|HEAD)$\"]}, {\"920180\": [\"^POST$\"]}, {\"920190\": [\"(\\\\d+)\\\\-(\\\\d+)\\\\,\"]}, {\"920210\": [\"\\\\b(?:keep-alive|close),\\\\s?(?:keep-alive|close)\\\\b\"]}, {\"920220\": [\"\\\\%(?:(?!$|\\\\W)|[0-9a-fA-F]{2}|u[0-9a-fA-F]{4})\"]}, {\"920240\": [\"^(?:application\\\\/x-www-form-urlencoded|text\\\\/xml)(?:;(?:\\\\s?charset\\\\s?=\\\\s?[\\\\w\\\\d\\\\-]{1,18})?)??$\"]}, {\"920260\": [\"\\\\%u[fF]{2}[0-9a-fA-F]{2}\"]}, {\"920290\": [\"^$\"]}, {\"920310\": [\"^$\"]}, {\"920311\": [\"^$\"]}, {\"920330\": [\"^$\"]}, {\"920340\": [\"^0$\"]}, {\"920350\": [\"^[\\\\d.:]+$\"]}, {\"920420\": [\"^(?:GET|HEAD|PROPFIND|OPTIONS)$\"]}, {\"920440\": [\"\\\\.(.*)$\"]}, {\"920450\": [\"^.*$\"]}, {\"920200\": [\"^bytes=(?:(?:\\\\d+)?\\\\-(?:\\\\d+)?\\\\s*,?\\\\s*){6}\"]}, {\"920230\": [\"\\\\%((?!$|\\\\W)|[0-9a-fA-F]{2}|u[0-9a-fA-F]{4})\"]}, {\"920121\": [\"['\\\\\\\";=]\"]}, {\"920460\": [\"(?<!\\\\Q\\\\\\\\\\\\E)\\\\Q\\\\\\\\\\\\E[cdeghijklmpqwxyz123456789]\"]}]}\n ```\n\n * -c, --correctness:\n * *Description:* Check the validity of the syntax\n * *Example:*\n ```\n $ secrules-parser -c -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\n Syntax OK: ../../../rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\n ```\n\n * -v, --verbose\n * *Description:* Print verbose messages\n * *Example:*\n ```\n $ secrules-parser -c -v -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\n ...\n ```\n\n * -o FILE, --output FILE\n * *Description:* Output results to file\n * *Example:*\n ```\n $ secrules-parser -c -o out.json -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf \n ```\n\n * --output-type github | plain\n * *Description:* Desired output format. Useful if running from Github Actions and you want annotated output\n * *Example:*\n ```\n $ secrules-parser -c --output-type github -f /owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\n ```\n\n## How to use it (API):\n\n### process_rules(list files)\nTakes a list of file path's and returns models\n```python\nimport glob\nimport os\nfrom secrules_parsing import parser\n\n# Extract all of our pathing\nfiles = glob.glob(\"../../rules/*.conf\")\n# Pass absolute paths because of module location\nfiles = [os.path.abspath(path) for path in files]\nmodels = parser.process_rules(files)\n```\n\n### get_correctness(list files, list models)\n```python\nimport glob\nimport os\nfrom secrules_parsing import parser\n\n# Extract all of our pathing\nfiles = glob.glob(\"../../rules/*.conf\")\n# Pass absolute paths because of module location\nfiles = [os.path.abspath(path) for path in files]\nmodels = parser.process_rules(files)\nparser.get_correctness(files, models)\n```\n\n## Development\n\nIf you want to modify this module, follow these steps:\n1. Clone this repository: `git clone git@github.com:coreruleset/secrules_parsing.git`\n1. Do not forget to install dependencies using [poetry](https://python-poetry.org/docs/): `poetry install` first!\n1. Edit and change the files you want.\n1. Write tests! Tests are in the `tests` subdirectory\n1. Create a PR [here](https://github.com/coreruleset/secrules_parsing/compare), and ask for review!\n\n## Misc\n\nTo visualize the syntax tree, use:\n\n```\ntextx visualize secrules.tx\ndot -Tpng -O secrules.tx.dot\n```\n\nThen review the generated PNG modsec.tx.dot.png!\n\nPlease file an [issue](https://github.com/coreruleset/secrules_parsing/issues) if you find a bug or you want some feature added.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "ModSecurity DSL Parser package using textX",
"version": "0.2.10",
"project_urls": {
"Bug Tracker": "https://github.com/coreruleset/secrules_parsing/issues",
"Documentation": "https://github.com/coreruleset/secrules_parsing",
"Homepage": "https://github.com/coreruleset/secrules_parsing",
"Repository": "https://github.com/coreruleset/secrules_parsing"
},
"split_keywords": [
"secrule",
" modsecurity",
" parser",
" textx"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4d41a549943434b41d79c9e8537035fbf25bd45610a9bba55c62fffbb207662a",
"md5": "2526b03470bba9e1ed7f0248babb7eca",
"sha256": "9018141bd091f6e639e2131e2973a7add63db5fcb08d79b0a828422b74f3663a"
},
"downloads": -1,
"filename": "secrules_parsing-0.2.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2526b03470bba9e1ed7f0248babb7eca",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.7",
"size": 15473,
"upload_time": "2024-05-29T21:13:43",
"upload_time_iso_8601": "2024-05-29T21:13:43.357709Z",
"url": "https://files.pythonhosted.org/packages/4d/41/a549943434b41d79c9e8537035fbf25bd45610a9bba55c62fffbb207662a/secrules_parsing-0.2.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f5af070d33f04428fd91a2103d1adeafcfff95fd41d8d9689862394c33ba6d75",
"md5": "0cd3a6bff10b8d7dd4cced67a39f6d25",
"sha256": "23e046d0ff1743540ac1c5f1feffb3a18bf3a2f28b65d3019d44b08640430ef0"
},
"downloads": -1,
"filename": "secrules_parsing-0.2.10.tar.gz",
"has_sig": false,
"md5_digest": "0cd3a6bff10b8d7dd4cced67a39f6d25",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.7",
"size": 14961,
"upload_time": "2024-05-29T21:13:44",
"upload_time_iso_8601": "2024-05-29T21:13:44.484564Z",
"url": "https://files.pythonhosted.org/packages/f5/af/070d33f04428fd91a2103d1adeafcfff95fd41d8d9689862394c33ba6d75/secrules_parsing-0.2.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-29 21:13:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "coreruleset",
"github_project": "secrules_parsing",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "secrules-parsing"
}
JSON
