# OWASP Appsec Discovery
OWASP Appsec Discovery cli tool scan provided code projects and extract structured protobuf, graphql, swaggers, database schemas, python, go and java object DTOs, used api clients and methods, and other kinds of external contracts. It scores risk level for found object fields with provided in config static keywords ruleset and store results in own format json or sarif reports for fast integration with exist vuln management systems like Defectdojo.
Cli tool can also use local LLM model Llama 3.2 3B from Huggingface and provided prompt to score objects without pre-existing knowledge about assets in code. Small open source models work fast on common hardware and are just enouth for such classification tasks.
Appsec Discovery service continuosly fetch changes from local Gitlab via api, clone code for particular projects, scan for objects in code and score them with provided via UI rules, store result objects with projects, branches and MRs from Gitlab in local db and alert about critical changes via messenger or comments to MR in Gitlab.
Under the hood tool powered by Semgrep OSS engine and specialy crafted discovery rules and parsers that extract particular objects from semgrep report meta variables.
## Cli mode
Install cli tool:
```bash
pip install appsec-discovery
```
Provided rules in conf.yaml or leave it empty for default list:
```yaml
score_tags:
pii:
high:
- 'first_name'
- 'last_name'
- 'phone'
- 'passport'
medium:
- 'address'
low:
- 'city'
finance:
high:
- 'pan'
- 'card_number'
medium:
- 'amount'
- 'balance'
auth:
high:
- 'password'
- 'pincode'
- 'codeword'
- 'token'
medium:
- 'login'
```
Run on code project folder with swaggers, protobuf and other structured contracts in code and get parsed objects and fields marked with severity and category tags:
```yaml
appsec-discovery --source tests/swagger_samples
- hash: 40140abef3b5f45d447d16e7180cc231
object_name: Route /user/login (GET)
object_type: route
parser: swagger
severity: high <<<<<<<<<<<<<<<<<<<<<<<< !!!
tags:
- auth <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
file: swagger.yaml
line: 1
properties:
path:
prop_name: path
prop_value: /user/login
severity: medium <<<<<<<<<<<<<<<<<< !!!
tags:
- auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
method:
prop_name: method
prop_value: GET
fields:
query.param.username:
field_name: query.param.username
field_type: string
file: swagger.yaml
line: 1
severity: medium <<<<<<<<<<<<<<<<<< !!!
tags:
- auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
query.param.password:
field_name: query.param.password
field_type: string
file: swagger.yaml
line: 1
severity: high <<<<<<<<<<<<<<<<<< !!!
tags:
- auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
output:
field_name: output
field_type: string
file: swagger.yaml
line: 1
...
- hash: 8a878eb2050c855faab96d2e52cc7cf8
object_name: Query Queries.promoterInfo
object_type: query
parser: graphql
severity: high <<<<<<<<<<<<<<<<<<<<<<<< !!!
tags:
- pii <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
file: query.graphql
line: 143
properties: {}
fields:
input.PromoterInfoInput.link:
field_name: input.PromoterInfoInput.link
field_type: String
file: query.graphql
line: 291
output.PromoterInfoPayload.firstName:
field_name: output.PromoterInfoPayload.firstName
field_type: String
file: query.graphql
line: 342
severity: high <<<<<<<<<<<<<<<<<< !!!
tags:
- pii <<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
output.PromoterInfoPayload.lastName:
field_name: output.PromoterInfoPayload.lastName
field_type: String
file: query.graphql
line: 365
severity: high
tags:
- pii <<<<<<<<<<<<<<<<<<<<<<<<<<< !!!
```
## Score object fields with local LLM model
Replace or combine exist static keyword ruleset with local LLM, fill conf.yaml with choosed LLM and prompt:
```yaml
ai_local:
model_folder: "/hf_models"
model_id: "Neurogen/Vikhr-Llama3.1-8B-Instruct-R-21-09-24-Q4_K_M-GGUF"
gguf_file: "vikhr-llama3.1-8b-instruct-r-21-09-24-q4_k_m.gguf"
system_prompt: "You are data security bot, for provided object and it field you must deside does it contain any personal, financial, authorization or other private data with special mesures to store and show."
```
Run scan with new settings and get objects and fields severity from local AI engine:
```yaml
appsec-discovery --source tests/swagger_samples --config tests/config_samples/ai_conf_vikhr_7b.yaml
- hash: 2e20a348a612aa28d24c1bd0498eebf0
object_name: Swagger route /user/login (GET)
object_type: route
parser: swagger
severity: medium <<<<<<<<<<<<<<<< !!!
tags:
- llm-pii <<<<<<<<<<<<<<<<<<<<<<< !!!
- llm-auth <<<<<<<<<<<<<<<<<<<<<< !!!
file: /swagger.yaml
line: 83
properties:
path:
prop_name: path
prop_value: /user/login
method:
prop_name: method
prop_value: get
fields:
...
Input.password:
field_name: Input.password
field_type: string
file: /swagger.yaml
line: 83
severity: medium <<<<<<<<<<<<<< !!!
tags:
- llm-auth <<<<<<<<<<<<<<<<<<<< !!!
...
```
At first run tool with download provided model from Huggingface into local cache dir, for next offline scans use this dir with pre downloaded models.
Play around with with various [models](https://huggingface.co/models?search=llama-3.2) from Huggingface and prompts for best results.
Also you can use external openai campatible LLM api with ai_api section of conf.yaml:
```yaml
ai_api:
base_url: "https://api.deepseek.com"
api_key: "some_api_key"
model: "deepseek-chat"
system_prompt: "You are data security bot, for provided object and it field you must deside does it contain any personal, financial, authorization or other private data with special mesures to store and show."
```
But remember that with great power comes great responsibility!
## Integrate scans into CI/CD
Run scan with sarif output format:
```bash
appsec-discovery --source tests/swagger_samples --config tests/config_samples/conf.yaml --output report.json --output-type sarif
```
Load result reports into vuln management system like Defectdojo:
## Service mode
Clone code to local folder:
```
git clone https://github.com/dmarushkin/appsec-discovery
cd appsec-discovery/appsec_discovery_service
```
Fillout .env file with your gitlab url and token, change passwords for local db and ui user, for alerts register new telegram bot or use exist one, or just leave TG args empty to only store objects:
```ini
POSTGRES_HOST=discovery_postgres
POSTGRES_DB=discovery_db
POSTGRES_USER=discovery_user
POSTGRES_PASSWORD=some_secret_str
GITLAB_PRIVATE_TOKEN=some_secret_str
GITLAB_URL=https://gitlab.examle.com
GITLAB_PROJECTS_PREFIX=backend/,frontend/,test/
UI_ADMIN_EMAIL=admin@example.com
UI_ADMIN_PASSWORD=admin
UI_JWT_KEY=some_secret_str
MAX_WORKERS=5
MR_ALERTS=1
TG_ALERT_TOKEN=test
TG_CHAT_ID=0000000000
```
Run service localy with docker compose:
```bash
docker-compose up --build
```
Service will continuosly fetch new projects and MRs for provided prefixes from Gitlab api, clone code and scan it for objects, score found ones and save into local postgres db for any analysis.
If sensitive fields in objects added on Merge requests service will alert via provided channel.
To ajust default rule list authorize in Rules Management UI at http://127.0.0.1/ and make some new rules or make exclude rules for false positives:
For now service does not provide any local UI for parsed and scored objects, so we recomend to use any kind of external analytic systems like Apache Superset, Grafana, Tableu etc.
For prod environments bake Docker images in your k8s env, use external db.
## Usage examples
- Appsec specialists can monitor codebase for critical changes and review them manualy, also sum scores for particular fields and get overall risk score for entire projects, and use it for prioritization of any kind of appsec rutines (triage vulns, plan security audits).
- Governance, Risk, and Compliance (GRC) specialists can use discovered data schemas for any kind of data governance (localize PII, payment and other critical data, dataflows), restricting access to and between critical services, focus on hardening environments that contain critical data.
- Monitoring or Incident Response specialists can focus attention on logs and anomalies in critical services or even particular routes in clients traffic.
- Infrastructure security specialists can use same approach to extract structured data about assets from IaC repositories like terraform or ansible (service now extracts VMs from terraform files).
Raw data
{
"_id": null,
"home_page": null,
"name": "appsec-discovery",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": "security, assets, discovery, scanning, appsec",
"author": "Dmitrii Mariushkin",
"author_email": "d.v.marushkin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/89/d7/3504efdfee1ba722ab47865783a00cc11de6b10551f4f744a87bf592daa5/appsec_discovery-0.7.2.tar.gz",
"platform": null,
"description": "# OWASP Appsec Discovery\n\nOWASP Appsec Discovery cli tool scan provided code projects and extract structured protobuf, graphql, swaggers, database schemas, python, go and java object DTOs, used api clients and methods, and other kinds of external contracts. It scores risk level for found object fields with provided in config static keywords ruleset and store results in own format json or sarif reports for fast integration with exist vuln management systems like Defectdojo.\n\nCli tool can also use local LLM model Llama 3.2 3B from Huggingface and provided prompt to score objects without pre-existing knowledge about assets in code. Small open source models work fast on common hardware and are just enouth for such classification tasks.\n\nAppsec Discovery service continuosly fetch changes from local Gitlab via api, clone code for particular projects, scan for objects in code and score them with provided via UI rules, store result objects with projects, branches and MRs from Gitlab in local db and alert about critical changes via messenger or comments to MR in Gitlab.\n\nUnder the hood tool powered by Semgrep OSS engine and specialy crafted discovery rules and parsers that extract particular objects from semgrep report meta variables.\n\n## Cli mode\n\nInstall cli tool:\n\n```bash\npip install appsec-discovery\n```\n\nProvided rules in conf.yaml or leave it empty for default list:\n\n```yaml\nscore_tags:\n pii:\n high:\n - 'first_name'\n - 'last_name'\n - 'phone'\n - 'passport'\n medium:\n - 'address'\n low:\n - 'city'\n finance:\n high:\n - 'pan'\n - 'card_number'\n medium:\n - 'amount'\n - 'balance'\n auth:\n high:\n - 'password'\n - 'pincode'\n - 'codeword'\n - 'token'\n medium:\n - 'login'\n```\n\nRun on code project folder with swaggers, protobuf and other structured contracts in code and get parsed objects and fields marked with severity and category tags:\n\n```yaml\nappsec-discovery --source tests/swagger_samples\n\n- hash: 40140abef3b5f45d447d16e7180cc231\n object_name: Route /user/login (GET)\n object_type: route\n parser: swagger\n severity: high <<<<<<<<<<<<<<<<<<<<<<<< !!!\n tags:\n - auth <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n file: swagger.yaml\n line: 1\n properties:\n path:\n prop_name: path\n prop_value: /user/login\n severity: medium <<<<<<<<<<<<<<<<<< !!!\n tags:\n - auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n method:\n prop_name: method\n prop_value: GET\n fields:\n query.param.username:\n field_name: query.param.username\n field_type: string\n file: swagger.yaml\n line: 1\n severity: medium <<<<<<<<<<<<<<<<<< !!!\n tags:\n - auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n query.param.password:\n field_name: query.param.password\n field_type: string\n file: swagger.yaml\n line: 1\n severity: high <<<<<<<<<<<<<<<<<< !!!\n tags:\n - auth <<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n output:\n field_name: output\n field_type: string\n file: swagger.yaml\n line: 1\n ...\n- hash: 8a878eb2050c855faab96d2e52cc7cf8\n object_name: Query Queries.promoterInfo\n object_type: query\n parser: graphql\n severity: high <<<<<<<<<<<<<<<<<<<<<<<< !!!\n tags:\n - pii <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n file: query.graphql\n line: 143\n properties: {}\n fields:\n input.PromoterInfoInput.link:\n field_name: input.PromoterInfoInput.link\n field_type: String\n file: query.graphql\n line: 291\n output.PromoterInfoPayload.firstName:\n field_name: output.PromoterInfoPayload.firstName\n field_type: String\n file: query.graphql\n line: 342\n severity: high <<<<<<<<<<<<<<<<<< !!!\n tags:\n - pii <<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n output.PromoterInfoPayload.lastName:\n field_name: output.PromoterInfoPayload.lastName\n field_type: String\n file: query.graphql\n line: 365\n severity: high\n tags:\n - pii <<<<<<<<<<<<<<<<<<<<<<<<<<< !!!\n```\n\n## Score object fields with local LLM model\n\nReplace or combine exist static keyword ruleset with local LLM, fill conf.yaml with choosed LLM and prompt:\n\n```yaml\nai_local:\n model_folder: \"/hf_models\"\n model_id: \"Neurogen/Vikhr-Llama3.1-8B-Instruct-R-21-09-24-Q4_K_M-GGUF\"\n gguf_file: \"vikhr-llama3.1-8b-instruct-r-21-09-24-q4_k_m.gguf\"\n system_prompt: \"You are data security bot, for provided object and it field you must deside does it contain any personal, financial, authorization or other private data with special mesures to store and show.\"\n```\n\nRun scan with new settings and get objects and fields severity from local AI engine:\n\n```yaml\nappsec-discovery --source tests/swagger_samples --config tests/config_samples/ai_conf_vikhr_7b.yaml\n\n- hash: 2e20a348a612aa28d24c1bd0498eebf0\n object_name: Swagger route /user/login (GET)\n object_type: route\n parser: swagger\n severity: medium <<<<<<<<<<<<<<<< !!!\n tags:\n - llm-pii <<<<<<<<<<<<<<<<<<<<<<< !!!\n - llm-auth <<<<<<<<<<<<<<<<<<<<<< !!!\n file: /swagger.yaml\n line: 83\n properties:\n path:\n prop_name: path\n prop_value: /user/login\n method:\n prop_name: method\n prop_value: get\n fields:\n ...\n Input.password:\n field_name: Input.password\n field_type: string\n file: /swagger.yaml\n line: 83\n severity: medium <<<<<<<<<<<<<< !!!\n tags:\n - llm-auth <<<<<<<<<<<<<<<<<<<< !!!\n ...\n```\n\nAt first run tool with download provided model from Huggingface into local cache dir, for next offline scans use this dir with pre downloaded models.\n\nPlay around with with various [models](https://huggingface.co/models?search=llama-3.2) from Huggingface and prompts for best results.\n\nAlso you can use external openai campatible LLM api with ai_api section of conf.yaml:\n\n```yaml\nai_api:\n base_url: \"https://api.deepseek.com\"\n api_key: \"some_api_key\"\n model: \"deepseek-chat\"\n system_prompt: \"You are data security bot, for provided object and it field you must deside does it contain any personal, financial, authorization or other private data with special mesures to store and show.\"\n```\n\nBut remember that with great power comes great responsibility!\n\n\n## Integrate scans into CI/CD\n\nRun scan with sarif output format: \n\n```bash\nappsec-discovery --source tests/swagger_samples --config tests/config_samples/conf.yaml --output report.json --output-type sarif\n```\n\nLoad result reports into vuln management system like Defectdojo:\n\n\n\n\n\n## Service mode\n\nClone code to local folder:\n\n```\ngit clone https://github.com/dmarushkin/appsec-discovery\ncd appsec-discovery/appsec_discovery_service\n```\n\nFillout .env file with your gitlab url and token, change passwords for local db and ui user, for alerts register new telegram bot or use exist one, or just leave TG args empty to only store objects:\n\n```ini\nPOSTGRES_HOST=discovery_postgres\nPOSTGRES_DB=discovery_db\nPOSTGRES_USER=discovery_user\nPOSTGRES_PASSWORD=some_secret_str\nGITLAB_PRIVATE_TOKEN=some_secret_str\nGITLAB_URL=https://gitlab.examle.com\nGITLAB_PROJECTS_PREFIX=backend/,frontend/,test/\nUI_ADMIN_EMAIL=admin@example.com\nUI_ADMIN_PASSWORD=admin\nUI_JWT_KEY=some_secret_str\nMAX_WORKERS=5\nMR_ALERTS=1\nTG_ALERT_TOKEN=test\nTG_CHAT_ID=0000000000\n```\n\nRun service localy with docker compose: \n\n```bash\ndocker-compose up --build\n```\n\nService will continuosly fetch new projects and MRs for provided prefixes from Gitlab api, clone code and scan it for objects, score found ones and save into local postgres db for any analysis.\n\nIf sensitive fields in objects added on Merge requests service will alert via provided channel.\n\nTo ajust default rule list authorize in Rules Management UI at http://127.0.0.1/ and make some new rules or make exclude rules for false positives:\n\n\n\nFor now service does not provide any local UI for parsed and scored objects, so we recomend to use any kind of external analytic systems like Apache Superset, Grafana, Tableu etc.\n\nFor prod environments bake Docker images in your k8s env, use external db.\n\n\n\n\n## Usage examples\n\n - Appsec specialists can monitor codebase for critical changes and review them manualy, also sum scores for particular fields and get overall risk score for entire projects, and use it for prioritization of any kind of appsec rutines (triage vulns, plan security audits).\n\n - Governance, Risk, and Compliance (GRC) specialists can use discovered data schemas for any kind of data governance (localize PII, payment and other critical data, dataflows), restricting access to and between critical services, focus on hardening environments that contain critical data.\n\n - Monitoring or Incident Response specialists can focus attention on logs and anomalies in critical services or even particular routes in clients traffic.\n\n - Infrastructure security specialists can use same approach to extract structured data about assets from IaC repositories like terraform or ansible (service now extracts VMs from terraform files).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Discover sensitive objects in project code",
"version": "0.7.2",
"project_urls": {
"Bug Tracker": "https://github.com/dmarushkin/appsec-discovery/issues",
"Homepage": "https://github.com/dmarushkin/appsec-discovery"
},
"split_keywords": [
"security",
" assets",
" discovery",
" scanning",
" appsec"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c510991aec347171c0d4918079dbbafc7f763f60922c85dc9cbee870dc592fc1",
"md5": "dcc47a3b6c543ee595cbc62aade8945c",
"sha256": "5620ba42d77978d17ba2e95abe628955c81872db32f27d0b628fe1f7aaf42751"
},
"downloads": -1,
"filename": "appsec_discovery-0.7.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dcc47a3b6c543ee595cbc62aade8945c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 30974,
"upload_time": "2025-01-28T20:32:36",
"upload_time_iso_8601": "2025-01-28T20:32:36.952231Z",
"url": "https://files.pythonhosted.org/packages/c5/10/991aec347171c0d4918079dbbafc7f763f60922c85dc9cbee870dc592fc1/appsec_discovery-0.7.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "89d73504efdfee1ba722ab47865783a00cc11de6b10551f4f744a87bf592daa5",
"md5": "8e611a35f26159f65c53ec59db560473",
"sha256": "ea403cd0b2a4f78e0864c37677473a5d098a3185cdb5c5b9661e84c02258e78d"
},
"downloads": -1,
"filename": "appsec_discovery-0.7.2.tar.gz",
"has_sig": false,
"md5_digest": "8e611a35f26159f65c53ec59db560473",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 21176,
"upload_time": "2025-01-28T20:32:38",
"upload_time_iso_8601": "2025-01-28T20:32:38.919809Z",
"url": "https://files.pythonhosted.org/packages/89/d7/3504efdfee1ba722ab47865783a00cc11de6b10551f4f744a87bf592daa5/appsec_discovery-0.7.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-28 20:32:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dmarushkin",
"github_project": "appsec-discovery",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "appsec-discovery"
}
JSON
