# API Validator
CLI Tool that validates an OpenAPI specification against a live application using [Newman](https://github.com/postmanlabs/newman). Optionally generates the OpenAPI spec from code using [NightVision](https://www.nightvision.net/).
## Installation
```bash
# pipx is recommended for installing CLI tools
pip install pipx
pipx install api-validator
# Or install it with pip
pip3 install api-validator --user
```
You'll also have to install some prerequisites that `api-validator` wraps around. Namely:
- [NightVision CLI](https://docs.nightvision.net/docs/installing-the-cli)
- [Newman](https://github.com/postmanlabs/newman)
- [postman-to-openapi](https://www.npmjs.com/package/postman-to-openapi)
- [oasdiff](https://github.com/Tufin/oasdiff)
- [GitHub CLI](https://cli.github.com/manual/)
- [Semgrep](https://github.com/semgrep/semgrep) (currently required for API discovery on Java only; migrating soon)
* You can do that quickly by running this command:
```bash
# Install some prerequisites
api-validator install
```
If that doesn't work, you can run these:
```bash
brew install tufin/tufin/oasdiff
brew install gh
brew install nvsecurity/taps/nightvision
brew install newman # or npm install -g newman
brew install semgrep
npm install postman-to-openapi -g
```
* We manage the config for this tool in a config file. We've bundled it into the distribution. You can create the config file like this:
```bash
api-validator create-config-file -o config.yml
```
* Now you can list the various jobs that are available:
```bash
# Note: api-validator will look for the config file in the current directory. You can specify a different location with the -c flag.
# Example: api-validator list-jobs -c custom-config.yml
# List all jobs
api-validator list-jobs -c config.yml
# Only list Python jobs
api-validator list-jobs --language python
```
It will print out the available jobs in the config file, like this:
<details open>
<summary>Jobs listing</summary>
<br>
```
Language: dotnet, Job Name: altinn-studio, Repo: https://github.com/Altinn/altinn-studio
Language: dotnet, Job Name: bitwarder-server, Repo: https://github.com/bitwarden/server
Language: dotnet, Job Name: dotnet-kavita, Repo: https://github.com/Kareadita/Kavita
Language: dotnet, Job Name: dvcsharp-api, Repo: https://github.com/appsecco/dvcsharp-api
Language: dotnet, Job Name: edwinvw-pitstop-customers, Repo: https://github.com/EdwinVW/pitstop
Language: dotnet, Job Name: edwinvw-pitstop-vehicles, Repo: https://github.com/EdwinVW/pitstop
Language: dotnet, Job Name: edwinvw-pitstop-workshop, Repo: https://github.com/EdwinVW/pitstop
Language: dotnet, Job Name: eshop-catalog-api, Repo: https://github.com/api-extraction-examples/eShop
Language: dotnet, Job Name: eshop-ordering-api, Repo: https://github.com/api-extraction-examples/eShop
Language: dotnet, Job Name: eshop-webhooks-api, Repo: https://github.com/api-extraction-examples/eShop
Language: dotnet, Job Name: featbit, Repo: https://github.com/featbit/featbit
Language: dotnet, Job Name: jellyfin, Repo: https://github.com/NightVisionExamples/jellyfin
Language: dotnet, Job Name: universalis, Repo: https://github.com/Universalis-FFXIV/Universalis
Language: dotnet, Job Name: wallet-wasabi, Repo: https://github.com/zkSNACKs/WalletWasabi
Language: go, Job Name: crAPI-go, Repo: https://github.com/vulnerable-apps/crAPI
Language: js, Job Name: blockchain-explorer, Repo: https://github.com/api-extraction-examples/blockchain-explorer
Language: js, Job Name: cve-services, Repo: https://github.com/api-extraction-examples/cve-services
Language: js, Job Name: dvws-node, Repo: https://github.com/vulnerable-apps/dvws-node
Language: js, Job Name: express-anything-llm, Repo: https://github.com/api-extraction-examples/anything-llm
Language: js, Job Name: express-rest-boilerplate, Repo: https://github.com/dnighvn/express-rest-boilerplate
Language: js, Job Name: hypertube, Repo: https://github.com/api-extraction-examples/Hypertube
Language: js, Job Name: infisicial, Repo: https://github.com/api-extraction-examples/infisical
Language: js, Job Name: juice-shop, Repo: https://github.com/vulnerable-apps/juice-shop
Language: js, Job Name: kubero, Repo: https://github.com/api-extraction-examples/kubero
Language: js, Job Name: nodejs-api-showcase, Repo: https://github.com/api-extraction-examples/nodejs-api-showcase
Language: js, Job Name: nodejs-goof, Repo: https://github.com/vulnerable-apps/nodejs-goof
Language: js, Job Name: valetudo, Repo: https://github.com/api-extraction-examples/Valetudo
Language: python, Job Name: Inventree-django, Repo: https://github.com/api-extraction-examples/InvenTree
Language: python, Job Name: a-flaskrestful-api, Repo: https://github.com/api-extraction-examples/a-flaskrestful-api
Language: python, Job Name: argus-eye-django, Repo: https://github.com/api-extraction-examples/Eye
Language: python, Job Name: cert-viewer-flask, Repo: https://github.com/blockchain-certificates/cert-viewer
Language: python, Job Name: cpa-network-django, Repo: https://github.com/api-extraction-examples/cpa-network
Language: python, Job Name: crAPI-python, Repo: https://github.com/vulnerable-apps/crAPI
Language: python, Job Name: defect-dojo-django, Repo: https://github.com/api-extraction-examples/django-DefectDojo
Language: python, Job Name: django-crm, Repo: https://github.com/api-extraction-examples/Django-CRM
Language: python, Job Name: greater-wms-django, Repo: https://github.com/api-extraction-examples/GreaterWMS
Language: python, Job Name: help-desk-service-django, Repo: https://github.com/api-extraction-examples/help-desk-service
Language: python, Job Name: intelowl-django, Repo: https://github.com/api-extraction-examples/IntelOwl
Language: python, Job Name: karrio-django, Repo: https://github.com/api-extraction-examples/karrio
Language: python, Job Name: librephotos-django, Repo: https://github.com/api-extraction-examples/librephotos
Language: python, Job Name: libretime-django, Repo: https://github.com/api-extraction-examples/libretime
Language: python, Job Name: mathesar-django, Repo: https://github.com/api-extraction-examples/mathesar
Language: python, Job Name: medileaf-backend, Repo: https://github.com/api-extraction-examples/MediLeaf_backend
Language: python, Job Name: netbox-django, Repo: https://github.com/api-extraction-examples/netbox
Language: python, Job Name: nimbler-django, Repo: https://github.com/NimblerSecurity/nimbler-django
Language: python, Job Name: posthog-django, Repo: https://github.com/api-extraction-examples/posthog
Language: python, Job Name: wger-django, Repo: https://github.com/api-extraction-examples/wger
Language: spring, Job Name: Alibaba-Nacos, Repo: https://github.com/api-extraction-examples/nacos
Language: spring, Job Name: Angular-SpringBoot-REST-JWT, Repo: https://github.com/mrin9/Angular-SpringBoot-REST-JWT
Language: spring, Job Name: Netflix-Conductor, Repo: https://github.com/api-extraction-examples/conductor
Language: spring, Job Name: Newbee-Mall, Repo: https://github.com/api-extraction-examples/newbee-mall
Language: spring, Job Name: Spring-boot-Banking, Repo: https://github.com/api-extraction-examples/Spring-boot-Banking
Language: spring, Job Name: ZHENFENG13-My-Blog, Repo: https://github.com/api-extraction-examples/ZHENFENG13-My-Blog
Language: spring, Job Name: apereo-cas, Repo: https://github.com/api-extraction-examples/cas
Language: spring, Job Name: crAPI-spring, Repo: https://github.com/vulnerable-apps/crAPI
Language: spring, Job Name: javaspringvulny, Repo: https://github.com/vulnerable-apps/javaspringvulny
Language: spring, Job Name: thingsboard, Repo: https://github.com/api-extraction-examples/thingsboard
```
</details>
### Mode 1: Comparing API Discovery to existing OpenAPI Specs
You can compare NightVision's API Discovery results to existing OpenAPI specs, based on the content of the config file. This is useful for grading the effectiveness of the API Discovery tool.
You can run these comparisons at different scopes:
1. Select job by job name
2. Bulk select jobs, filtered by language
3. Bulk select all jobs
* Run a comparison for a single job:
```bash
api-validator compare \
--config-file config.yml \
--job juice-shop \
--output-file juice-shop.md
```
* Run a comparison for a single job:
```bash
api-validator compare \
--config-file config.yml \
--job juice-shop \
--output-file juice-shop.md
```
The output will look like this:
<details open>
<summary>Juice Shop output</summary>
```
Thread 0 will process cloning for jobs: juice-shop
juice-shop/juice-shop: Cloning...
juice-shop/juice-shop: Local repo already exists. Skipping clone.
juice-shop/juice-shop: Thread 0 progress: Repository cloned for: juice-shop
Thread 0 will process extraction for jobs: juice-shop
juice-shop/juice-shop: Working on Job: juice-shop
juice-shop/juice-shop: Repo: https://github.com/juice-shop/juice-shop, Swagger File: https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml, Language: js
juice-shop/juice-shop: Downloading base Swagger file...
juice-shop/juice-shop: Data downloaded from https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml and saved as /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/base/juice-shop.yml
juice-shop/juice-shop: Running extraction...
juice-shop/juice-shop: Running command: api-excavator --log-level info --output /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/revision/juice-shop.yml -l js /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/repos/juice-shop
juice-shop/juice-shop: INFO Initializing language provider
juice-shop/juice-shop: INFO Finished initializing language provider
juice-shop/juice-shop: INFO Starting language provider execution
juice-shop/juice-shop: INFO Finished language provider execution
juice-shop/juice-shop: INFO Starting generating OpenAPI document
juice-shop/juice-shop: INFO OpenAPI document generated in 653.800208ms
juice-shop/juice-shop: Number of discovered paths: 87
juice-shop/juice-shop: Number of discovered classes: 0
juice-shop/juice-shop: INFO Generated the OpenAPI document.
juice-shop/juice-shop: INFO Successfully validated the output.
juice-shop/juice-shop: Performing OASDiff operation...
Running oasdiff command:
oasdiff diff /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/base/juice-shop.yml /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/revision/juice-shop.yml --exclude-elements description,examples,title,summary
juice-shop/juice-shop: Completed work on Job: juice-shop
juice-shop/juice-shop: Thread 0 progress: Completed: juice-shop
Thread 0 final status: Completed: juice-shop
All threads completed.
Saved comparison-juice-shop.md
```
<br>
</details>
* Run a comparison for Python apps:
```bash
api-validator compare \
--config-file config.yml \
--language python \
--output-file comparison-python.md
```
* Run a comparison for all jobs:
```bash
api-validator compare \
--config-file config.yml \
--all \
--output-file comparison-all.md
```
## Mode 2: Validating OpenAPI specs based on traffic
We took another route to validating our OpenAPI specs - testing the OpenAPI spec against live applications.
We use [Newman](https://github.com/postmanlabs/newman) to confirm that each of the URL paths and HTTP methods are valid. It's not a perfect evaluation system, but it's a solid data point of the effectiveness.
Assumptions:
* **Valid endpoints**: For instance, if the endpoint responds with HTTP 200, we assume it is valid. Also, if it responds with `401 Unauthorized` or `403 Forbidden`, we assume it is valid because it is a valid endpoint, but we do not have access to it. With `400 Bad Request`, we assume it is valid because the request body was not formatted correctly. We do not attempt to format the request body to match the API spec.
* **Invalid endpoints**: If the endpoint responds with HTTP `404 Not Found`, we assume it is invalid. If it responds with `405 Method Not Allowed`, we treated as invalid because the generated API spec was incorrect about that method. For `500 Internal Server Error`, this might be a valid endpoint but there is not a way to tell for sure without testing it manually.
In the case of Juice Shop, you can see that out of the **97** endpoints were discovered, of which **79** were valid and **18** were invalid/hallucinated. This gives us a success rate of **74.53%**.
Here's an example of how to run the traffic validation:
* Now run an example app with Docker:
```bash
docker run --restart always -d -p 3000:3000 --name juice-shop bkimminich/juice-shop
```
* Now run the validator to test the API and generate a markdown-formatted report:
```bash
api-validator yolo-traffic \
--config-file config.yml \
--swagger-file juice-shop.yml \
--server http://localhost:3000 \
--app-name juice-shop
```
It will generate a file called `./summary.md`.
## Config File
The config file is a YAML file that contains details about the applications you are scanning. For example, you might want to skip certain endpoints that are destructive or that you don't want to test. You should also specify the repository URL and language of the application; that information is used in the generated Markdown report, but it's not the end of the world if you don't include it.
Here is an example:
```yaml
apps:
nodejs-goof:
repo: 'https://github.com/vulnerable-apps/nodejs-goof'
language: js
github_stars: 485
provided_swagger_file: ""
skip_endpoints:
- path: '/destroy/:id'
method: GET
description: Destroy an endpoint
juice-shop:
repo: 'https://github.com/vulnerable-apps/juice-shop'
language: js
provided_swagger_file: "https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml"
github_stars: 8900
skip_endpoints:
- path: '/file-upload'
method: POST
description: Upload a file
- path: '/profile/image/file'
method: POST
description: Upload a file
```
We've provided the default config file in the distribution, which you can generate with `api-validator create-config-file -o config.yml`.
Raw data
{
"_id": null,
"home_page": "https://github.com/nvsecurity/api-validator",
"name": "api-validator",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "openapi",
"author": "Kinnaird McQuade",
"author_email": "kinnaird@nightvision.net",
"download_url": "https://files.pythonhosted.org/packages/1b/dc/2c8b94c90185ed96f8307486483a13ae128be630d23510958bee50fb4e12/api_validator-0.3.15.tar.gz",
"platform": null,
"description": "\n# API Validator\n\nCLI Tool that validates an OpenAPI specification against a live application using [Newman](https://github.com/postmanlabs/newman). Optionally generates the OpenAPI spec from code using [NightVision](https://www.nightvision.net/).\n\n## Installation\n\n```bash\n# pipx is recommended for installing CLI tools\npip install pipx\npipx install api-validator\n\n# Or install it with pip\npip3 install api-validator --user\n```\n\nYou'll also have to install some prerequisites that `api-validator` wraps around. Namely:\n- [NightVision CLI](https://docs.nightvision.net/docs/installing-the-cli)\n- [Newman](https://github.com/postmanlabs/newman)\n- [postman-to-openapi](https://www.npmjs.com/package/postman-to-openapi)\n- [oasdiff](https://github.com/Tufin/oasdiff)\n- [GitHub CLI](https://cli.github.com/manual/)\n- [Semgrep](https://github.com/semgrep/semgrep) (currently required for API discovery on Java only; migrating soon)\n\n* You can do that quickly by running this command:\n\n```bash\n# Install some prerequisites \napi-validator install\n```\n\nIf that doesn't work, you can run these:\n\n```bash\nbrew install tufin/tufin/oasdiff\nbrew install gh\nbrew install nvsecurity/taps/nightvision\nbrew install newman # or npm install -g newman\nbrew install semgrep\n\nnpm install postman-to-openapi -g\n```\n\n* We manage the config for this tool in a config file. We've bundled it into the distribution. You can create the config file like this:\n\n```bash\napi-validator create-config-file -o config.yml\n```\n\n* Now you can list the various jobs that are available:\n\n```bash\n# Note: api-validator will look for the config file in the current directory. You can specify a different location with the -c flag.\n# Example: api-validator list-jobs -c custom-config.yml\n\n# List all jobs\napi-validator list-jobs -c config.yml\n# Only list Python jobs\napi-validator list-jobs --language python\n```\n\n\nIt will print out the available jobs in the config file, like this:\n\n<details open>\n<summary>Jobs listing</summary>\n<br>\n\n```\nLanguage: dotnet, Job Name: altinn-studio, Repo: https://github.com/Altinn/altinn-studio\nLanguage: dotnet, Job Name: bitwarder-server, Repo: https://github.com/bitwarden/server\nLanguage: dotnet, Job Name: dotnet-kavita, Repo: https://github.com/Kareadita/Kavita\nLanguage: dotnet, Job Name: dvcsharp-api, Repo: https://github.com/appsecco/dvcsharp-api\nLanguage: dotnet, Job Name: edwinvw-pitstop-customers, Repo: https://github.com/EdwinVW/pitstop\nLanguage: dotnet, Job Name: edwinvw-pitstop-vehicles, Repo: https://github.com/EdwinVW/pitstop\nLanguage: dotnet, Job Name: edwinvw-pitstop-workshop, Repo: https://github.com/EdwinVW/pitstop\nLanguage: dotnet, Job Name: eshop-catalog-api, Repo: https://github.com/api-extraction-examples/eShop\nLanguage: dotnet, Job Name: eshop-ordering-api, Repo: https://github.com/api-extraction-examples/eShop\nLanguage: dotnet, Job Name: eshop-webhooks-api, Repo: https://github.com/api-extraction-examples/eShop\nLanguage: dotnet, Job Name: featbit, Repo: https://github.com/featbit/featbit\nLanguage: dotnet, Job Name: jellyfin, Repo: https://github.com/NightVisionExamples/jellyfin\nLanguage: dotnet, Job Name: universalis, Repo: https://github.com/Universalis-FFXIV/Universalis\nLanguage: dotnet, Job Name: wallet-wasabi, Repo: https://github.com/zkSNACKs/WalletWasabi\nLanguage: go, Job Name: crAPI-go, Repo: https://github.com/vulnerable-apps/crAPI\nLanguage: js, Job Name: blockchain-explorer, Repo: https://github.com/api-extraction-examples/blockchain-explorer\nLanguage: js, Job Name: cve-services, Repo: https://github.com/api-extraction-examples/cve-services\nLanguage: js, Job Name: dvws-node, Repo: https://github.com/vulnerable-apps/dvws-node\nLanguage: js, Job Name: express-anything-llm, Repo: https://github.com/api-extraction-examples/anything-llm\nLanguage: js, Job Name: express-rest-boilerplate, Repo: https://github.com/dnighvn/express-rest-boilerplate\nLanguage: js, Job Name: hypertube, Repo: https://github.com/api-extraction-examples/Hypertube\nLanguage: js, Job Name: infisicial, Repo: https://github.com/api-extraction-examples/infisical\nLanguage: js, Job Name: juice-shop, Repo: https://github.com/vulnerable-apps/juice-shop\nLanguage: js, Job Name: kubero, Repo: https://github.com/api-extraction-examples/kubero\nLanguage: js, Job Name: nodejs-api-showcase, Repo: https://github.com/api-extraction-examples/nodejs-api-showcase\nLanguage: js, Job Name: nodejs-goof, Repo: https://github.com/vulnerable-apps/nodejs-goof\nLanguage: js, Job Name: valetudo, Repo: https://github.com/api-extraction-examples/Valetudo\nLanguage: python, Job Name: Inventree-django, Repo: https://github.com/api-extraction-examples/InvenTree\nLanguage: python, Job Name: a-flaskrestful-api, Repo: https://github.com/api-extraction-examples/a-flaskrestful-api\nLanguage: python, Job Name: argus-eye-django, Repo: https://github.com/api-extraction-examples/Eye\nLanguage: python, Job Name: cert-viewer-flask, Repo: https://github.com/blockchain-certificates/cert-viewer\nLanguage: python, Job Name: cpa-network-django, Repo: https://github.com/api-extraction-examples/cpa-network\nLanguage: python, Job Name: crAPI-python, Repo: https://github.com/vulnerable-apps/crAPI\nLanguage: python, Job Name: defect-dojo-django, Repo: https://github.com/api-extraction-examples/django-DefectDojo\nLanguage: python, Job Name: django-crm, Repo: https://github.com/api-extraction-examples/Django-CRM\nLanguage: python, Job Name: greater-wms-django, Repo: https://github.com/api-extraction-examples/GreaterWMS\nLanguage: python, Job Name: help-desk-service-django, Repo: https://github.com/api-extraction-examples/help-desk-service\nLanguage: python, Job Name: intelowl-django, Repo: https://github.com/api-extraction-examples/IntelOwl\nLanguage: python, Job Name: karrio-django, Repo: https://github.com/api-extraction-examples/karrio\nLanguage: python, Job Name: librephotos-django, Repo: https://github.com/api-extraction-examples/librephotos\nLanguage: python, Job Name: libretime-django, Repo: https://github.com/api-extraction-examples/libretime\nLanguage: python, Job Name: mathesar-django, Repo: https://github.com/api-extraction-examples/mathesar\nLanguage: python, Job Name: medileaf-backend, Repo: https://github.com/api-extraction-examples/MediLeaf_backend\nLanguage: python, Job Name: netbox-django, Repo: https://github.com/api-extraction-examples/netbox\nLanguage: python, Job Name: nimbler-django, Repo: https://github.com/NimblerSecurity/nimbler-django\nLanguage: python, Job Name: posthog-django, Repo: https://github.com/api-extraction-examples/posthog\nLanguage: python, Job Name: wger-django, Repo: https://github.com/api-extraction-examples/wger\nLanguage: spring, Job Name: Alibaba-Nacos, Repo: https://github.com/api-extraction-examples/nacos\nLanguage: spring, Job Name: Angular-SpringBoot-REST-JWT, Repo: https://github.com/mrin9/Angular-SpringBoot-REST-JWT\nLanguage: spring, Job Name: Netflix-Conductor, Repo: https://github.com/api-extraction-examples/conductor\nLanguage: spring, Job Name: Newbee-Mall, Repo: https://github.com/api-extraction-examples/newbee-mall\nLanguage: spring, Job Name: Spring-boot-Banking, Repo: https://github.com/api-extraction-examples/Spring-boot-Banking\nLanguage: spring, Job Name: ZHENFENG13-My-Blog, Repo: https://github.com/api-extraction-examples/ZHENFENG13-My-Blog\nLanguage: spring, Job Name: apereo-cas, Repo: https://github.com/api-extraction-examples/cas\nLanguage: spring, Job Name: crAPI-spring, Repo: https://github.com/vulnerable-apps/crAPI\nLanguage: spring, Job Name: javaspringvulny, Repo: https://github.com/vulnerable-apps/javaspringvulny\nLanguage: spring, Job Name: thingsboard, Repo: https://github.com/api-extraction-examples/thingsboard\n```\n\n</details>\n\n\n### Mode 1: Comparing API Discovery to existing OpenAPI Specs\n\nYou can compare NightVision's API Discovery results to existing OpenAPI specs, based on the content of the config file. This is useful for grading the effectiveness of the API Discovery tool.\n\nYou can run these comparisons at different scopes:\n1. Select job by job name\n2. Bulk select jobs, filtered by language\n3. Bulk select all jobs\n\n* Run a comparison for a single job:\n\n```bash\napi-validator compare \\\n --config-file config.yml \\\n --job juice-shop \\\n --output-file juice-shop.md\n```\n\n* Run a comparison for a single job:\n\n```bash\napi-validator compare \\\n --config-file config.yml \\\n --job juice-shop \\\n --output-file juice-shop.md\n```\n\nThe output will look like this:\n\n<details open>\n<summary>Juice Shop output</summary>\n\n```\nThread 0 will process cloning for jobs: juice-shop\n\tjuice-shop/juice-shop: Cloning...\n\tjuice-shop/juice-shop: Local repo already exists. Skipping clone.\njuice-shop/juice-shop: Thread 0 progress: Repository cloned for: juice-shop\nThread 0 will process extraction for jobs: juice-shop\n\tjuice-shop/juice-shop: Working on Job: juice-shop\n\tjuice-shop/juice-shop: Repo: https://github.com/juice-shop/juice-shop, Swagger File: https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml, Language: js\n\tjuice-shop/juice-shop: Downloading base Swagger file...\n\tjuice-shop/juice-shop: Data downloaded from https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml and saved as /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/base/juice-shop.yml\n\tjuice-shop/juice-shop: Running extraction...\n\t\tjuice-shop/juice-shop: Running command: api-excavator --log-level info --output /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/revision/juice-shop.yml -l js /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/repos/juice-shop\n\t\tjuice-shop/juice-shop: INFO Initializing language provider\n\t\tjuice-shop/juice-shop: INFO Finished initializing language provider\n\t\tjuice-shop/juice-shop: INFO Starting language provider execution\n\t\tjuice-shop/juice-shop: INFO Finished language provider execution\n\t\tjuice-shop/juice-shop: INFO Starting generating OpenAPI document\n\t\tjuice-shop/juice-shop: INFO OpenAPI document generated in 653.800208ms\n\t\tjuice-shop/juice-shop: Number of discovered paths: 87\n\t\tjuice-shop/juice-shop: Number of discovered classes: 0\n\t\tjuice-shop/juice-shop: INFO Generated the OpenAPI document.\n\t\tjuice-shop/juice-shop: INFO Successfully validated the output.\n\tjuice-shop/juice-shop: Performing OASDiff operation...\n\tRunning oasdiff command:\n\t\toasdiff diff /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/base/juice-shop.yml /Users/kinnaird/github.com/nvsecurity/api-validator/analysis/revision/juice-shop.yml --exclude-elements description,examples,title,summary\n\tjuice-shop/juice-shop: Completed work on Job: juice-shop\n\njuice-shop/juice-shop: Thread 0 progress: Completed: juice-shop\nThread 0 final status: Completed: juice-shop\nAll threads completed.\nSaved comparison-juice-shop.md\n```\n\n<br>\n\n</details>\n\n* Run a comparison for Python apps:\n\n```bash\napi-validator compare \\\n --config-file config.yml \\\n --language python \\\n --output-file comparison-python.md\n```\n\n* Run a comparison for all jobs:\n\n```bash\napi-validator compare \\\n --config-file config.yml \\\n --all \\\n --output-file comparison-all.md\n```\n\n## Mode 2: Validating OpenAPI specs based on traffic\n\nWe took another route to validating our OpenAPI specs - testing the OpenAPI spec against live applications.\n\nWe use [Newman](https://github.com/postmanlabs/newman) to confirm that each of the URL paths and HTTP methods are valid. It's not a perfect evaluation system, but it's a solid data point of the effectiveness.\n\nAssumptions:\n* **Valid endpoints**: For instance, if the endpoint responds with HTTP 200, we assume it is valid. Also, if it responds with `401 Unauthorized` or `403 Forbidden`, we assume it is valid because it is a valid endpoint, but we do not have access to it. With `400 Bad Request`, we assume it is valid because the request body was not formatted correctly. We do not attempt to format the request body to match the API spec.\n* **Invalid endpoints**: If the endpoint responds with HTTP `404 Not Found`, we assume it is invalid. If it responds with `405 Method Not Allowed`, we treated as invalid because the generated API spec was incorrect about that method. For `500 Internal Server Error`, this might be a valid endpoint but there is not a way to tell for sure without testing it manually.\n\nIn the case of Juice Shop, you can see that out of the **97** endpoints were discovered, of which **79** were valid and **18** were invalid/hallucinated. This gives us a success rate of **74.53%**.\n\nHere's an example of how to run the traffic validation:\n\n* Now run an example app with Docker:\n\n```bash\ndocker run --restart always -d -p 3000:3000 --name juice-shop bkimminich/juice-shop\n```\n\n* Now run the validator to test the API and generate a markdown-formatted report:\n\n```bash\napi-validator yolo-traffic \\\n --config-file config.yml \\\n --swagger-file juice-shop.yml \\\n --server http://localhost:3000 \\\n --app-name juice-shop\n```\n\nIt will generate a file called `./summary.md`. \n\n## Config File\n\nThe config file is a YAML file that contains details about the applications you are scanning. For example, you might want to skip certain endpoints that are destructive or that you don't want to test. You should also specify the repository URL and language of the application; that information is used in the generated Markdown report, but it's not the end of the world if you don't include it.\n\nHere is an example:\n\n```yaml\napps:\n nodejs-goof:\n repo: 'https://github.com/vulnerable-apps/nodejs-goof'\n language: js\n github_stars: 485\n provided_swagger_file: \"\"\n skip_endpoints:\n - path: '/destroy/:id'\n method: GET\n description: Destroy an endpoint\n juice-shop:\n repo: 'https://github.com/vulnerable-apps/juice-shop'\n language: js\n provided_swagger_file: \"https://raw.githubusercontent.com/api-extraction-examples/juice-shop/master/swagger.yml\"\n github_stars: 8900\n skip_endpoints:\n - path: '/file-upload'\n method: POST\n description: Upload a file\n - path: '/profile/image/file'\n method: POST\n description: Upload a file\n```\n\nWe've provided the default config file in the distribution, which you can generate with `api-validator create-config-file -o config.yml`. \n",
"bugtrack_url": null,
"license": null,
"summary": "Validate OpenAPI specs by sending traffic",
"version": "0.3.15",
"project_urls": {
"Homepage": "https://github.com/nvsecurity/api-validator"
},
"split_keywords": [
"openapi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9d78da215c9296c4d0abe7afaa51b474c85657049c0c541bab3abc0ddf1ac73f",
"md5": "853dc319b8a69220e80eae876f166a86",
"sha256": "a88e5a43767492e48b2c0ae09506a2fca13909f9ff62702d5cef8b4f28e2e9d2"
},
"downloads": -1,
"filename": "api_validator-0.3.15-py3-none-any.whl",
"has_sig": false,
"md5_digest": "853dc319b8a69220e80eae876f166a86",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 97995,
"upload_time": "2024-10-01T11:33:23",
"upload_time_iso_8601": "2024-10-01T11:33:23.386970Z",
"url": "https://files.pythonhosted.org/packages/9d/78/da215c9296c4d0abe7afaa51b474c85657049c0c541bab3abc0ddf1ac73f/api_validator-0.3.15-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1bdc2c8b94c90185ed96f8307486483a13ae128be630d23510958bee50fb4e12",
"md5": "2e7ae0210c3ff68698cc51d80a2bb155",
"sha256": "2a9affcacaea0f6dc2c3106edecf112f15dc7c6e582f01db4d8977ac3140ce9d"
},
"downloads": -1,
"filename": "api_validator-0.3.15.tar.gz",
"has_sig": false,
"md5_digest": "2e7ae0210c3ff68698cc51d80a2bb155",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 72774,
"upload_time": "2024-10-01T11:33:24",
"upload_time_iso_8601": "2024-10-01T11:33:24.889043Z",
"url": "https://files.pythonhosted.org/packages/1b/dc/2c8b94c90185ed96f8307486483a13ae128be630d23510958bee50fb4e12/api_validator-0.3.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-01 11:33:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nvsecurity",
"github_project": "api-validator",
"github_not_found": true,
"lcname": "api-validator"
}
JSON
