<pre style="color: lime; background-color: black;">
███████╗██╗ ██╗██████╗ ██╗ ██╗██╗███████╗
██╔════╝██║ ██║██╔══██╗ ██║ ██║██║╚══███╔╝
███████╗██║ ██║██████╔╝ ██║ █╗ ██║██║ ███╔╝
╚════██║██║ ██║██╔══██╗ ██║███╗██║██║ ███╔╝
███████║╚██████╔╝██████╔╝ ╚███╔███╔╝██║███████╗
╚══════╝ ╚═════╝ ╚═════╝ ╚══╝╚══╝ ╚═╝╚══════╝
</pre>
A recon tool that uses AI to predict subdomains. Then returns those that resolve.
### Installation
```pip install subwiz```
### Recommended Use
Use [subfinder](https://github.com/projectdiscovery/subfinder) ❤️ to find subdomains from passive sources:
```subfinder -d example.com -o subdomains.txt```
Seed subwiz with these subdomains:
```subwiz -i subdomains.txt```
### Supported Switches
```commandline
usage: cli.py [-h] -i INPUT_FILE [-o OUTPUT_FILE] [-n NUM_PREDICTIONS]
[--no-resolve] [--force-download] [-t TEMPERATURE]
[-d {auto,cpu,cuda,mps}] [-q MAX_NEW_TOKENS]
[--resolution_concurrency RESOLUTION_LIM]
options:
-h, --help show this help message and exit
-i INPUT_FILE, --input-file INPUT_FILE
file containing new-line-separated subdomains.
(default: None)
-o OUTPUT_FILE, --output-file OUTPUT_FILE
output file to write new-line separated subdomains to.
(default: None)
-n NUM_PREDICTIONS, --num_predictions NUM_PREDICTIONS
number of subdomains to predict. (default: 500)
--no-resolve do not resolve the output subdomains. (default: False)
--force-download download model and tokenizer files, even if cached.
(default: False)
-t TEMPERATURE, --temperature TEMPERATURE
add randomness to the model, recommended ≤ 0.3)
(default: 0.0)
-d {auto,cpu,cuda,mps}, --device {auto,cpu,cuda,mps}
hardware to run the transformer model on. (default:
auto)
-q MAX_NEW_TOKENS, --max_new_tokens MAX_NEW_TOKENS
maximum length of predicted subdomains in tokens.
(default: 10)
--resolution_concurrency RESOLUTION_LIM
number of concurrent resolutions. (default: 128)
```
### In Python
Use subwiz in Python, with the same inputs as the command line interface.
```
import subwiz
known_subdomains = ['test1.example.com', 'test2.example.com']
new_subdomains = subwiz.run(input_domains=known_subdomains)
```
---
### Model
Use the `--no-resolve` flag to inspect model outputs without checking if they resolve.
#### Architecture
Subwiz is a ultra-lightweight transformer model based on [nanoGPT](https://github.com/karpathy/nanoGPT/tree/master) ❤️:
- 17.3M parameters.
- Trained on 26M tokens, lists of subdomains from passive sources.
- Tokenizer trained on same lists of subdomains (8192 tokens).
#### Hugging Face
The model is saved in Hugging Face as [HadrianSecurity/subwiz](https://huggingface.co/HadrianSecurity/subwiz).
It is downloaded when you first run subwiz.
#### Inference
Typically, generative transformer models (e.g. ChatGPT) predict a single output sequence.
Subwiz predicts the N most likely sequences using a beam search algorithm.
*Beam algorithm to predict the N most likely outputs from a generative transformer model.*
Raw data
{
"_id": null,
"home_page": null,
"name": "subwiz",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.9",
"maintainer_email": null,
"keywords": "machine learning, recon, subdomains, transformers",
"author": null,
"author_email": "Klaas Meinke <klaas@hadrian.io>",
"download_url": "https://files.pythonhosted.org/packages/f6/69/d02cf1af85c358096fba6983896fb7e9f40b07edccf9b64bc359ff50aad4/subwiz-0.1.5.tar.gz",
"platform": null,
"description": "<pre style=\"color: lime; background-color: black;\">\n\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2557 \u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2557 \u2588\u2588\u2557\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\n\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2551\u255a\u2550\u2550\u2588\u2588\u2588\u2554\u255d\n\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d \u2588\u2588\u2551 \u2588\u2557 \u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2588\u2554\u255d \n\u255a\u2550\u2550\u2550\u2550\u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557 \u2588\u2588\u2551\u2588\u2588\u2588\u2557\u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2588\u2554\u255d \n\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2551\u255a\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d \u255a\u2588\u2588\u2588\u2554\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\n\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u255d \u255a\u2550\u2550\u255d\u255a\u2550\u2550\u255d \u255a\u2550\u255d\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\n</pre>\n\nA recon tool that uses AI to predict subdomains. Then returns those that resolve.\n\n### Installation\n\n```pip install subwiz```\n\n### Recommended Use\n\nUse [subfinder](https://github.com/projectdiscovery/subfinder) \u2764\ufe0f to find subdomains from passive sources:\n\n```subfinder -d example.com -o subdomains.txt```\n\nSeed subwiz with these subdomains:\n\n```subwiz -i subdomains.txt```\n\n### Supported Switches\n\n```commandline\nusage: cli.py [-h] -i INPUT_FILE [-o OUTPUT_FILE] [-n NUM_PREDICTIONS]\n [--no-resolve] [--force-download] [-t TEMPERATURE]\n [-d {auto,cpu,cuda,mps}] [-q MAX_NEW_TOKENS]\n [--resolution_concurrency RESOLUTION_LIM]\n\noptions:\n -h, --help show this help message and exit\n -i INPUT_FILE, --input-file INPUT_FILE\n file containing new-line-separated subdomains.\n (default: None)\n -o OUTPUT_FILE, --output-file OUTPUT_FILE\n output file to write new-line separated subdomains to.\n (default: None)\n -n NUM_PREDICTIONS, --num_predictions NUM_PREDICTIONS\n number of subdomains to predict. (default: 500)\n --no-resolve do not resolve the output subdomains. (default: False)\n --force-download download model and tokenizer files, even if cached.\n (default: False)\n -t TEMPERATURE, --temperature TEMPERATURE\n add randomness to the model, recommended \u2264 0.3)\n (default: 0.0)\n -d {auto,cpu,cuda,mps}, --device {auto,cpu,cuda,mps}\n hardware to run the transformer model on. (default:\n auto)\n -q MAX_NEW_TOKENS, --max_new_tokens MAX_NEW_TOKENS\n maximum length of predicted subdomains in tokens.\n (default: 10)\n --resolution_concurrency RESOLUTION_LIM\n number of concurrent resolutions. (default: 128)\n\n```\n\n### In Python\n\nUse subwiz in Python, with the same inputs as the command line interface.\n\n```\nimport subwiz\n\nknown_subdomains = ['test1.example.com', 'test2.example.com']\nnew_subdomains = subwiz.run(input_domains=known_subdomains)\n```\n\n---\n### Model\n\nUse the `--no-resolve` flag to inspect model outputs without checking if they resolve.\n\n#### Architecture\n\nSubwiz is a ultra-lightweight transformer model based on [nanoGPT](https://github.com/karpathy/nanoGPT/tree/master) \u2764\ufe0f:\n\n- 17.3M parameters.\n- Trained on 26M tokens, lists of subdomains from passive sources.\n- Tokenizer trained on same lists of subdomains (8192 tokens).\n\n#### Hugging Face\nThe model is saved in Hugging Face as [HadrianSecurity/subwiz](https://huggingface.co/HadrianSecurity/subwiz).\nIt is downloaded when you first run subwiz.\n\n#### Inference\n\nTypically, generative transformer models (e.g. ChatGPT) predict a single output sequence.\nSubwiz predicts the N most likely sequences using a beam search algorithm.\n\n\n\n*Beam algorithm to predict the N most likely outputs from a generative transformer model.*\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "A recon tool that uses AI to predict subdomains. Then returns those that resolve.",
"version": "0.1.5",
"project_urls": {
"Source": "https://github.com/hadriansecurity/subwiz"
},
"split_keywords": [
"machine learning",
" recon",
" subdomains",
" transformers"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "63c39172089081b8ff257bbb91e31e19ec3043088d6b609064edbef30708cdee",
"md5": "7500e5bf37ffec22f77b978855495119",
"sha256": "61c4775c959fcd764378db017fabcb2210b9c9edad0956792fd17c359dab976a"
},
"downloads": -1,
"filename": "subwiz-0.1.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7500e5bf37ffec22f77b978855495119",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.9",
"size": 13895,
"upload_time": "2024-11-21T10:32:13",
"upload_time_iso_8601": "2024-11-21T10:32:13.425821Z",
"url": "https://files.pythonhosted.org/packages/63/c3/9172089081b8ff257bbb91e31e19ec3043088d6b609064edbef30708cdee/subwiz-0.1.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f669d02cf1af85c358096fba6983896fb7e9f40b07edccf9b64bc359ff50aad4",
"md5": "c17fbb64b19f4580b1129bc207c946a4",
"sha256": "4b41b2e1a41e894063253d9c09cc8f0a1da5343aadb9af48f0659f463a9600c3"
},
"downloads": -1,
"filename": "subwiz-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "c17fbb64b19f4580b1129bc207c946a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.9",
"size": 324096,
"upload_time": "2024-11-21T10:32:15",
"upload_time_iso_8601": "2024-11-21T10:32:15.513770Z",
"url": "https://files.pythonhosted.org/packages/f6/69/d02cf1af85c358096fba6983896fb7e9f40b07edccf9b64bc359ff50aad4/subwiz-0.1.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-21 10:32:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hadriansecurity",
"github_project": "subwiz",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiodns",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "huggingface-hub",
"specs": [
[
">=",
"0.5.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"1.10.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"8.0.0"
]
]
},
{
"name": "tldextract",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "torch",
"specs": [
[
">=",
"2.3.0"
]
]
},
{
"name": "transformers",
"specs": [
[
">=",
"4.34.0"
]
]
}
],
"lcname": "subwiz"
}
JSON
