# Deep-Dir - Advanced Directory Enumeration Tool
**Deep-Dir** is a powerful, next-generation directory enumeration tool that combines brute-forcing, intelligent crawling, and advanced fuzzing techniques to discover hidden web directories and files. It's designed to be more effective than traditional tools like dirsearch and gospider by incorporating "deep and wild" features.
## 🚀 Installation
### Via PyPI (Recommended)
```bash
pip install deepdir
```
### From Source
```bash
git clone https://github.com/LifeJiggy/Deep-Dir.git
cd Deep-Dir
pip install -r requirements.txt
python setup.py install
```
### Docker
```bash
docker build -t deepdir .
docker run -it deepdir --help
```
## Features
### 🚀 Core Capabilities
- **Hybrid Scanning**: Combines brute-force enumeration with intelligent web crawling
- **Advanced Fuzzing**: Smart pattern generation with mutations, date-based patterns, and intelligent word variations
- **Anti-WAF Techniques**: Built-in WAF bypass methods including header rotation, encoding variations, and request randomization
- **Recursive Discovery**: Deep recursive scanning with smart stopping conditions
- **Multi-threaded Performance**: Configurable threading with rate limiting and delays
- **Content Analysis**: Advanced content analysis for technology detection, secret finding, and file type identification
- **Intelligent Filtering**: Smart filtering to remove false positives and prioritize important findings
- **Real-time Monitoring**: Live progress tracking with statistics and performance metrics
### 🎯 Advanced Features ("Deep and Wild")
- **Pattern-based Fuzzing**: Generates fuzzing patterns using templates like `{year}`, `{month}`, `{id}`
- **Smart Mutations**: Automatically creates variations of discovered paths (case changes, encoding, suffixes)
- **Date-based Discovery**: Scans for backup files using year/month patterns
- **Encoding Variations**: Tests multiple URL encodings to bypass filters
- **Header Randomization**: Rotates User-Agents and headers to avoid detection
- **Technology Fingerprinting**: Detects web frameworks, CMS, and server technologies
- **Secret Detection**: Finds API keys, tokens, passwords, and other sensitive data
- **File Type Classification**: Identifies admin panels, config files, backups, logs, and more
- **Endpoint Discovery**: Extracts API endpoints and internal URLs from JavaScript and HTML
- **Intelligent Filtering**: Advanced response filtering based on size, content, and regex patterns
### 🛠️ Technical Features
- **Multiple Output Formats**: JSON, CSV, HTML, and plain text reports
- **Configuration Management**: JSON-based config files with CLI override
- **Extensible Architecture**: Modular design for easy feature addition
- **Proxy Support**: HTTP/SOCKS proxy support with proxy rotation
- **Session Management**: Persistent sessions with cookie support
- **Logging**: Comprehensive logging with file and console output
## Installation
### Requirements
- Python 3.9+
- pip
### Install from source
```bash
git clone https://github.com/yourusername/deepdir.git
cd deepdir
pip install -r requirements.txt
```
### Install via pip
```bash
pip install deepdir
```
## Usage
### Basic Usage
```bash
# Simple scan
python deepdir.py -u https://example.com
# With wordlist
python deepdir.py -u https://example.com -w wordlists/common.txt
# Enable all advanced features
python deepdir.py -u https://example.com --fuzz-patterns --anti-waf --smart-mutations
```
### Advanced Usage
```bash
# Recursive scanning with custom extensions
python deepdir.py -u https://example.com -r -d 3 -e php,html,js,txt
# Anti-WAF mode with delays
python deepdir.py -u https://example.com --anti-waf --delay 1 --random-delay 0.5 2.0
# Hybrid mode (brute + crawl) with output
python deepdir.py -u https://example.com --hybrid -o results.json -f json
# Multiple targets
python deepdir.py -l targets.txt -t 20 --quiet
```
### Command Line Options
#### Target Options
- `-u, --url URL`: Target URL
- `-l, --url-list PATH`: File containing list of URLs
#### Wordlist Options
- `-w, --wordlist PATH`: Wordlist file
- `--wordlists PATH [PATH ...]`: Multiple wordlist files
- `-e, --extensions EXT`: File extensions (comma-separated)
#### Scan Modes
- `--brute`: Enable brute-force mode only
- `--crawl`: Enable crawling mode only
- `--hybrid`: Enable hybrid mode (default)
#### Advanced Features
- `-r, --recursive`: Enable recursive scanning
- `-d, --depth INT`: Maximum recursion depth (default: 2)
- `--fuzz-patterns`: Enable advanced fuzzing patterns
- `--anti-waf`: Enable anti-WAF techniques
- `--smart-mutations`: Enable smart word mutations
#### Performance Options
- `-t, --threads INT`: Number of threads (default: 10)
- `--delay FLOAT`: Delay between requests
- `--random-delay MIN MAX`: Random delay range
#### Filtering Options
- `-i, --include-status CODES`: Include status codes
- `-x, --exclude-status CODES`: Exclude status codes
- `--min-size SIZE`: Minimum response size
- `--max-size SIZE`: Maximum response size
#### Output Options
- `-o, --output PATH`: Output file
- `-f, --format FORMAT`: Output format (txt, json, csv, html)
- `-q, --quiet`: Quiet mode
- `-v, --verbose`: Verbose output
#### Configuration
- `-c, --config PATH`: Configuration file
- `--save-config PATH`: Save current config
## Configuration
DeepDir uses JSON-based configuration files. Example config:
```json
{
"threads": 15,
"recursive": true,
"max_depth": 3,
"fuzz_patterns": true,
"anti_waf": true,
"extensions": ["php", "html", "js", "txt"],
"include_status_codes": [200, 301, 302, 403],
"exclude_status_codes": [404, 429],
"user_agent": "DeepDir/5.0",
"timeout": 10.0,
"delay": 0.5
}
```
## Wordlists
DeepDir comes with built-in wordlists and supports custom wordlists. The tool automatically applies extensions and generates variations.
### Built-in Wordlists
- `wordlists/common.txt`: Common web directories and files
- `wordlists/admin.txt`: Administrative panels
- `wordlists/api.txt`: API endpoints
- `wordlists/backup.txt`: Backup files
### Wordlist Format
Wordlists are simple text files with one entry per line:
```
admin
login.php
config
backup.sql
```
## Advanced Features
### Fuzzing Patterns
DeepDir generates intelligent fuzzing patterns:
- Date-based: `backup2023`, `bak202312`
- ID-based: `user1`, `admin2`, `test123`
- Mutation-based: `Admin`, `ADMIN`, `admin2`, `admin_old`
### Anti-WAF Techniques
- User-Agent rotation
- Header randomization
- Request encoding variations
- Path traversal encoding
- Junk header injection
### Smart Filtering
- Response size filtering
- Content-based filtering
- Regex pattern matching
- Status code filtering
## Output Formats
### Text (default)
```
200 1234 https://example.com/admin
403 234 https://example.com/backup
```
### JSON
```json
[
{
"url": "https://example.com/admin",
"status_code": 200,
"content_length": 1234,
"scan_type": "brute_force"
}
]
```
### CSV
```csv
url,status_code,content_length,scan_type
https://example.com/admin,200,1234,brute_force
```
### HTML
Generates a formatted HTML report with tables and styling.
## Examples
### Basic Directory Discovery
```bash
python deepdir.py -u https://example.com -w wordlists/common.txt
```
### Advanced Reconnaissance
```bash
python deepdir.py -u https://example.com \
--hybrid \
--recursive \
--depth 3 \
--fuzz-patterns \
--anti-waf \
--extensions php,html,js,txt,bak \
--threads 20 \
--delay 0.5 \
--output results.json \
--format json \
--verbose
```
### API Discovery
```bash
python deepdir.py -u https://api.example.com \
--wordlist wordlists/api.txt \
--extensions json,xml,yaml \
--include-status 200,201,401,403
```
### Backup File Hunting
```bash
python deepdir.py -u https://example.com \
--fuzz-patterns \
--extensions bak,backup,old,orig,tmp \
--recursive
```
## Contributing
Contributions are welcome! Please feel free to submit pull requests, report bugs, or suggest features.
### Development Setup
```bash
git clone https://github.com/yourusername/deepdir.git
cd deepdir
pip install -r requirements.txt
pip install -r requirements-dev.txt
```
### Running Tests
```bash
python -m pytest tests/
```
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Disclaimer
This tool is for educational and authorized testing purposes only. Users are responsible for complying with applicable laws and regulations. The authors are not responsible for any misuse or damage caused by this tool.
## Credits
Inspired by dirsearch, gospider, and other directory enumeration tools. Built with ❤️ for the security community.
Raw data
{
"_id": null,
"home_page": "https://github.com/LifeJiggy/Deep-Dir.git",
"name": "deepdir",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "directory enumeration brute-force web-security penetration-testing reconnaissance",
"author": "ArkhAngelLifeJiggy",
"author_email": "Bloomtonjovish@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e2/55/a9994ca8e6adc1c607903646047afc2c2f35fbbc542409e4df8a07217bc8/deepdir-5.0.0.tar.gz",
"platform": null,
"description": "# Deep-Dir - Advanced Directory Enumeration Tool\r\n\r\n**Deep-Dir** is a powerful, next-generation directory enumeration tool that combines brute-forcing, intelligent crawling, and advanced fuzzing techniques to discover hidden web directories and files. It's designed to be more effective than traditional tools like dirsearch and gospider by incorporating \"deep and wild\" features.\r\n\r\n## \ud83d\ude80 Installation\r\n\r\n### Via PyPI (Recommended)\r\n```bash\r\npip install deepdir\r\n```\r\n\r\n### From Source\r\n```bash\r\ngit clone https://github.com/LifeJiggy/Deep-Dir.git\r\ncd Deep-Dir\r\npip install -r requirements.txt\r\npython setup.py install\r\n```\r\n\r\n### Docker\r\n```bash\r\ndocker build -t deepdir .\r\ndocker run -it deepdir --help\r\n```\r\n\r\n## Features\r\n\r\n### \ud83d\ude80 Core Capabilities\r\n- **Hybrid Scanning**: Combines brute-force enumeration with intelligent web crawling\r\n- **Advanced Fuzzing**: Smart pattern generation with mutations, date-based patterns, and intelligent word variations\r\n- **Anti-WAF Techniques**: Built-in WAF bypass methods including header rotation, encoding variations, and request randomization\r\n- **Recursive Discovery**: Deep recursive scanning with smart stopping conditions\r\n- **Multi-threaded Performance**: Configurable threading with rate limiting and delays\r\n- **Content Analysis**: Advanced content analysis for technology detection, secret finding, and file type identification\r\n- **Intelligent Filtering**: Smart filtering to remove false positives and prioritize important findings\r\n- **Real-time Monitoring**: Live progress tracking with statistics and performance metrics\r\n\r\n### \ud83c\udfaf Advanced Features (\"Deep and Wild\")\r\n- **Pattern-based Fuzzing**: Generates fuzzing patterns using templates like `{year}`, `{month}`, `{id}`\r\n- **Smart Mutations**: Automatically creates variations of discovered paths (case changes, encoding, suffixes)\r\n- **Date-based Discovery**: Scans for backup files using year/month patterns\r\n- **Encoding Variations**: Tests multiple URL encodings to bypass filters\r\n- **Header Randomization**: Rotates User-Agents and headers to avoid detection\r\n- **Technology Fingerprinting**: Detects web frameworks, CMS, and server technologies\r\n- **Secret Detection**: Finds API keys, tokens, passwords, and other sensitive data\r\n- **File Type Classification**: Identifies admin panels, config files, backups, logs, and more\r\n- **Endpoint Discovery**: Extracts API endpoints and internal URLs from JavaScript and HTML\r\n- **Intelligent Filtering**: Advanced response filtering based on size, content, and regex patterns\r\n\r\n### \ud83d\udee0\ufe0f Technical Features\r\n- **Multiple Output Formats**: JSON, CSV, HTML, and plain text reports\r\n- **Configuration Management**: JSON-based config files with CLI override\r\n- **Extensible Architecture**: Modular design for easy feature addition\r\n- **Proxy Support**: HTTP/SOCKS proxy support with proxy rotation\r\n- **Session Management**: Persistent sessions with cookie support\r\n- **Logging**: Comprehensive logging with file and console output\r\n\r\n## Installation\r\n\r\n### Requirements\r\n- Python 3.9+\r\n- pip\r\n\r\n### Install from source\r\n```bash\r\ngit clone https://github.com/yourusername/deepdir.git\r\ncd deepdir\r\npip install -r requirements.txt\r\n```\r\n\r\n### Install via pip\r\n```bash\r\npip install deepdir\r\n```\r\n\r\n## Usage\r\n\r\n### Basic Usage\r\n```bash\r\n# Simple scan\r\npython deepdir.py -u https://example.com\r\n\r\n# With wordlist\r\npython deepdir.py -u https://example.com -w wordlists/common.txt\r\n\r\n# Enable all advanced features\r\npython deepdir.py -u https://example.com --fuzz-patterns --anti-waf --smart-mutations\r\n```\r\n\r\n### Advanced Usage\r\n```bash\r\n# Recursive scanning with custom extensions\r\npython deepdir.py -u https://example.com -r -d 3 -e php,html,js,txt\r\n\r\n# Anti-WAF mode with delays\r\npython deepdir.py -u https://example.com --anti-waf --delay 1 --random-delay 0.5 2.0\r\n\r\n# Hybrid mode (brute + crawl) with output\r\npython deepdir.py -u https://example.com --hybrid -o results.json -f json\r\n\r\n# Multiple targets\r\npython deepdir.py -l targets.txt -t 20 --quiet\r\n```\r\n\r\n### Command Line Options\r\n\r\n#### Target Options\r\n- `-u, --url URL`: Target URL\r\n- `-l, --url-list PATH`: File containing list of URLs\r\n\r\n#### Wordlist Options\r\n- `-w, --wordlist PATH`: Wordlist file\r\n- `--wordlists PATH [PATH ...]`: Multiple wordlist files\r\n- `-e, --extensions EXT`: File extensions (comma-separated)\r\n\r\n#### Scan Modes\r\n- `--brute`: Enable brute-force mode only\r\n- `--crawl`: Enable crawling mode only\r\n- `--hybrid`: Enable hybrid mode (default)\r\n\r\n#### Advanced Features\r\n- `-r, --recursive`: Enable recursive scanning\r\n- `-d, --depth INT`: Maximum recursion depth (default: 2)\r\n- `--fuzz-patterns`: Enable advanced fuzzing patterns\r\n- `--anti-waf`: Enable anti-WAF techniques\r\n- `--smart-mutations`: Enable smart word mutations\r\n\r\n#### Performance Options\r\n- `-t, --threads INT`: Number of threads (default: 10)\r\n- `--delay FLOAT`: Delay between requests\r\n- `--random-delay MIN MAX`: Random delay range\r\n\r\n#### Filtering Options\r\n- `-i, --include-status CODES`: Include status codes\r\n- `-x, --exclude-status CODES`: Exclude status codes\r\n- `--min-size SIZE`: Minimum response size\r\n- `--max-size SIZE`: Maximum response size\r\n\r\n#### Output Options\r\n- `-o, --output PATH`: Output file\r\n- `-f, --format FORMAT`: Output format (txt, json, csv, html)\r\n- `-q, --quiet`: Quiet mode\r\n- `-v, --verbose`: Verbose output\r\n\r\n#### Configuration\r\n- `-c, --config PATH`: Configuration file\r\n- `--save-config PATH`: Save current config\r\n\r\n## Configuration\r\n\r\nDeepDir uses JSON-based configuration files. Example config:\r\n\r\n```json\r\n{\r\n \"threads\": 15,\r\n \"recursive\": true,\r\n \"max_depth\": 3,\r\n \"fuzz_patterns\": true,\r\n \"anti_waf\": true,\r\n \"extensions\": [\"php\", \"html\", \"js\", \"txt\"],\r\n \"include_status_codes\": [200, 301, 302, 403],\r\n \"exclude_status_codes\": [404, 429],\r\n \"user_agent\": \"DeepDir/5.0\",\r\n \"timeout\": 10.0,\r\n \"delay\": 0.5\r\n}\r\n```\r\n\r\n## Wordlists\r\n\r\nDeepDir comes with built-in wordlists and supports custom wordlists. The tool automatically applies extensions and generates variations.\r\n\r\n### Built-in Wordlists\r\n- `wordlists/common.txt`: Common web directories and files\r\n- `wordlists/admin.txt`: Administrative panels\r\n- `wordlists/api.txt`: API endpoints\r\n- `wordlists/backup.txt`: Backup files\r\n\r\n### Wordlist Format\r\nWordlists are simple text files with one entry per line:\r\n```\r\nadmin\r\nlogin.php\r\nconfig\r\nbackup.sql\r\n```\r\n\r\n## Advanced Features\r\n\r\n### Fuzzing Patterns\r\nDeepDir generates intelligent fuzzing patterns:\r\n- Date-based: `backup2023`, `bak202312`\r\n- ID-based: `user1`, `admin2`, `test123`\r\n- Mutation-based: `Admin`, `ADMIN`, `admin2`, `admin_old`\r\n\r\n### Anti-WAF Techniques\r\n- User-Agent rotation\r\n- Header randomization\r\n- Request encoding variations\r\n- Path traversal encoding\r\n- Junk header injection\r\n\r\n### Smart Filtering\r\n- Response size filtering\r\n- Content-based filtering\r\n- Regex pattern matching\r\n- Status code filtering\r\n\r\n## Output Formats\r\n\r\n### Text (default)\r\n```\r\n200 1234 https://example.com/admin\r\n403 234 https://example.com/backup\r\n```\r\n\r\n### JSON\r\n```json\r\n[\r\n {\r\n \"url\": \"https://example.com/admin\",\r\n \"status_code\": 200,\r\n \"content_length\": 1234,\r\n \"scan_type\": \"brute_force\"\r\n }\r\n]\r\n```\r\n\r\n### CSV\r\n```csv\r\nurl,status_code,content_length,scan_type\r\nhttps://example.com/admin,200,1234,brute_force\r\n```\r\n\r\n### HTML\r\nGenerates a formatted HTML report with tables and styling.\r\n\r\n## Examples\r\n\r\n### Basic Directory Discovery\r\n```bash\r\npython deepdir.py -u https://example.com -w wordlists/common.txt\r\n```\r\n\r\n### Advanced Reconnaissance\r\n```bash\r\npython deepdir.py -u https://example.com \\\r\n --hybrid \\\r\n --recursive \\\r\n --depth 3 \\\r\n --fuzz-patterns \\\r\n --anti-waf \\\r\n --extensions php,html,js,txt,bak \\\r\n --threads 20 \\\r\n --delay 0.5 \\\r\n --output results.json \\\r\n --format json \\\r\n --verbose\r\n```\r\n\r\n### API Discovery\r\n```bash\r\npython deepdir.py -u https://api.example.com \\\r\n --wordlist wordlists/api.txt \\\r\n --extensions json,xml,yaml \\\r\n --include-status 200,201,401,403\r\n```\r\n\r\n### Backup File Hunting\r\n```bash\r\npython deepdir.py -u https://example.com \\\r\n --fuzz-patterns \\\r\n --extensions bak,backup,old,orig,tmp \\\r\n --recursive\r\n```\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please feel free to submit pull requests, report bugs, or suggest features.\r\n\r\n### Development Setup\r\n```bash\r\ngit clone https://github.com/yourusername/deepdir.git\r\ncd deepdir\r\npip install -r requirements.txt\r\npip install -r requirements-dev.txt\r\n```\r\n\r\n### Running Tests\r\n```bash\r\npython -m pytest tests/\r\n```\r\n\r\n## License\r\n\r\nThis project is licensed under the MIT License - see the LICENSE file for details.\r\n\r\n## Disclaimer\r\n\r\nThis tool is for educational and authorized testing purposes only. Users are responsible for complying with applicable laws and regulations. The authors are not responsible for any misuse or damage caused by this tool.\r\n\r\n## Credits\r\n\r\nInspired by dirsearch, gospider, and other directory enumeration tools. Built with \u2764\ufe0f for the security community.\r\n",
"bugtrack_url": null,
"license": null,
"summary": "Advanced Directory Enumeration Tool - More powerful than dirsearch and gospider",
"version": "5.0.0",
"project_urls": {
"Bug Reports": "https://github.com/LifeJiggy/Deep-Dir/issues",
"Documentation": "https://github.com/LifeJiggy/Deep-Dir#readme",
"Homepage": "https://github.com/LifeJiggy/Deep-Dir.git",
"Source": "https://github.com/LifeJiggy/Deep-Dir.git"
},
"split_keywords": [
"directory",
"enumeration",
"brute-force",
"web-security",
"penetration-testing",
"reconnaissance"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "c1aa280f0cdb38052f18c8b635254d506650757e5a0f1b390fbc9f72a6aa5f06",
"md5": "6287f78d8cfe575e97a2fa08f71c65c2",
"sha256": "4e42d0964bcff1bcef3a0e455631981f8d02840ab6b656f173679dca861df7ff"
},
"downloads": -1,
"filename": "deepdir-5.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6287f78d8cfe575e97a2fa08f71c65c2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 31138,
"upload_time": "2025-11-02T10:58:18",
"upload_time_iso_8601": "2025-11-02T10:58:18.611482Z",
"url": "https://files.pythonhosted.org/packages/c1/aa/280f0cdb38052f18c8b635254d506650757e5a0f1b390fbc9f72a6aa5f06/deepdir-5.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e255a9994ca8e6adc1c607903646047afc2c2f35fbbc542409e4df8a07217bc8",
"md5": "2f8c1166a9d33ffad398afb492f1c2c5",
"sha256": "287c28062b47c1a5690086ea507b713bd5d735010f3064208cd4c3e0d43d3e99"
},
"downloads": -1,
"filename": "deepdir-5.0.0.tar.gz",
"has_sig": false,
"md5_digest": "2f8c1166a9d33ffad398afb492f1c2c5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 38247,
"upload_time": "2025-11-02T10:58:22",
"upload_time_iso_8601": "2025-11-02T10:58:22.127434Z",
"url": "https://files.pythonhosted.org/packages/e2/55/a9994ca8e6adc1c607903646047afc2c2f35fbbc542409e4df8a07217bc8/deepdir-5.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-02 10:58:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "LifeJiggy",
"github_project": "Deep-Dir",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "requests",
"specs": [
[
">=",
"2.25.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
">=",
"4.9.0"
]
]
},
{
"name": "lxml",
"specs": [
[
">=",
"4.6.0"
]
]
},
{
"name": "urllib3",
"specs": [
[
">=",
"1.26.0"
]
]
},
{
"name": "dataclasses",
"specs": [
[
">=",
"0.6"
]
]
},
{
"name": "colorama",
"specs": [
[
">=",
"0.4.4"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.62.0"
]
]
},
{
"name": "pyyaml",
"specs": [
[
">=",
"5.4.0"
]
]
},
{
"name": "jsonschema",
"specs": [
[
">=",
"3.2.0"
]
]
},
{
"name": "validators",
"specs": [
[
">=",
"0.18.0"
]
]
},
{
"name": "dnspython",
"specs": [
[
">=",
"2.1.0"
]
]
},
{
"name": "ipaddress",
"specs": [
[
">=",
"1.0.23"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
">=",
"4.0.0"
]
]
}
],
"lcname": "deepdir"
}
JSON
