# uro
Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; **uro** aims to solve that.
It doesn't make any http requests to the URLs and removes:
- incremental urls e.g. `/page/1/` and `/page/2/`
- blog posts and similar human written content e.g. `/posts/a-brief-history-of-time`
- urls with same path but parameter value difference e.g. `/page.php?id=1` and `/page.php?id=2`
- images, js, css and other "useless" files
#### Installation
The recommended way to install uro is through pip as follows:
```
pip3 install uro
```
### Basic Usage
The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.
```
cat urls.txt | uro
```
### Advanced usage
#### Reading urls from a file (-i/--input)
`uro -i input.txt`
#### Writing urls to a file (-o/--output)
If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.
`uro -i input.txt -o output.txt`
#### Whitelist (`-w/--whitelist`)
uro will ignore all other extensions except the ones provided.
`uro -w php asp html`
**Note:** Extensionless pages e.g. `/books/1` will still be included. To remove them too, use `--filter hasext`.
#### Blacklist (`-b/--blacklist`)
uro will ignore the given extensions.
`uro -b jpg png js pdf`
**Note:** uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use `--filter hasext`.
#### Filters (-f/--filters)
For granular control, uro supports the following filters:
1. **hasparams:** only output urls that have query parameters e.g. `http://example.com/page.php?id=`
2. **noparams:** only output urls that have no query parameters e.g. `http://example.com/page.php`
3. **hasext:** only output urls that have extensions e.g. `http://example.com/page.php`
4. **noext:** only output urls that have no extensions e.g. `http://example.com/page`
5. **keepcontent:** keep human written content e.g. blogs.
6. **keepslash:** don't remove trailing slash from urls e.g. `http://example.com/page/`
7. **vuln:** only output urls with parameters that are know to be vulnerable. [More info.](https://github.com/s0md3v/parth)
Example: `uro --filters hasexts hasparams`
Raw data
{
"_id": null,
"home_page": "https://github.com/s0md3v/uro",
"name": "uro",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "declutter, crawling, pentesting",
"author": "s0md3v",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ad/a5/9ffcecfda1d40085ebbece6248856311c653b1fe81752219f63f16f98ac1/uro-1.0.1.tar.gz",
"platform": null,
"description": "# uro\nUsing a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; **uro** aims to solve that.\n\nIt doesn't make any http requests to the URLs and removes:\n- incremental urls e.g. `/page/1/` and `/page/2/`\n- blog posts and similar human written content e.g. `/posts/a-brief-history-of-time`\n- urls with same path but parameter value difference e.g. `/page.php?id=1` and `/page.php?id=2`\n- images, js, css and other \"useless\" files\n\n\n\n#### Installation\nThe recommended way to install uro is through pip as follows:\n```\npip3 install uro\n```\n\n### Basic Usage\nThe quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.\n```\ncat urls.txt | uro\n```\n\n### Advanced usage\n#### Reading urls from a file (-i/--input)\n\n`uro -i input.txt`\n\n#### Writing urls to a file (-o/--output)\nIf the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.\n\n`uro -i input.txt -o output.txt`\n\n#### Whitelist (`-w/--whitelist`)\nuro will ignore all other extensions except the ones provided.\n\n`uro -w php asp html`\n\n**Note:** Extensionless pages e.g. `/books/1` will still be included. To remove them too, use `--filter hasext`.\n\n#### Blacklist (`-b/--blacklist`)\nuro will ignore the given extensions.\n\n`uro -b jpg png js pdf`\n\n**Note:** uro has a list of \"useless\" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use `--filter hasext`.\n\n#### Filters (-f/--filters)\nFor granular control, uro supports the following filters:\n\n1. **hasparams:** only output urls that have query parameters e.g. `http://example.com/page.php?id=`\n2. **noparams:** only output urls that have no query parameters e.g. `http://example.com/page.php`\n3. **hasext:** only output urls that have extensions e.g. `http://example.com/page.php`\n4. **noext:** only output urls that have no extensions e.g. `http://example.com/page`\n5. **keepcontent:** keep human written content e.g. blogs.\n6. **keepslash:** don't remove trailing slash from urls e.g. `http://example.com/page/`\n7. **vuln:** only output urls with parameters that are know to be vulnerable. [More info.](https://github.com/s0md3v/parth)\n\nExample: `uro --filters hasexts hasparams`\n",
"bugtrack_url": null,
"license": "Apache-2.0 License",
"summary": "A python tool to declutter url lists for crawling/pentesting",
"version": "1.0.1",
"project_urls": {
"Download": "https://github.com/s0md3v/uro/archive/v1.0.1.zip",
"Homepage": "https://github.com/s0md3v/uro"
},
"split_keywords": [
"declutter",
" crawling",
" pentesting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ada59ffcecfda1d40085ebbece6248856311c653b1fe81752219f63f16f98ac1",
"md5": "b5e54d6f9a6e378bf2c4ac430427311b",
"sha256": "a332786b633c01fe097296d0453167333778b1d759b89f597968bba3483a6032"
},
"downloads": -1,
"filename": "uro-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "b5e54d6f9a6e378bf2c4ac430427311b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 9974,
"upload_time": "2024-11-02T20:39:02",
"upload_time_iso_8601": "2024-11-02T20:39:02.855804Z",
"url": "https://files.pythonhosted.org/packages/ad/a5/9ffcecfda1d40085ebbece6248856311c653b1fe81752219f63f16f98ac1/uro-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-02 20:39:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "s0md3v",
"github_project": "uro",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "uro"
}
JSON
