.. role:: python(code)
:language: python
Advocate
========
.. image:: https://travis-ci.org/JordanMilne/Advocate.svg?branch=master
:target: https://travis-ci.org/JordanMilne/Advocate/
.. image:: https://codecov.io/github/JordanMilne/Advocate/coverage.svg?branch=master
:target: https://codecov.io/github/JordanMilne/Advocate
.. image:: https://img.shields.io/pypi/pyversions/advocate.svg
.. image:: https://img.shields.io/pypi/v/advocate.svg
:target: https://pypi.python.org/pypi/advocate
Advocate is a set of tools based around the `requests library <https://github.com/kennethreitz/requests>`_ for safely making
HTTP requests on behalf of a third party. Specifically, it aims to prevent
common techniques that enable `SSRF attacks <https://cwe.mitre.org/data/definitions/918.html>`_.
Advocate was inspired by `fin1te's SafeCurl project <https://github.com/fin1te/safecurl>`_.
Installation
============
.. code-block:: bash
pip install advocate
Advocate is officially supported on CPython 2.7+, CPython 3.4+ and PyPy 2. PyPy 3 may work as well, but
you'll need a copy of the ipaddress module from elsewhere.
See it in action
================
If you want to try out Advocate to see what kind of things it catches, there's a `test site up on advocate.saynotolinux.com <http://advocate.saynotolinux.com/>`_.
Examples
========
Advocate is more-or-less a drop-in replacement for requests. In most cases you can just replace "requests" with
"advocate" where necessary and be good to go:
.. code-block:: python
>>> import advocate
>>> print advocate.get("http://google.com/")
<Response [200]>
Advocate also provides a subclassed :python:`requests.Session` with sane defaults for
validation already set up:
.. code-block:: python
>>> import advocate
>>> sess = advocate.Session()
>>> print sess.get("http://google.com/")
<Response [200]>
>>> print sess.get("http://localhost/")
advocate.exceptions.UnacceptableAddressException: ('localhost', 80)
All of the wrapped request functions accept a :python:`validator` kwarg where you
can set additional rules:
.. code-block:: python
>>> import advocate
>>> validator = advocate.AddrValidator(hostname_blacklist={"*.museum",})
>>> print advocate.get("http://educational.MUSEUM/", validator=validator)
advocate.exceptions.UnacceptableAddressException: educational.MUSEUM
If you require more advanced rules than the defaults, but don't want to have to pass
the validator kwarg everywhere, there's :python:`RequestsAPIWrapper` . You can
define a wrapper in a common file and import it instead of advocate:
.. code-block:: python
>>> from advocate import AddrValidator, RequestsAPIWrapper
>>> from advocate.packages import ipaddress
>>> dougs_advocate = RequestsAPIWrapper(AddrValidator(ip_blacklist={
... # Contains data incomprehensible to mere mortals
... ipaddress.ip_network("42.42.42.42/32")
... }))
>>> print dougs_advocate.get("http://42.42.42.42/")
advocate.exceptions.UnacceptableAddressException: ('42.42.42.42', 80)
Other than that, you can do just about everything with Advocate that you can
with an unwrapped requests. Advocate passes requests' test suite with the
exception of tests that require :python:`Session.mount()`.
Conditionally bypassing protection
==================================
If you want to allow certain users to bypass Advocate's restrictions, just
use plain 'ol requests by doing something like:
.. code-block:: python
if user == "mr_skeltal":
requests_module = requests
else:
requests_module = advocate
resp = requests_module.get("http://example.com/doot_doot")
requests-futures support
========================
A thin wrapper around `requests-futures <https://github.com/ross/requests-futures>`_ is provided to ease writing async-friendly code:
.. code-block:: python
>>> from advocate.futures import FuturesSession
>>> sess = FuturesSession()
>>> fut = sess.get("http://example.com/")
>>> fut
<Future at 0x10c717f28 state=finished returned Response>
>>> fut.result()
<Response [200]>
You can do basically everything you can do with regular :python:`FuturesSession` s and :python:`advocate.Session` s:
.. code-block:: python
>>> from advocate import AddrValidator
>>> from advocate.futures import FuturesSession
>>> sess = FuturesSession(max_workers=20, validator=AddrValidator(hostname_blacklist={"*.museum"}))
>>> fut = sess.get("http://anice.museum/")
>>> fut
<Future at 0x10c696668 state=running>
>>> fut.result()
Traceback (most recent call last):
# [...]
advocate.exceptions.UnacceptableAddressException: anice.museum
When should I use Advocate?
===========================
Any time you're fetching resources over HTTP for / from someone you don't trust!
When should I not use Advocate?
===============================
That's a tough one. There are a few cases I can think of where I wouldn't:
* When good, safe support for IPv6 is important
* When internal hosts use globally routable addresses and you can't guess their prefix to blacklist it ahead of time
* You already have a good handle on network security within your network
Actually, if you're comfortable enough with Squid and network security, you should set up a secured Squid instance on a segregated subnet
and proxy through that instead. Advocate attempts to guess whether an address references an internal host
and block access, but it's definitely preferable to proxy through a host can't access anything internal in the first place!
Of course, if you're writing an app / library that's meant to be usable OOTB on other people's networks, Advocate + a user-configurable
blacklist is probably the safer bet.
This seems like it's been done before
=====================================
There've been a few similar projects, but in my opinion Advocate's approach is the best because:
It sees URLs the same as the underlying HTTP library
----------------------------------------------------
Parsing URLs is hard, and no two URL parsers seem to behave exactly the same. The tiniest
differences in parsing between your validator and the underlying HTTP library can lead
to vulnerabilities. For example, differences between PHP's :python:`parse_url` and cURL's
URL parser `allowed a blacklist bypass in SafeCurl <https://github.com/fin1te/safecurl/issues/5>`_.
Advocate doesn't do URL parsing at all, and lets requests handle it. Advocate only looks at the
address requests actually tries to open a socket to.
It deals with DNS rebinding
---------------------------
Two consecutive calls to :python:`socket.getaddrinfo` aren't guaranteed to return the same
info, depending on the system configuration. If the "safe" looking record TTLs between
the verification lookup and the lookup for actually opening the socket, we may end
up connecting to a very different server than the one we OK'd!
Advocate gets around this by only using one :python:`getaddrinfo` call for both verification
and connecting the socket. In pseudocode:
.. code-block:: python
def connect_socket(host, port):
for res in socket.getaddrinfo(host, port):
# where `res` will be a tuple containing the IP for the host
if not is_blacklisted(res):
# ... connect the socket using `res`
See `Wikipedia's article on DNS rebinding attacks <https://en.wikipedia.org/wiki/DNS_rebinding>`_ for more info.
It handles redirects sanely
---------------------------
Most of the other SSRF-prevention libs cover this, but I've seen a lot
of sample code online that doesn't. Advocate will catch it since it inspects
*every* connection attempt the underlying HTTP lib makes.
TODO
====
Proper IPv6 Support?
--------------------
Advocate's IPv6 support is still a work-in-progress, since I'm not
that familiar with the spec, and there are so many ways to tunnel IPv4 over IPv6,
as well as other non-obvious gotchas. IPv6 records are ignored by default
for now, but you can enable by using an :python:`AddrValidator` with :python:`allow_ipv6=True`.
It should mostly work as expected, but Advocate's approach might not even make sense with
most IPv6 deployments, see `Issue #3 <https://github.com/JordanMilne/Advocate/issues/3>`_ for
more info.
If you can think of any improvements to the IPv6 handling, please submit an issue or PR!
Caveats
=======
* This is beta-quality software, the API might change without warning!
* :python:`mount()` ing other adapters is disallowed to prevent Advocate's validating adapters from being clobbered.
* Advocate does not, and might never support the use of HTTP proxies.
* Proper IPv6 support is still a WIP as noted above.
Acknowledgements
================
* https://github.com/fin1te/safecurl for inspiration
* https://github.com/kennethreitz/requests for the lovely requests module
* https://bitbucket.org/kwi/py2-ipaddress for the backport of ipaddress
* https://github.com/hakobe/paranoidhttp a similar project targeting golang
* https://github.com/uber-common/paranoid-request a similar project targeting Node
* http://search.cpan.org/~tsibley/LWP-UserAgent-Paranoid/ a similar project targeting Perl 5
Raw data
{
"_id": null,
"home_page": "https://github.com/JordanMilne/Advocate",
"name": "advocate",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "http requests security ssrf proxy rebinding advocate",
"author": "Jordan Milne",
"author_email": "advocate@saynotolinux.com",
"download_url": "https://files.pythonhosted.org/packages/fe/5e/3103b1c63c6cc2a0cdbb1c1e399f700501e1001675c700d39364f9f8df28/advocate-1.0.0.tar.gz",
"platform": "",
"description": ".. role:: python(code)\n :language: python\n\nAdvocate\n========\n\n.. image:: https://travis-ci.org/JordanMilne/Advocate.svg?branch=master\n :target: https://travis-ci.org/JordanMilne/Advocate/\n.. image:: https://codecov.io/github/JordanMilne/Advocate/coverage.svg?branch=master\n :target: https://codecov.io/github/JordanMilne/Advocate\n.. image:: https://img.shields.io/pypi/pyversions/advocate.svg\n.. image:: https://img.shields.io/pypi/v/advocate.svg\n :target: https://pypi.python.org/pypi/advocate\n\n\nAdvocate is a set of tools based around the `requests library <https://github.com/kennethreitz/requests>`_ for safely making\nHTTP requests on behalf of a third party. Specifically, it aims to prevent \ncommon techniques that enable `SSRF attacks <https://cwe.mitre.org/data/definitions/918.html>`_. \n\nAdvocate was inspired by `fin1te's SafeCurl project <https://github.com/fin1te/safecurl>`_.\n\nInstallation\n============\n\n.. code-block:: bash\n\n pip install advocate\n\nAdvocate is officially supported on CPython 2.7+, CPython 3.4+ and PyPy 2. PyPy 3 may work as well, but \nyou'll need a copy of the ipaddress module from elsewhere.\n\nSee it in action\n================\n\nIf you want to try out Advocate to see what kind of things it catches, there's a `test site up on advocate.saynotolinux.com <http://advocate.saynotolinux.com/>`_.\n\nExamples\n========\n\nAdvocate is more-or-less a drop-in replacement for requests. In most cases you can just replace \"requests\" with\n\"advocate\" where necessary and be good to go:\n\n.. code-block:: python\n\n >>> import advocate\n >>> print advocate.get(\"http://google.com/\")\n <Response [200]>\n\nAdvocate also provides a subclassed :python:`requests.Session` with sane defaults for\nvalidation already set up:\n\n.. code-block:: python\n\n >>> import advocate\n >>> sess = advocate.Session()\n >>> print sess.get(\"http://google.com/\")\n <Response [200]>\n >>> print sess.get(\"http://localhost/\")\n advocate.exceptions.UnacceptableAddressException: ('localhost', 80)\n\nAll of the wrapped request functions accept a :python:`validator` kwarg where you\ncan set additional rules:\n\n.. code-block:: python\n\n >>> import advocate\n >>> validator = advocate.AddrValidator(hostname_blacklist={\"*.museum\",})\n >>> print advocate.get(\"http://educational.MUSEUM/\", validator=validator)\n advocate.exceptions.UnacceptableAddressException: educational.MUSEUM\n\nIf you require more advanced rules than the defaults, but don't want to have to pass\nthe validator kwarg everywhere, there's :python:`RequestsAPIWrapper` . You can\ndefine a wrapper in a common file and import it instead of advocate:\n\n.. code-block:: python\n\n >>> from advocate import AddrValidator, RequestsAPIWrapper\n >>> from advocate.packages import ipaddress\n >>> dougs_advocate = RequestsAPIWrapper(AddrValidator(ip_blacklist={\n ... # Contains data incomprehensible to mere mortals\n ... ipaddress.ip_network(\"42.42.42.42/32\")\n ... }))\n >>> print dougs_advocate.get(\"http://42.42.42.42/\")\n advocate.exceptions.UnacceptableAddressException: ('42.42.42.42', 80)\n\n\nOther than that, you can do just about everything with Advocate that you can\nwith an unwrapped requests. Advocate passes requests' test suite with the\nexception of tests that require :python:`Session.mount()`.\n\nConditionally bypassing protection\n==================================\n\nIf you want to allow certain users to bypass Advocate's restrictions, just\nuse plain 'ol requests by doing something like:\n\n.. code-block:: python\n\n if user == \"mr_skeltal\":\n requests_module = requests\n else:\n requests_module = advocate\n resp = requests_module.get(\"http://example.com/doot_doot\")\n\n\nrequests-futures support\n========================\n\nA thin wrapper around `requests-futures <https://github.com/ross/requests-futures>`_ is provided to ease writing async-friendly code:\n\n.. code-block:: python\n\n >>> from advocate.futures import FuturesSession\n >>> sess = FuturesSession()\n >>> fut = sess.get(\"http://example.com/\")\n >>> fut\n <Future at 0x10c717f28 state=finished returned Response>\n >>> fut.result()\n <Response [200]>\n\nYou can do basically everything you can do with regular :python:`FuturesSession` s and :python:`advocate.Session` s:\n\n.. code-block:: python\n\n >>> from advocate import AddrValidator\n >>> from advocate.futures import FuturesSession\n >>> sess = FuturesSession(max_workers=20, validator=AddrValidator(hostname_blacklist={\"*.museum\"}))\n >>> fut = sess.get(\"http://anice.museum/\")\n >>> fut\n <Future at 0x10c696668 state=running>\n >>> fut.result()\n Traceback (most recent call last):\n # [...]\n advocate.exceptions.UnacceptableAddressException: anice.museum\n\n\nWhen should I use Advocate?\n===========================\n\nAny time you're fetching resources over HTTP for / from someone you don't trust!\n\nWhen should I not use Advocate?\n===============================\n\nThat's a tough one. There are a few cases I can think of where I wouldn't:\n\n* When good, safe support for IPv6 is important\n* When internal hosts use globally routable addresses and you can't guess their prefix to blacklist it ahead of time\n* You already have a good handle on network security within your network\n\nActually, if you're comfortable enough with Squid and network security, you should set up a secured Squid instance on a segregated subnet\nand proxy through that instead. Advocate attempts to guess whether an address references an internal host\nand block access, but it's definitely preferable to proxy through a host can't access anything internal in the first place!\n\nOf course, if you're writing an app / library that's meant to be usable OOTB on other people's networks, Advocate + a user-configurable\nblacklist is probably the safer bet.\n\n\nThis seems like it's been done before\n=====================================\n\nThere've been a few similar projects, but in my opinion Advocate's approach is the best because:\n\nIt sees URLs the same as the underlying HTTP library\n----------------------------------------------------\n\nParsing URLs is hard, and no two URL parsers seem to behave exactly the same. The tiniest\ndifferences in parsing between your validator and the underlying HTTP library can lead\nto vulnerabilities. For example, differences between PHP's :python:`parse_url` and cURL's\nURL parser `allowed a blacklist bypass in SafeCurl <https://github.com/fin1te/safecurl/issues/5>`_.\n\nAdvocate doesn't do URL parsing at all, and lets requests handle it. Advocate only looks at the\naddress requests actually tries to open a socket to.\n\nIt deals with DNS rebinding\n---------------------------\n\nTwo consecutive calls to :python:`socket.getaddrinfo` aren't guaranteed to return the same\ninfo, depending on the system configuration. If the \"safe\" looking record TTLs between\nthe verification lookup and the lookup for actually opening the socket, we may end\nup connecting to a very different server than the one we OK'd!\n\nAdvocate gets around this by only using one :python:`getaddrinfo` call for both verification\nand connecting the socket. In pseudocode:\n\n.. code-block:: python\n\n def connect_socket(host, port):\n for res in socket.getaddrinfo(host, port):\n # where `res` will be a tuple containing the IP for the host\n if not is_blacklisted(res):\n # ... connect the socket using `res`\n\nSee `Wikipedia's article on DNS rebinding attacks <https://en.wikipedia.org/wiki/DNS_rebinding>`_ for more info.\n\nIt handles redirects sanely\n---------------------------\n\nMost of the other SSRF-prevention libs cover this, but I've seen a lot\nof sample code online that doesn't. Advocate will catch it since it inspects\n*every* connection attempt the underlying HTTP lib makes. \n\n\nTODO\n====\n\nProper IPv6 Support?\n--------------------\n\nAdvocate's IPv6 support is still a work-in-progress, since I'm not\nthat familiar with the spec, and there are so many ways to tunnel IPv4 over IPv6,\nas well as other non-obvious gotchas. IPv6 records are ignored by default\nfor now, but you can enable by using an :python:`AddrValidator` with :python:`allow_ipv6=True`.\n\nIt should mostly work as expected, but Advocate's approach might not even make sense with\nmost IPv6 deployments, see `Issue #3 <https://github.com/JordanMilne/Advocate/issues/3>`_ for\nmore info.\n\nIf you can think of any improvements to the IPv6 handling, please submit an issue or PR!\n\n\nCaveats\n=======\n\n* This is beta-quality software, the API might change without warning!\n* :python:`mount()` ing other adapters is disallowed to prevent Advocate's validating adapters from being clobbered.\n* Advocate does not, and might never support the use of HTTP proxies.\n* Proper IPv6 support is still a WIP as noted above.\n\nAcknowledgements\n================\n\n* https://github.com/fin1te/safecurl for inspiration\n* https://github.com/kennethreitz/requests for the lovely requests module\n* https://bitbucket.org/kwi/py2-ipaddress for the backport of ipaddress\n* https://github.com/hakobe/paranoidhttp a similar project targeting golang\n* https://github.com/uber-common/paranoid-request a similar project targeting Node\n* http://search.cpan.org/~tsibley/LWP-UserAgent-Paranoid/ a similar project targeting Perl 5\n\n\n",
"bugtrack_url": null,
"license": "Apache 2",
"summary": "A wrapper around the requests library for safely making HTTP requests on behalf of a third party",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/JordanMilne/Advocate"
},
"split_keywords": [
"http",
"requests",
"security",
"ssrf",
"proxy",
"rebinding",
"advocate"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4f48ab74ba882cf6f5080ed3fc2f353166930376ac4484d784ceb4dfd5f19c78",
"md5": "e0d413fcc95c4b4b52bf953b4550eb17",
"sha256": "e8b340e49fadc0e416fbc9e81ef52d74858ccad16357dabde6cf9d99a7407d70"
},
"downloads": -1,
"filename": "advocate-1.0.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "e0d413fcc95c4b4b52bf953b4550eb17",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 34179,
"upload_time": "2020-07-14T15:34:10",
"upload_time_iso_8601": "2020-07-14T15:34:10.013645Z",
"url": "https://files.pythonhosted.org/packages/4f/48/ab74ba882cf6f5080ed3fc2f353166930376ac4484d784ceb4dfd5f19c78/advocate-1.0.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fe5e3103b1c63c6cc2a0cdbb1c1e399f700501e1001675c700d39364f9f8df28",
"md5": "1a8798c3b0d6f4888cb19f7e987ddaff",
"sha256": "1bf1170e41334279996580329c594e017540ab0eaf7a152323e743f0a85a353d"
},
"downloads": -1,
"filename": "advocate-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "1a8798c3b0d6f4888cb19f7e987ddaff",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 39981,
"upload_time": "2020-07-14T15:34:11",
"upload_time_iso_8601": "2020-07-14T15:34:11.396055Z",
"url": "https://files.pythonhosted.org/packages/fe/5e/3103b1c63c6cc2a0cdbb1c1e399f700501e1001675c700d39364f9f8df28/advocate-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-07-14 15:34:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JordanMilne",
"github_project": "Advocate",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "advocate"
}
JSON
