# cloudscraper
[](https://badge.fury.io/py/cloudscraper)
[](https://opensource.org/licenses/MIT)
[](https://pypi.org/project/cloudscraper/)
[](https://travis-ci.com/VeNoMouS/cloudscraper)
[](https://www.buymeacoffee.com/venomous)
A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with [Requests](https://github.com/kennethreitz/requests). Cloudflare changes their techniques periodically, so I will update this repo frequently.
This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports Javascript, though they may add additional techniques in the future.
Due to Cloudflare continually changing and hardening their protection page, cloudscraper requires a JavaScript Engine/interpreter to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript.
For reference, this is the default message Cloudflare uses for these sorts of pages:
```
Checking your browser before accessing website.com.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds...
```
Any script using cloudscraper will sleep for ~5 seconds for the first visit to any site with Cloudflare anti-bots enabled, though no delay will occur after the first request.
# Donations
If you feel like showing your love and/or appreciation for this project, then how about shouting me a coffee or beer :)
<a href="https://buymeacoff.ee/venomous" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>
# Installation
Simply run `pip install cloudscraper`. The PyPI package is at https://pypi.python.org/pypi/cloudscraper/
Alternatively, clone this repository and run `python setup.py install`.
# Dependencies
- Python 3.x
- **[Requests](https://github.com/kennethreitz/requests)** >= 2.9.2
- **[requests_toolbelt](https://pypi.org/project/requests-toolbelt/)** >= 0.9.1
`python setup.py install` will install the Python dependencies automatically. The javascript interpreters and/or engines you decide to use are the only things you need to install yourself, excluding js2py which is part of the requirements as the default.
# Javascript Interpreters and Engines
We support the following Javascript interpreters/engines.
- **[ChakraCore](https://github.com/microsoft/ChakraCore):** Library binaries can also be located [here](https://www.github.com/VeNoMouS/cloudscraper/tree/ChakraCore/).
- **[js2py](https://github.com/PiotrDabkowski/Js2Py):** >=0.67
- **native**: Self made native python solver **(Default)**
- **[Node.js](https://nodejs.org/)**
- **[V8](https://github.com/sony/v8eval/):** We use Sony's [v8eval](https://v8.dev)() python module.
# Usage
The simplest way to use cloudscraper is by calling `create_scraper()`.
```python
import cloudscraper
scraper = cloudscraper.create_scraper() # returns a CloudScraper instance
# Or: scraper = cloudscraper.CloudScraper() # CloudScraper inherits from requests.Session
print(scraper.get("http://somesite.com").text) # => "<!DOCTYPE html><html><head>..."
```
That's it...
Any requests made from this session object to websites protected by Cloudflare anti-bot will be handled automatically. Websites not using Cloudflare will be treated normally. You don't need to configure or call anything further, and you can effectively treat all websites as if they're not protected with anything.
You use cloudscraper exactly the same way you use Requests. `cloudScraper` works identically to a Requests `Session` object, just instead of calling `requests.get()` or `requests.post()`, you call `scraper.get()` or `scraper.post()`.
Consult [Requests' documentation](http://docs.python-requests.org/en/latest/user/quickstart/) for more information.
## Options
### Disable Cloudflare V1
#### Description
If you don't want to even attempt Cloudflare v1 (Deprecated) solving..
#### Parameters
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|disableCloudflareV1|(boolean)|False|
#### Example
```python
scraper = cloudscraper.create_scraper(disableCloudflareV1=True)
```
------
### Brotli
#### Description
[Brotli](https://en.wikipedia.org/wiki/Brotli) decompression support has been added, and it is enabled by default.
#### Parameters
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|allow_brotli|(boolean)|True|
#### Example
```python
scraper = cloudscraper.create_scraper(allow_brotli=False)
```
------
### Browser / User-Agent Filtering
#### Description
Control how and which User-Agent is "randomly" selected.
#### Parameters
Can be passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|browser|(string) `chrome` or `firefox`|None|
Or
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|browser|(dict)||
##### `browser` *_dict_* Parameters
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|browser|(string) `chrome` or `firefox`|None|
|mobile|(boolean)|True|
|desktop|(boolean)|True|
|platform|(string) `'linux', 'windows', 'darwin', 'android', 'ios'`|None|
|custom|(string)|None|
#### Example
```python
scraper = cloudscraper.create_scraper(browser='chrome')
```
or
```python
# will give you only mobile chrome User-Agents on Android
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'android',
'desktop': False
}
)
# will give you only desktop firefox User-Agents on Windows
scraper = cloudscraper.create_scraper(
browser={
'browser': 'firefox',
'platform': 'windows',
'mobile': False
}
)
# Custom will also try find the user-agent string in the browsers.json,
# If a match is found, it will use the headers and cipherSuite from that "browser",
# Otherwise a generic set of headers and cipherSuite will be used.
scraper = cloudscraper.create_scraper(
browser={
'custom': 'ScraperBot/1.0',
}
)
```
------
### Debug
#### Description
Prints out header and content information of the request for debugging.
#### Parameters
Can be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|debug|(boolean)|False|
#### Example
```python
scraper = cloudscraper.create_scraper(debug=True)
```
------
### Delays
#### Description
Cloudflare IUAM challenge requires the browser to wait ~5 seconds before submitting the challenge answer, If you would like to override this delay.
#### Parameters
Can be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|delay|(float)|extracted from IUAM page|
#### Example
```python
scraper = cloudscraper.create_scraper(delay=10)
```
------
### Existing session
#### Description:
If you already have an existing Requests session, you can pass it to the function `create_scraper()` to continue using that session.
#### Parameters
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|sess|(requests.session)|None|
#### Example
```python
session = requests.session()
scraper = cloudscraper.create_scraper(sess=session)
```
#### Note
Unfortunately, not all of Requests session attributes are easily transferable, so if you run into problems with this,
You should replace your initial session initialization call
From:
```python
sess = requests.session()
```
To:
```python
sess = cloudscraper.create_scraper()
```
------
### JavaScript Engines and Interpreters
#### Description
cloudscraper currently supports the following JavaScript Engines/Interpreters
- **[ChakraCore](https://github.com/microsoft/ChakraCore)**
- **[js2py](https://github.com/PiotrDabkowski/Js2Py)**
- **native**: Self made native python solver **(Default)**
- **[Node.js](https://nodejs.org/)**
- **[V8](https://github.com/sony/v8eval/)**
#### Parameters
Can be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|interpreter|(string)|`native`|
#### Example
```python
scraper = cloudscraper.create_scraper(interpreter='nodejs')
```
------
### 3rd Party Captcha Solvers
#### Description
`cloudscraper` currently supports the following 3rd party Captcha solvers, should you require them.
- **[2captcha](https://www.2captcha.com/)**
- **[anticaptcha](https://www.anti-captcha.com/)**
- **[CapSolver](https://capsolver.com/)**
- **[CapMonster Cloud](https://capmonster.cloud/)**
- **[deathbycaptcha](https://www.deathbycaptcha.com/)**
- **[9kw](https://www.9kw.eu/)**
- **__return_response__**
#### Note
I am working on adding more 3rd party solvers, if you wish to have a service added that is not currently supported, please raise a support ticket on github.
##### Required Parameters
Can be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|captcha|(dict)|None|
------
#### 2captcha
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `2captcha`| yes||
|api_key|(string)| yes||
|no_proxy|(boolean)|no|False|
##### Note
if proxies are set you can disable sending the proxies to 2captcha by setting `no_proxy` to `True`
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': '2captcha',
'api_key': 'your_2captcha_api_key'
}
)
```
------
#### anticaptcha
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `anticaptcha`|yes||
|api_key|(string)|yes||
|no_proxy|(boolean)|no|False|
##### Note
if proxies are set you can disable sending the proxies to anticaptcha by setting `no_proxy` to `True`
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'anticaptcha',
'api_key': 'your_anticaptcha_api_key'
}
)
```
------
#### CapSolver
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `captchaai`|yes||
|api_key|(string)|yes||
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'capsolver',
'api_key': 'your_captchaai_api_key'
}
)
```
------
#### CapMonster Cloud
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `capmonster`| yes||
|clientKey|(string)| yes||
|no_proxy|(boolean)|no|False|
##### Note
if proxies are set you can disable sending the proxies to CapMonster by setting `no_proxy` to `True`
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'capmonster',
'clientKey': 'your_capmonster_clientKey'
}
)
```
------
#### deathbycaptcha
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `deathbycaptcha`|yes||
|username|(string)|yes||
|password|(string)|yes||
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'deathbycaptcha',
'username': 'your_deathbycaptcha_username',
'password': 'your_deathbycaptcha_password',
}
)
```
------
#### 9kw
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `9kw`|yes||
|api_key|(string)|yes||
|maxtimeout|(int)|no|180|
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={
'provider': '9kw',
'api_key': 'your_9kw_api_key',
'maxtimeout': 300
}
)
```
------
#### return_response
Use this if you want the requests response payload without solving the Captcha.
##### Required `captcha` Parameters
|Parameter|Value|Required|Default|
|-------------|:-------------:|:-----:|:-----:|
|provider|(string) `return_response`| yes||
##### Example
```python
scraper = cloudscraper.create_scraper(
captcha={'provider': 'return_response'}
)
```
## Integration
It's easy to integrate `cloudscraper` with other applications and tools. Cloudflare uses two cookies as tokens: one to verify you made it past their challenge page and one to track your session. To bypass the challenge page, simply include both of these cookies (with the appropriate user-agent) in all HTTP requests you make.
To retrieve just the cookies (as a dictionary), use `cloudscraper.get_tokens()`. To retrieve them as a full `Cookie` HTTP header, use `cloudscraper.get_cookie_string()`.
`get_tokens` and `get_cookie_string` both accept Requests' usual keyword arguments (like `get_tokens(url, proxies={"http": "socks5://localhost:9050"})`).
Please read [Requests' documentation on request arguments](http://docs.python-requests.org/en/master/api/#requests.Session.request) for more information.
------
### User-Agent Handling
The two integration functions return a tuple of `(cookie, user_agent_string)`.
**You must use the same user-agent string for obtaining tokens and for making requests with those tokens, otherwise Cloudflare will flag you as a bot.**
That means you have to pass the returned `user_agent_string` to whatever script, tool, or service you are passing the tokens to (e.g. curl, or a specialized scraping tool), and it must use that passed user-agent when it makes HTTP requests.
------
### Integration examples
Remember, you must always use the same user-agent when retrieving or using these cookies. These functions all return a tuple of `(cookie_dict, user_agent_string)`.
------
#### Retrieving a cookie dict through a proxy
`get_tokens` is a convenience function for returning a Python dict containing Cloudflare's session cookies. For demonstration, we will configure this request to use a proxy. (Please note that if you request Cloudflare clearance tokens through a proxy, you must always use the same proxy when those tokens are passed to the server. Cloudflare requires that the challenge-solving IP and the visitor IP stay the same.)
If you do not wish to use a proxy, just don't pass the `proxies` keyword argument. These convenience functions support all of Requests' normal keyword arguments, like `params`, `data`, and `headers`.
```python
import cloudscraper
proxies = {"http": "http://localhost:8080", "https": "http://localhost:8080"}
tokens, user_agent = cloudscraper.get_tokens("http://somesite.com", proxies=proxies)
print(tokens)
# => {
'cf_clearance': 'c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600',
'__cfduid': 'dd8ec03dfdbcb8c2ea63e920f1335c1001426733158'
}
```
------
#### Retrieving a cookie string
`get_cookie_string` is a convenience function for returning the tokens as a string for use as a `Cookie` HTTP header value.
This is useful when crafting an HTTP request manually, or working with an external application or library that passes on raw cookie headers.
```python
import cloudscraper
cookie_value, user_agent = cloudscraper.get_cookie_string('http://somesite.com')
print('GET / HTTP/1.1\nCookie: {}\nUser-Agent: {}\n'.format(cookie_value, user_agent))
# GET / HTTP/1.1
# Cookie: cf_clearance=c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600; __cfduid=dd8ec03dfdbcb8c2ea63e920f1335c1001426733158
# User-Agent: Some/User-Agent String
```
------
#### curl example
Here is an example of integrating cloudscraper with curl. As you can see, all you have to do is pass the cookies and user-agent to curl.
```python
import subprocess
import cloudscraper
# With get_tokens() cookie dict:
# tokens, user_agent = cloudscraper.get_tokens("http://somesite.com")
# cookie_arg = 'cf_clearance={}; __cfduid={}'.format(tokens['cf_clearance'], tokens['__cfduid'])
# With get_cookie_string() cookie header; recommended for curl and similar external applications:
cookie_arg, user_agent = cloudscraper.get_cookie_string('http://somesite.com')
# With a custom user-agent string you can optionally provide:
# ua = "Scraping Bot"
# cookie_arg, user_agent = cloudscraper.get_cookie_string("http://somesite.com", user_agent=ua)
result = subprocess.check_output(
[
'curl',
'--cookie',
cookie_arg,
'-A',
user_agent,
'http://somesite.com'
]
)
```
Trimmed down version. Prints page contents of any site protected with Cloudflare, via curl.
**Warning: `shell=True` can be dangerous to use with `subprocess` in real code.**
```python
url = "http://somesite.com"
cookie_arg, user_agent = cloudscraper.get_cookie_string(url)
cmd = "curl --cookie {cookie_arg} -A {user_agent} {url}"
print(
subprocess.check_output(
cmd.format(
cookie_arg=cookie_arg,
user_agent=user_agent,
url=url
),
shell=True
)
)
```
### Cryptography
#### Description
Control communication between client and server
#### Parameters
Can be passed as an argument to `create_scraper()`.
|Parameter|Value|Default|
|-------------|:-------------:|:-----:|
|cipherSuite|(string)|None|
|ecdhCurve|(string)|prime256v1|
|server_hostname|(string)|None|
#### Example
```python
# Some servers require the use of a more complex ecdh curve than the default "prime256v1"
# It may can solve handshake failure
scraper = cloudscraper.create_scraper(ecdhCurve='secp384r1')
```
```python
# Manipulate server_hostname
scraper = cloudscraper.create_scraper(server_hostname='www.somesite.com')
scraper.get(
'https://backend.hosting.com/',
headers={'Host': 'www.somesite.com'}
)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/venomous/cloudscraper",
"name": "cloudscraper",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "cloudflare,scraping,ddos,scrape,webscraper,anti-bot,waf,iuam,bypass,challenge",
"author": "VeNoMouS",
"author_email": "venom@gen-x.co.nz",
"download_url": "https://files.pythonhosted.org/packages/ac/25/6d0481860583f44953bd791de0b7c4f6d7ead7223f8a17e776247b34a5b4/cloudscraper-1.2.71.tar.gz",
"platform": null,
"description": "# cloudscraper\n\n[](https://badge.fury.io/py/cloudscraper)\n[](https://opensource.org/licenses/MIT)\n[](https://pypi.org/project/cloudscraper/)\n[](https://travis-ci.com/VeNoMouS/cloudscraper)\n[](https://www.buymeacoffee.com/venomous)\n\nA simple Python module to bypass Cloudflare's anti-bot page (also known as \"I'm Under Attack Mode\", or IUAM), implemented with [Requests](https://github.com/kennethreitz/requests). Cloudflare changes their techniques periodically, so I will update this repo frequently.\n\nThis can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports Javascript, though they may add additional techniques in the future.\n\nDue to Cloudflare continually changing and hardening their protection page, cloudscraper requires a JavaScript Engine/interpreter to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript.\n\nFor reference, this is the default message Cloudflare uses for these sorts of pages:\n\n```\nChecking your browser before accessing website.com.\n\nThis process is automatic. Your browser will redirect to your requested content shortly.\n\nPlease allow up to 5 seconds...\n```\n\nAny script using cloudscraper will sleep for ~5 seconds for the first visit to any site with Cloudflare anti-bots enabled, though no delay will occur after the first request.\n\n# Donations\n\nIf you feel like showing your love and/or appreciation for this project, then how about shouting me a coffee or beer :)\n\n<a href=\"https://buymeacoff.ee/venomous\" target=\"_blank\"><img src=\"https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png\" alt=\"Buy Me A Coffee\" style=\"height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;\" ></a>\n\n# Installation\n\nSimply run `pip install cloudscraper`. The PyPI package is at https://pypi.python.org/pypi/cloudscraper/\n\nAlternatively, clone this repository and run `python setup.py install`.\n\n# Dependencies\n\n- Python 3.x\n- **[Requests](https://github.com/kennethreitz/requests)** >= 2.9.2\n- **[requests_toolbelt](https://pypi.org/project/requests-toolbelt/)** >= 0.9.1\n\n`python setup.py install` will install the Python dependencies automatically. The javascript interpreters and/or engines you decide to use are the only things you need to install yourself, excluding js2py which is part of the requirements as the default.\n\n# Javascript Interpreters and Engines\n\nWe support the following Javascript interpreters/engines.\n\n- **[ChakraCore](https://github.com/microsoft/ChakraCore):** Library binaries can also be located [here](https://www.github.com/VeNoMouS/cloudscraper/tree/ChakraCore/).\n- **[js2py](https://github.com/PiotrDabkowski/Js2Py):** >=0.67\n- **native**: Self made native python solver **(Default)**\n- **[Node.js](https://nodejs.org/)**\n- **[V8](https://github.com/sony/v8eval/):** We use Sony's [v8eval](https://v8.dev)() python module.\n\n# Usage\n\nThe simplest way to use cloudscraper is by calling `create_scraper()`.\n\n```python\nimport cloudscraper\n\nscraper = cloudscraper.create_scraper() # returns a CloudScraper instance\n# Or: scraper = cloudscraper.CloudScraper() # CloudScraper inherits from requests.Session\nprint(scraper.get(\"http://somesite.com\").text) # => \"<!DOCTYPE html><html><head>...\"\n```\n\nThat's it...\n\nAny requests made from this session object to websites protected by Cloudflare anti-bot will be handled automatically. Websites not using Cloudflare will be treated normally. You don't need to configure or call anything further, and you can effectively treat all websites as if they're not protected with anything.\n\nYou use cloudscraper exactly the same way you use Requests. `cloudScraper` works identically to a Requests `Session` object, just instead of calling `requests.get()` or `requests.post()`, you call `scraper.get()` or `scraper.post()`.\n\nConsult [Requests' documentation](http://docs.python-requests.org/en/latest/user/quickstart/) for more information.\n\n## Options\n\n### Disable Cloudflare V1\n#### Description\n\nIf you don't want to even attempt Cloudflare v1 (Deprecated) solving..\n\n#### Parameters\n\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|disableCloudflareV1|(boolean)|False|\n\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(disableCloudflareV1=True)\n```\n\n------\n\n### Brotli\n\n#### Description\n\n[Brotli](https://en.wikipedia.org/wiki/Brotli) decompression support has been added, and it is enabled by default.\n\n#### Parameters\n\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|allow_brotli|(boolean)|True|\n\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(allow_brotli=False)\n```\n\n------\n\n### Browser / User-Agent Filtering\n\n#### Description\n\nControl how and which User-Agent is \"randomly\" selected.\n\n#### Parameters\n\nCan be passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|browser|(string) `chrome` or `firefox`|None|\n\nOr\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|browser|(dict)||\n\n##### `browser` *_dict_* Parameters\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|browser|(string) `chrome` or `firefox`|None|\n|mobile|(boolean)|True|\n|desktop|(boolean)|True|\n|platform|(string) `'linux', 'windows', 'darwin', 'android', 'ios'`|None|\n|custom|(string)|None|\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(browser='chrome')\n```\n\nor\n\n```python\n# will give you only mobile chrome User-Agents on Android\nscraper = cloudscraper.create_scraper(\n browser={\n 'browser': 'chrome',\n 'platform': 'android',\n 'desktop': False\n }\n)\n\n# will give you only desktop firefox User-Agents on Windows\nscraper = cloudscraper.create_scraper(\n browser={\n 'browser': 'firefox',\n 'platform': 'windows',\n 'mobile': False\n }\n)\n\n# Custom will also try find the user-agent string in the browsers.json,\n# If a match is found, it will use the headers and cipherSuite from that \"browser\",\n# Otherwise a generic set of headers and cipherSuite will be used.\nscraper = cloudscraper.create_scraper(\n browser={\n 'custom': 'ScraperBot/1.0',\n }\n)\n```\n------\n\n### Debug\n\n#### Description\n\nPrints out header and content information of the request for debugging.\n\n#### Parameters\n\nCan be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|debug|(boolean)|False|\n\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(debug=True)\n```\n\n------\n\n### Delays\n\n#### Description\n\nCloudflare IUAM challenge requires the browser to wait ~5 seconds before submitting the challenge answer, If you would like to override this delay.\n\n#### Parameters\n\nCan be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|delay|(float)|extracted from IUAM page|\n\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(delay=10)\n```\n\n------\n\n### Existing session\n\n#### Description:\n\nIf you already have an existing Requests session, you can pass it to the function `create_scraper()` to continue using that session.\n\n#### Parameters\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|sess|(requests.session)|None|\n\n#### Example\n\n```python\nsession = requests.session()\nscraper = cloudscraper.create_scraper(sess=session)\n```\n\n#### Note\n\nUnfortunately, not all of Requests session attributes are easily transferable, so if you run into problems with this,\n\nYou should replace your initial session initialization call\n\nFrom:\n```python\nsess = requests.session()\n```\n\nTo:\n\n```python\nsess = cloudscraper.create_scraper()\n```\n\n------\n\n### JavaScript Engines and Interpreters\n\n#### Description\ncloudscraper currently supports the following JavaScript Engines/Interpreters\n\n- **[ChakraCore](https://github.com/microsoft/ChakraCore)**\n- **[js2py](https://github.com/PiotrDabkowski/Js2Py)**\n- **native**: Self made native python solver **(Default)**\n- **[Node.js](https://nodejs.org/)**\n- **[V8](https://github.com/sony/v8eval/)**\n\n\n#### Parameters\nCan be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|interpreter|(string)|`native`|\n\n#### Example\n\n```python\nscraper = cloudscraper.create_scraper(interpreter='nodejs')\n```\n\n------\n\n### 3rd Party Captcha Solvers\n\n#### Description\n`cloudscraper` currently supports the following 3rd party Captcha solvers, should you require them.\n\n- **[2captcha](https://www.2captcha.com/)**\n- **[anticaptcha](https://www.anti-captcha.com/)**\n- **[CapSolver](https://capsolver.com/)**\n- **[CapMonster Cloud](https://capmonster.cloud/)**\n- **[deathbycaptcha](https://www.deathbycaptcha.com/)**\n- **[9kw](https://www.9kw.eu/)**\n- **__return_response__**\n\n#### Note\n\nI am working on adding more 3rd party solvers, if you wish to have a service added that is not currently supported, please raise a support ticket on github.\n\n##### Required Parameters\n\nCan be set as an attribute via your `cloudscraper` object or passed as an argument to `create_scraper()`, `get_tokens()`, `get_cookie_string()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|captcha|(dict)|None|\n\n------\n\n#### 2captcha\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `2captcha`| yes||\n|api_key|(string)| yes||\n|no_proxy|(boolean)|no|False|\n\n##### Note\n\nif proxies are set you can disable sending the proxies to 2captcha by setting `no_proxy` to `True`\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': '2captcha',\n 'api_key': 'your_2captcha_api_key'\n }\n)\n```\n\n------\n\n#### anticaptcha\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `anticaptcha`|yes||\n|api_key|(string)|yes||\n|no_proxy|(boolean)|no|False|\n\n##### Note\n\nif proxies are set you can disable sending the proxies to anticaptcha by setting `no_proxy` to `True`\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': 'anticaptcha',\n 'api_key': 'your_anticaptcha_api_key'\n }\n)\n```\n\n------\n\n#### CapSolver\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `captchaai`|yes||\n|api_key|(string)|yes||\n\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': 'capsolver',\n 'api_key': 'your_captchaai_api_key'\n }\n)\n```\n\n------\n\n#### CapMonster Cloud\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `capmonster`| yes||\n|clientKey|(string)| yes||\n|no_proxy|(boolean)|no|False|\n\n##### Note\n\nif proxies are set you can disable sending the proxies to CapMonster by setting `no_proxy` to `True`\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': 'capmonster',\n 'clientKey': 'your_capmonster_clientKey'\n }\n)\n```\n\n------\n\n#### deathbycaptcha\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `deathbycaptcha`|yes||\n|username|(string)|yes||\n|password|(string)|yes||\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': 'deathbycaptcha',\n 'username': 'your_deathbycaptcha_username',\n 'password': 'your_deathbycaptcha_password',\n }\n)\n```\n\n------\n\n#### 9kw\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `9kw`|yes||\n|api_key|(string)|yes||\n|maxtimeout|(int)|no|180|\n\n##### Example\n\n```python\nscraper = cloudscraper.create_scraper(\n captcha={\n 'provider': '9kw',\n 'api_key': 'your_9kw_api_key',\n 'maxtimeout': 300\n }\n)\n```\n\n------\n\n#### return_response\n\nUse this if you want the requests response payload without solving the Captcha.\n\n##### Required `captcha` Parameters\n\n|Parameter|Value|Required|Default|\n|-------------|:-------------:|:-----:|:-----:|\n|provider|(string) `return_response`| yes||\n\n##### Example\n```python\nscraper = cloudscraper.create_scraper(\n captcha={'provider': 'return_response'}\n)\n```\n\n## Integration\n\nIt's easy to integrate `cloudscraper` with other applications and tools. Cloudflare uses two cookies as tokens: one to verify you made it past their challenge page and one to track your session. To bypass the challenge page, simply include both of these cookies (with the appropriate user-agent) in all HTTP requests you make.\n\nTo retrieve just the cookies (as a dictionary), use `cloudscraper.get_tokens()`. To retrieve them as a full `Cookie` HTTP header, use `cloudscraper.get_cookie_string()`.\n\n`get_tokens` and `get_cookie_string` both accept Requests' usual keyword arguments (like `get_tokens(url, proxies={\"http\": \"socks5://localhost:9050\"})`).\n\nPlease read [Requests' documentation on request arguments](http://docs.python-requests.org/en/master/api/#requests.Session.request) for more information.\n\n------\n\n### User-Agent Handling\n\nThe two integration functions return a tuple of `(cookie, user_agent_string)`.\n\n**You must use the same user-agent string for obtaining tokens and for making requests with those tokens, otherwise Cloudflare will flag you as a bot.**\n\nThat means you have to pass the returned `user_agent_string` to whatever script, tool, or service you are passing the tokens to (e.g. curl, or a specialized scraping tool), and it must use that passed user-agent when it makes HTTP requests.\n\n------\n\n### Integration examples\n\nRemember, you must always use the same user-agent when retrieving or using these cookies. These functions all return a tuple of `(cookie_dict, user_agent_string)`.\n\n------\n\n#### Retrieving a cookie dict through a proxy\n\n`get_tokens` is a convenience function for returning a Python dict containing Cloudflare's session cookies. For demonstration, we will configure this request to use a proxy. (Please note that if you request Cloudflare clearance tokens through a proxy, you must always use the same proxy when those tokens are passed to the server. Cloudflare requires that the challenge-solving IP and the visitor IP stay the same.)\n\nIf you do not wish to use a proxy, just don't pass the `proxies` keyword argument. These convenience functions support all of Requests' normal keyword arguments, like `params`, `data`, and `headers`.\n\n```python\nimport cloudscraper\n\nproxies = {\"http\": \"http://localhost:8080\", \"https\": \"http://localhost:8080\"}\ntokens, user_agent = cloudscraper.get_tokens(\"http://somesite.com\", proxies=proxies)\nprint(tokens)\n# => {\n 'cf_clearance': 'c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600',\n '__cfduid': 'dd8ec03dfdbcb8c2ea63e920f1335c1001426733158'\n}\n```\n\n------\n\n#### Retrieving a cookie string\n\n`get_cookie_string` is a convenience function for returning the tokens as a string for use as a `Cookie` HTTP header value.\n\nThis is useful when crafting an HTTP request manually, or working with an external application or library that passes on raw cookie headers.\n\n```python\nimport cloudscraper\n\ncookie_value, user_agent = cloudscraper.get_cookie_string('http://somesite.com')\n\nprint('GET / HTTP/1.1\\nCookie: {}\\nUser-Agent: {}\\n'.format(cookie_value, user_agent))\n\n# GET / HTTP/1.1\n# Cookie: cf_clearance=c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600; __cfduid=dd8ec03dfdbcb8c2ea63e920f1335c1001426733158\n# User-Agent: Some/User-Agent String\n```\n\n------\n\n#### curl example\n\nHere is an example of integrating cloudscraper with curl. As you can see, all you have to do is pass the cookies and user-agent to curl.\n\n```python\nimport subprocess\nimport cloudscraper\n\n# With get_tokens() cookie dict:\n\n# tokens, user_agent = cloudscraper.get_tokens(\"http://somesite.com\")\n# cookie_arg = 'cf_clearance={}; __cfduid={}'.format(tokens['cf_clearance'], tokens['__cfduid'])\n\n# With get_cookie_string() cookie header; recommended for curl and similar external applications:\n\ncookie_arg, user_agent = cloudscraper.get_cookie_string('http://somesite.com')\n\n# With a custom user-agent string you can optionally provide:\n\n# ua = \"Scraping Bot\"\n# cookie_arg, user_agent = cloudscraper.get_cookie_string(\"http://somesite.com\", user_agent=ua)\n\nresult = subprocess.check_output(\n [\n 'curl',\n '--cookie',\n cookie_arg,\n '-A',\n user_agent,\n 'http://somesite.com'\n ]\n)\n```\n\nTrimmed down version. Prints page contents of any site protected with Cloudflare, via curl.\n\n**Warning: `shell=True` can be dangerous to use with `subprocess` in real code.**\n\n```python\nurl = \"http://somesite.com\"\ncookie_arg, user_agent = cloudscraper.get_cookie_string(url)\ncmd = \"curl --cookie {cookie_arg} -A {user_agent} {url}\"\nprint(\n subprocess.check_output(\n cmd.format(\n cookie_arg=cookie_arg,\n user_agent=user_agent,\n url=url\n ),\n shell=True\n )\n)\n```\n\n### Cryptography\n\n#### Description\n\nControl communication between client and server\n\n#### Parameters\n\nCan be passed as an argument to `create_scraper()`.\n\n|Parameter|Value|Default|\n|-------------|:-------------:|:-----:|\n|cipherSuite|(string)|None|\n|ecdhCurve|(string)|prime256v1|\n|server_hostname|(string)|None|\n\n#### Example\n\n```python\n# Some servers require the use of a more complex ecdh curve than the default \"prime256v1\"\n# It may can solve handshake failure\nscraper = cloudscraper.create_scraper(ecdhCurve='secp384r1')\n```\n\n```python\n# Manipulate server_hostname\nscraper = cloudscraper.create_scraper(server_hostname='www.somesite.com')\nscraper.get(\n 'https://backend.hosting.com/',\n headers={'Host': 'www.somesite.com'}\n)\n```\n\n",
"bugtrack_url": null,
"license": "",
"summary": "A Python module to bypass Cloudflare's anti-bot page.",
"version": "1.2.71",
"split_keywords": [
"cloudflare",
"scraping",
"ddos",
"scrape",
"webscraper",
"anti-bot",
"waf",
"iuam",
"bypass",
"challenge"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8197fc88803a451029688dffd7eb446dc1b529657577aec13aceff1cc9628c5d",
"md5": "b2fe568e6e401ff7bc6feaf33f82133f",
"sha256": "76f50ca529ed2279e220837befdec892626f9511708e200d48d5bb76ded679b0"
},
"downloads": -1,
"filename": "cloudscraper-1.2.71-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "b2fe568e6e401ff7bc6feaf33f82133f",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 99652,
"upload_time": "2023-04-25T23:20:15",
"upload_time_iso_8601": "2023-04-25T23:20:15.974618Z",
"url": "https://files.pythonhosted.org/packages/81/97/fc88803a451029688dffd7eb446dc1b529657577aec13aceff1cc9628c5d/cloudscraper-1.2.71-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ac256d0481860583f44953bd791de0b7c4f6d7ead7223f8a17e776247b34a5b4",
"md5": "e90af53f2a5b8e4b633285054b0ddeaa",
"sha256": "429c6e8aa6916d5bad5c8a5eac50f3ea53c9ac22616f6cb21b18dcc71517d0d3"
},
"downloads": -1,
"filename": "cloudscraper-1.2.71.tar.gz",
"has_sig": false,
"md5_digest": "e90af53f2a5b8e4b633285054b0ddeaa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 93261,
"upload_time": "2023-04-25T23:20:19",
"upload_time_iso_8601": "2023-04-25T23:20:19.467461Z",
"url": "https://files.pythonhosted.org/packages/ac/25/6d0481860583f44953bd791de0b7c4f6d7ead7223f8a17e776247b34a5b4/cloudscraper-1.2.71.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-25 23:20:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "venomous",
"github_project": "cloudscraper",
"travis_ci": true,
"coveralls": true,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "cloudscraper"
}
JSON
