# Scrapy Requests Downloader Middleware
This package will make scrapy support requests. Everything is same with requests.
## Installation
```shell script
pip3 install scrapy-requests-manipulate
```
## Usage
After add this middleware, all requests will be sent by requests.
The usage is very simple, for request, specify params in meta.
### Settings for Request
```python
params = {
'key1': 'value1',
'key2': 'value2',
}
data = {
'key1': 'value1',
'key2': 'value2',
}
# turn cookie jar into dict, and remove the " mark, use ' mark
cookies = {
'key1': 'value1',
'key2': 'value2',
}
payload = {
'key1': 'value1',
'key2': 'value2'
}
proxy_ = 'http://username:password@ip:port' # https also works
or
proxy_ = [
'http://username:password@ip:port',
'http://username:password@ip:port',
] # if the type of proxy is list, every request will get a random proxy in the list
meta_data = {
'params': params,
'data': data,
'cookies': cookies,
'json': payload,
'proxy_': proxy_
}
or
meta_data = {
'params': json.dumps(params),
'data': json.dumps(data),
'cookies': json.dumps(cookies),
'json': json.dumps(payload),
'proxy_': json.dumps(proxy_)
}
yield scrapy.Request(url=url, headers=headers, meta=meta_data)
```
And you also need to enable `RequestsDownloaderMiddleware` in `DOWNLOADER_MIDDLEWARES`:
```python
DOWNLOADER_MIDDLEWARES = {
'scrapy_requests_manipulate.downloaderMiddleware.RequestsDownloaderMiddleware': 543,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': None,
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/dylankeepon/ScrapyRequestsMiddleware.git",
"name": "scrapy-requests-manipulate",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7.0",
"maintainer_email": "",
"keywords": "",
"author": "Dylan Chen",
"author_email": "dylankeep@163.com",
"download_url": "https://files.pythonhosted.org/packages/cc/f2/debc7f6ec0340126ccfad80657b144df34f80fa6c70ebefa334e47fa9cc6/scrapy-requests-manipulate-0.0.2.tar.gz",
"platform": null,
"description": "# Scrapy Requests Downloader Middleware\r\n\r\nThis package will make scrapy support requests. Everything is same with requests.\r\n\r\n## Installation\r\n\r\n```shell script\r\npip3 install scrapy-requests-manipulate\r\n```\r\n\r\n## Usage\r\n\r\nAfter add this middleware, all requests will be sent by requests.\r\n\r\nThe usage is very simple, for request, specify params in meta.\r\n\r\n\r\n### Settings for Request\r\n\r\n```python\r\nparams = {\r\n 'key1': 'value1',\r\n 'key2': 'value2',\r\n}\r\ndata = {\r\n 'key1': 'value1',\r\n 'key2': 'value2',\r\n}\r\n# turn cookie jar into dict, and remove the \" mark, use ' mark\r\ncookies = {\r\n 'key1': 'value1',\r\n 'key2': 'value2',\r\n}\r\npayload = {\r\n 'key1': 'value1',\r\n 'key2': 'value2'\r\n}\r\nproxy_ = 'http://username:password@ip:port' # https also works\r\nor \r\nproxy_ = [\r\n 'http://username:password@ip:port',\r\n 'http://username:password@ip:port',\r\n] # if the type of proxy is list, every request will get a random proxy in the list\r\n\r\nmeta_data = {\r\n 'params': params,\r\n 'data': data,\r\n 'cookies': cookies,\r\n 'json': payload,\r\n 'proxy_': proxy_\r\n}\r\nor \r\nmeta_data = {\r\n 'params': json.dumps(params),\r\n 'data': json.dumps(data),\r\n 'cookies': json.dumps(cookies),\r\n 'json': json.dumps(payload),\r\n 'proxy_': json.dumps(proxy_)\r\n}\r\nyield scrapy.Request(url=url, headers=headers, meta=meta_data)\r\n```\r\n\r\nAnd you also need to enable `RequestsDownloaderMiddleware` in `DOWNLOADER_MIDDLEWARES`:\r\n\r\n```python\r\nDOWNLOADER_MIDDLEWARES = {\r\n 'scrapy_requests_manipulate.downloaderMiddleware.RequestsDownloaderMiddleware': 543,\r\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': None,\r\n}\r\n```\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "requests downloader middleware for scrapy, send request by requests.",
"version": "0.0.2",
"project_urls": {
"Homepage": "https://github.com/dylankeepon/ScrapyRequestsMiddleware.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e2d092f481b49b4b30871eedf91650c90eebdc28fd30843a37fbddbd1086ae6a",
"md5": "8ef90923046931414436ab71ff167598",
"sha256": "46893c00255906d2789d2c8efe8abf825ef115499f3ca1519cd39ee77acffec6"
},
"downloads": -1,
"filename": "scrapy_requests_manipulate-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8ef90923046931414436ab71ff167598",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.0",
"size": 6021,
"upload_time": "2024-03-06T02:19:17",
"upload_time_iso_8601": "2024-03-06T02:19:17.831409Z",
"url": "https://files.pythonhosted.org/packages/e2/d0/92f481b49b4b30871eedf91650c90eebdc28fd30843a37fbddbd1086ae6a/scrapy_requests_manipulate-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ccf2debc7f6ec0340126ccfad80657b144df34f80fa6c70ebefa334e47fa9cc6",
"md5": "3ffddd9d7b7f47d8b3f20f28790137fd",
"sha256": "1bd0119ad8f0a2c51ca9b6300cdf7cb2868e4cd77a4654bb9195cef4d0a435d1"
},
"downloads": -1,
"filename": "scrapy-requests-manipulate-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "3ffddd9d7b7f47d8b3f20f28790137fd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.0",
"size": 5576,
"upload_time": "2024-03-06T02:19:19",
"upload_time_iso_8601": "2024-03-06T02:19:19.553962Z",
"url": "https://files.pythonhosted.org/packages/cc/f2/debc7f6ec0340126ccfad80657b144df34f80fa6c70ebefa334e47fa9cc6/scrapy-requests-manipulate-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-06 02:19:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dylankeepon",
"github_project": "ScrapyRequestsMiddleware",
"github_not_found": true,
"lcname": "scrapy-requests-manipulate"
}
JSON
