# spider-brew-kit
A library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.
## install
```shell
pip install spider-brew-kit
```
## usage
### pipelines
#### mongo pipeline
A pipeline saved into MongoDB asynchronously with txmongo
use database
db.createUser(
{
user: "username",
pwd: "password",
roles: [ { role: "readWrite", db: "database" } ]
}
)
how use:
1. add to settings.py
```python
ITEM_PIPELINES = {
'scrapy_kit.pipelines.MongoPipeline': 300,
}
```
2. add mongo config to settings.py
```python
MONGO_URI = "mongodb://username:password@host:port"
MONGO_DATABASE_NAME = "database"
MONGO_COLLECTION_NAME = "collection"
```
### middlewares
#### proxy connection close middleware
Proxy close connection multiplexing middleware
Tunnel Proxy Dynamic Edition request found that the number of requests in the Personal Centre Tunnel Proxy Usage
Statistics is very small, which is seriously inconsistent with the real number of requests.
Moreover, there is no IP change when using Tunnel Broker Dynamic Edition.
The reason for this is that the tunnel sends requests that reuse previously established connections.
You need to add Connection: close to the header.
How to use it:
1. Add in settings.py:
```python
DOWNLOADER_MIDDLEWARES = {
'scrapy_kit.middlewares.ProxyConnectionCloseMiddleware': 543,
}
```
## development
```shell
git clone
cd spider-brew-kit
poetry install
```
Raw data
{
"_id": null,
"home_page": "https://spider-brew-kit.readthedocs.io/en/latest/",
"name": "spider-brew-kit",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "spider,scrapy,spider-tool-kit,spider-kit,spider-tools,spider-tool-kit",
"author": "Pony.Ma",
"author_email": "mtf201013@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/30/1d/fb8b6f340a4ea21cf8e1004aacee20e4d2ef50fb9c9ec215cf95979cafe1/spider_brew_kit-0.1.6.tar.gz",
"platform": null,
"description": "# spider-brew-kit\n\nA library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.\n\n## install\n\n```shell\npip install spider-brew-kit\n```\n\n## usage\n\n### pipelines\n\n#### mongo pipeline\n\nA pipeline saved into MongoDB asynchronously with txmongo\n\nuse database\ndb.createUser(\n{\nuser: \"username\",\npwd: \"password\",\nroles: [ { role: \"readWrite\", db: \"database\" } ]\n}\n)\n\nhow use:\n\n1. add to settings.py\n\n```python\nITEM_PIPELINES = {\n 'scrapy_kit.pipelines.MongoPipeline': 300,\n}\n```\n\n2. add mongo config to settings.py\n\n```python\nMONGO_URI = \"mongodb://username:password@host:port\"\nMONGO_DATABASE_NAME = \"database\"\nMONGO_COLLECTION_NAME = \"collection\"\n```\n\n### middlewares\n\n#### proxy connection close middleware\n\nProxy close connection multiplexing middleware\n\nTunnel Proxy Dynamic Edition request found that the number of requests in the Personal Centre Tunnel Proxy Usage\nStatistics is very small, which is seriously inconsistent with the real number of requests.\nMoreover, there is no IP change when using Tunnel Broker Dynamic Edition.\nThe reason for this is that the tunnel sends requests that reuse previously established connections.\nYou need to add Connection: close to the header.\n\nHow to use it:\n\n1. Add in settings.py:\n\n```python\nDOWNLOADER_MIDDLEWARES = {\n 'scrapy_kit.middlewares.ProxyConnectionCloseMiddleware': 543,\n}\n\n```\n\n## development\n\n```shell\ngit clone\ncd spider-brew-kit\npoetry install\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "",
"version": "0.1.6",
"project_urls": {
"Documentation": "https://spider-brew-kit.readthedocs.io/en/latest/",
"Homepage": "https://spider-brew-kit.readthedocs.io/en/latest/",
"Repository": "https://github.com/ma-pony/spider-tool-kit"
},
"split_keywords": [
"spider",
"scrapy",
"spider-tool-kit",
"spider-kit",
"spider-tools",
"spider-tool-kit"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fed8bc8b17f5995b386aa44de21dac4827f9ed83a010ac4af90fd27cebb6199a",
"md5": "ae446f162f306c0a7352517c0b875fca",
"sha256": "ca0fb973348b35c104b5c45875730fe1fa5565ad31b8a1534c201ecdb4647026"
},
"downloads": -1,
"filename": "spider_brew_kit-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ae446f162f306c0a7352517c0b875fca",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 14055,
"upload_time": "2024-02-18T03:21:28",
"upload_time_iso_8601": "2024-02-18T03:21:28.704568Z",
"url": "https://files.pythonhosted.org/packages/fe/d8/bc8b17f5995b386aa44de21dac4827f9ed83a010ac4af90fd27cebb6199a/spider_brew_kit-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "301dfb8b6f340a4ea21cf8e1004aacee20e4d2ef50fb9c9ec215cf95979cafe1",
"md5": "0278b4fe8e1711b845b9c840cd149917",
"sha256": "a04c473e82b93df973fa9c7f64f7dd5f7c77dc477eae89afb560f450bf8c5a3b"
},
"downloads": -1,
"filename": "spider_brew_kit-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "0278b4fe8e1711b845b9c840cd149917",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 12217,
"upload_time": "2024-02-18T03:21:29",
"upload_time_iso_8601": "2024-02-18T03:21:29.778501Z",
"url": "https://files.pythonhosted.org/packages/30/1d/fb8b6f340a4ea21cf8e1004aacee20e4d2ef50fb9c9ec215cf95979cafe1/spider_brew_kit-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-18 03:21:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ma-pony",
"github_project": "spider-tool-kit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "spider-brew-kit"
}
JSON
