gerapy-rabbitmq

Name	gerapy-rabbitmq JSON
Version	0.1.1 JSON
	download
home_page	https://github.com/Gerapy/GerapyRabbitMQ
Summary	Distribution Support for Scrapy & Gerapy using RabbitMQ
upload_time	2023-11-27 15:52:54
maintainer
docs_url	None
author	Germey
requires_python	>=3.5.0
license	MIT
keywords
VCS
bugtrack_url
requirements	pika gerapy-redis
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            
# Gerapy RabbitMQ

This is a package for supporting distribution in Scrapy using RabbitMQ, also this
package is a module in [Gerapy](https://github.com/Gerapy/Gerapy).

## Installation

You can install with this command:

```shell script
pip3 install gerapy-rabbitmq
```

## Usage

Required configuration:

```python
# Use RabbitMQ for queue
SCHEDULER = "gerapy_rabbitmq.scheduler.Scheduler"
SCHEDULER_QUEUE_KEY = '%(spider)s_requests'

# RabbitMQ Connection Parameters, see https://pika.readthedocs.io/en/stable/modules/parameters.html
RABBITMQ_CONNECTION_PARAMETERS = {
    'host': 'localhost'
}

# Use Redis for dupefilter
DUPEFILTER_CLASS = "gerapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_DUPEFILTER_KEY = '%(spider)s:dupefilter'
```

Optional configuration:

```python
# RabbitMQ Queue Configuration
SCHEDULER_QUEUE_DURABLE = True
SCHEDULER_QUEUE_MAX_PRIORITY = 100
SCHEDULER_QUEUE_PRIORITY_OFFSET = 30
SCHEDULER_QUEUE_FORCE_FLUSH = True
SCHEDULER_PERSIST = False
SCHEDULER_IDLE_BEFORE_CLOSE = 0
SCHEDULER_FLUSH_ON_START = False
SCHEDULER_PRE_ENQUEUE_ALL_START_REQUESTS = True
```

## More

For more detail, you can refer to [example](./example).

## RabbitMQ Preview

![](https://qiniu.cuiqingcai.com/zjc82.png)

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/Gerapy/GerapyRabbitMQ",
    "name": "gerapy-rabbitmq",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.5.0",
    "maintainer_email": "",
    "keywords": "",
    "author": "Germey",
    "author_email": "cqc@cuiqingcai.com",
    "download_url": "https://files.pythonhosted.org/packages/7f/05/f2796ea580d466aab79f9836ad59a61d61e1ea152d9650b920f7ba12f789/gerapy-rabbitmq-0.1.1.tar.gz",
    "platform": null,
    "description": "\n# Gerapy RabbitMQ\n\nThis is a package for supporting distribution in Scrapy using RabbitMQ, also this\npackage is a module in [Gerapy](https://github.com/Gerapy/Gerapy).\n\n## Installation\n\nYou can install with this command:\n\n```shell script\npip3 install gerapy-rabbitmq\n```\n\n## Usage\n\nRequired configuration:\n\n```python\n# Use RabbitMQ for queue\nSCHEDULER = \"gerapy_rabbitmq.scheduler.Scheduler\"\nSCHEDULER_QUEUE_KEY = '%(spider)s_requests'\n\n# RabbitMQ Connection Parameters, see https://pika.readthedocs.io/en/stable/modules/parameters.html\nRABBITMQ_CONNECTION_PARAMETERS = {\n    'host': 'localhost'\n}\n\n# Use Redis for dupefilter\nDUPEFILTER_CLASS = \"gerapy_redis.dupefilter.RFPDupeFilter\"\nSCHEDULER_DUPEFILTER_KEY = '%(spider)s:dupefilter'\n```\n\nOptional configuration:\n\n```python\n# RabbitMQ Queue Configuration\nSCHEDULER_QUEUE_DURABLE = True\nSCHEDULER_QUEUE_MAX_PRIORITY = 100\nSCHEDULER_QUEUE_PRIORITY_OFFSET = 30\nSCHEDULER_QUEUE_FORCE_FLUSH = True\nSCHEDULER_PERSIST = False\nSCHEDULER_IDLE_BEFORE_CLOSE = 0\nSCHEDULER_FLUSH_ON_START = False\nSCHEDULER_PRE_ENQUEUE_ALL_START_REQUESTS = True\n```\n\n## More\n\nFor more detail, you can refer to [example](./example).\n\n## RabbitMQ Preview\n\n![](https://qiniu.cuiqingcai.com/zjc82.png)\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Distribution Support for Scrapy & Gerapy using RabbitMQ",
    "version": "0.1.1",
    "project_urls": {
        "Homepage": "https://github.com/Gerapy/GerapyRabbitMQ"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "61ee7bc368c43e1eceff6a43fa11c89357c2dca12bf3d55ce216e7834daa6d04",
                "md5": "84db7fd053acc9e55db094c3b529f81f",
                "sha256": "b9a8b1131e5e48f2167c7282162e2f70c1e4576474eab7af81165075bb5a3663"
            },
            "downloads": -1,
            "filename": "gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "84db7fd053acc9e55db094c3b529f81f",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=3.5.0",
            "size": 6671,
            "upload_time": "2023-11-27T15:52:52",
            "upload_time_iso_8601": "2023-11-27T15:52:52.016880Z",
            "url": "https://files.pythonhosted.org/packages/61/ee/7bc368c43e1eceff6a43fa11c89357c2dca12bf3d55ce216e7834daa6d04/gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "7f05f2796ea580d466aab79f9836ad59a61d61e1ea152d9650b920f7ba12f789",
                "md5": "16b05da3b759b741407f839d6507bf28",
                "sha256": "1e944ee404c1ffe18c2f324a7438e2678311c1b496ac1f0da739711c1adea3f9"
            },
            "downloads": -1,
            "filename": "gerapy-rabbitmq-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "16b05da3b759b741407f839d6507bf28",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5.0",
            "size": 6346,
            "upload_time": "2023-11-27T15:52:54",
            "upload_time_iso_8601": "2023-11-27T15:52:54.581190Z",
            "url": "https://files.pythonhosted.org/packages/7f/05/f2796ea580d466aab79f9836ad59a61d61e1ea152d9650b920f7ba12f789/gerapy-rabbitmq-0.1.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-11-27 15:52:54",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "Gerapy",
    "github_project": "GerapyRabbitMQ",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [
        {
            "name": "pika",
            "specs": []
        },
        {
            "name": "gerapy-redis",
            "specs": []
        }
    ],
    "lcname": "gerapy-rabbitmq"
}