******************************
Behoof Scrapy Cluster Template
******************************
Overview
--------
The ``bhfutils`` package is a collection of utilities that are used by any spider of Behoof project.
Requirements
------------
- Unix based machine (Linux or OS X)
- Python 2.7 or 3.6
Installation
------------
Inside a virtualenv, run ``pip install -U bhfutils``. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)
Documentation
-------------
Full documentation for the ``bhfutils`` package does not exist
custom_cookies.py
==================
The ``custom_cookies`` module is custom Cookies Middleware to pass our required cookies along but not persist between calls
distributed_scheduler.py
========================
The ``distributed_scheduler`` module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster
redis_domain_max_page_filter.py
===============================
The ``redis_domain_max_page_filter`` module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded
redis_dupefilter.py
===================
The ``redis_dupefilter`` module is redis-based request duplication filter
redis_global_page_per_domain_filter.py
======================================
The ``redis_global_page_per_domain_filter`` module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.
Raw data
{
"_id": null,
"home_page": "https://behoof.app/",
"name": "bhfutils",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "behoof, scrapy-cluster, utilities",
"author": "Teplygin Vladimir",
"author_email": "vvteplygin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1b/ab/589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709/bhfutils-0.1.20.tar.gz",
"platform": null,
"description": "******************************\nBehoof Scrapy Cluster Template\n******************************\n\nOverview\n--------\n\nThe ``bhfutils`` package is a collection of utilities that are used by any spider of Behoof project.\n\nRequirements\n------------\n\n- Unix based machine (Linux or OS X)\n- Python 2.7 or 3.6\n\nInstallation\n------------\n\nInside a virtualenv, run ``pip install -U bhfutils``. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)\n\nDocumentation\n-------------\n\nFull documentation for the ``bhfutils`` package does not exist\n\ncustom_cookies.py\n==================\n\nThe ``custom_cookies`` module is custom Cookies Middleware to pass our required cookies along but not persist between calls\n\ndistributed_scheduler.py\n========================\n\nThe ``distributed_scheduler`` module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster\n\nredis_domain_max_page_filter.py\n===============================\n\nThe ``redis_domain_max_page_filter`` module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded \n\nredis_dupefilter.py\n===================\n\nThe ``redis_dupefilter`` module is redis-based request duplication filter\n\nredis_global_page_per_domain_filter.py\n======================================\n\nThe ``redis_global_page_per_domain_filter`` module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Utilities that are used by any spider of Behoof project",
"version": "0.1.20",
"project_urls": {
"Homepage": "https://behoof.app/"
},
"split_keywords": [
"behoof",
" scrapy-cluster",
" utilities"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "830c3fe6ba0eae632515854f7dec64ee838b4aaec54807f30b00d9f293f3bdef",
"md5": "6375d6c61b8a4a385e16cfbaf876099f",
"sha256": "e5f19cb5fa0ecd7bcd79f1068ca8ecebb24a1dd280ac88c5e8c715bef06ec4ec"
},
"downloads": -1,
"filename": "bhfutils-0.1.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6375d6c61b8a4a385e16cfbaf876099f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 124843,
"upload_time": "2025-07-14T10:21:09",
"upload_time_iso_8601": "2025-07-14T10:21:09.497951Z",
"url": "https://files.pythonhosted.org/packages/83/0c/3fe6ba0eae632515854f7dec64ee838b4aaec54807f30b00d9f293f3bdef/bhfutils-0.1.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1bab589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709",
"md5": "88b0b555f6f108c8387e3d84f6e9d091",
"sha256": "86be83c194f00c506836f22d341526b0c4352dbb876ad875f5c54212d6c67ec0"
},
"downloads": -1,
"filename": "bhfutils-0.1.20.tar.gz",
"has_sig": false,
"md5_digest": "88b0b555f6f108c8387e3d84f6e9d091",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 91292,
"upload_time": "2025-07-14T10:21:12",
"upload_time_iso_8601": "2025-07-14T10:21:12.280391Z",
"url": "https://files.pythonhosted.org/packages/1b/ab/589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709/bhfutils-0.1.20.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-14 10:21:12",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "bhfutils"
}
JSON