bhfutils


Namebhfutils JSON
Version 0.1.20 PyPI version JSON
download
home_pagehttps://behoof.app/
SummaryUtilities that are used by any spider of Behoof project
upload_time2025-07-14 10:21:12
maintainerNone
docs_urlNone
authorTeplygin Vladimir
requires_pythonNone
licenseMIT
keywords behoof scrapy-cluster utilities
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            ******************************
Behoof Scrapy Cluster Template
******************************

Overview
--------

The ``bhfutils`` package is a collection of utilities that are used by any spider of Behoof project.

Requirements
------------

- Unix based machine (Linux or OS X)
- Python 2.7 or 3.6

Installation
------------

Inside a virtualenv, run ``pip install -U bhfutils``.  This will install the latest version of the Behoof Scrapy Cluster Spider utilities.  After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation
-------------

Full documentation for the ``bhfutils`` package does not exist

custom_cookies.py
==================

The ``custom_cookies`` module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py
========================

The ``distributed_scheduler`` module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py
===============================

The ``redis_domain_max_page_filter`` module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded 

redis_dupefilter.py
===================

The ``redis_dupefilter`` module is redis-based request duplication filter

redis_global_page_per_domain_filter.py
======================================

The ``redis_global_page_per_domain_filter`` module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

            

Raw data

            {
    "_id": null,
    "home_page": "https://behoof.app/",
    "name": "bhfutils",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "behoof, scrapy-cluster, utilities",
    "author": "Teplygin Vladimir",
    "author_email": "vvteplygin@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/1b/ab/589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709/bhfutils-0.1.20.tar.gz",
    "platform": null,
    "description": "******************************\nBehoof Scrapy Cluster Template\n******************************\n\nOverview\n--------\n\nThe ``bhfutils`` package is a collection of utilities that are used by any spider of Behoof project.\n\nRequirements\n------------\n\n- Unix based machine (Linux or OS X)\n- Python 2.7 or 3.6\n\nInstallation\n------------\n\nInside a virtualenv, run ``pip install -U bhfutils``.  This will install the latest version of the Behoof Scrapy Cluster Spider utilities.  After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)\n\nDocumentation\n-------------\n\nFull documentation for the ``bhfutils`` package does not exist\n\ncustom_cookies.py\n==================\n\nThe ``custom_cookies`` module is custom Cookies Middleware to pass our required cookies along but not persist between calls\n\ndistributed_scheduler.py\n========================\n\nThe ``distributed_scheduler`` module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster\n\nredis_domain_max_page_filter.py\n===============================\n\nThe ``redis_domain_max_page_filter`` module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded \n\nredis_dupefilter.py\n===================\n\nThe ``redis_dupefilter`` module is redis-based request duplication filter\n\nredis_global_page_per_domain_filter.py\n======================================\n\nThe ``redis_global_page_per_domain_filter`` module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Utilities that are used by any spider of Behoof project",
    "version": "0.1.20",
    "project_urls": {
        "Homepage": "https://behoof.app/"
    },
    "split_keywords": [
        "behoof",
        " scrapy-cluster",
        " utilities"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "830c3fe6ba0eae632515854f7dec64ee838b4aaec54807f30b00d9f293f3bdef",
                "md5": "6375d6c61b8a4a385e16cfbaf876099f",
                "sha256": "e5f19cb5fa0ecd7bcd79f1068ca8ecebb24a1dd280ac88c5e8c715bef06ec4ec"
            },
            "downloads": -1,
            "filename": "bhfutils-0.1.20-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "6375d6c61b8a4a385e16cfbaf876099f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 124843,
            "upload_time": "2025-07-14T10:21:09",
            "upload_time_iso_8601": "2025-07-14T10:21:09.497951Z",
            "url": "https://files.pythonhosted.org/packages/83/0c/3fe6ba0eae632515854f7dec64ee838b4aaec54807f30b00d9f293f3bdef/bhfutils-0.1.20-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "1bab589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709",
                "md5": "88b0b555f6f108c8387e3d84f6e9d091",
                "sha256": "86be83c194f00c506836f22d341526b0c4352dbb876ad875f5c54212d6c67ec0"
            },
            "downloads": -1,
            "filename": "bhfutils-0.1.20.tar.gz",
            "has_sig": false,
            "md5_digest": "88b0b555f6f108c8387e3d84f6e9d091",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 91292,
            "upload_time": "2025-07-14T10:21:12",
            "upload_time_iso_8601": "2025-07-14T10:21:12.280391Z",
            "url": "https://files.pythonhosted.org/packages/1b/ab/589b6839b0ff889a901543468178f9778291325916da545290aaebeaf709/bhfutils-0.1.20.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-07-14 10:21:12",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "bhfutils"
}
        
Elapsed time: 1.03845s