ultimate-hosts-blacklist-whitelist-dev


Nameultimate-hosts-blacklist-whitelist-dev JSON
Version 3.23.1 PyPI version JSON
download
home_pagehttps://github.com/Ultimate-Hosts-Blacklist/whitelist/tree/script
SummaryThe whitelisting tool of the Ultimate Hosts Blacklist project.
upload_time2020-08-01 22:53:28
maintainer
docs_urlNone
author
requires_python
licenseMIT
keywords python hosts hosts file whitelist
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            The whitelisting tool from the Ultimate Hosts Blacklist project
===============================================================

This is the whitelisting tool provided by the Ultimate Hosts Blacklist project.
The script is mainly used to whitelist subject (domains, IP, URL) into our
infrastructure but it can also easily be used outside our infrastructure.

The ultimate hosts blacklist whitelist (**UHBW**) tool allows you:

* to append your own whitelist as complementary to our whitelist.

::

    uhb_whitelist -f inputfile -o outputfile -w whitelistfile



* to anti-whitelist (reverse) only particular domains while still using our core list.

::

    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile



* to whitelist and anti-whitelist while still using our core list.

::

    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile -w whitelistfile



* to override our core whitelist whilst still applying your own personal whitelist and anti-whitelist.

::

    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile -w whitelistfile -wc


* to have a whitelist tool ready to use as a Python module.


Installation
------------

::

    $ pip3 install --user ultimate-hosts-blacklist-whitelist



The hosted whitelist
--------------------

The hosted whitelist can be found at `whitelist`_
This white-list is maintained by the team of good peoples behind the `whitelist`_
project.

Complementary whitelist
-----------------------

`UHBW`_ allows you to link one or more file(s) to the system which will be used as
complementary to the hosted `whitelist`_, which is downloaded and used by default.

Special markers
---------------

If you already have tried to use a whitelist, you'll probably know, that in
generally you can only add one domain or URL per line in a file, for which you
want to whitelist.

With UHBW you can do this, but in addition to that tedious way, UHBW allows you
to use ``Regex``, :code:`RZD` and :code:`ALL`

:code:`ALL`
^^^^^^^^^^^

The :code:`ALL` marker will tell the system to escape and regex check against
what follows.

INVALID characters
""""""""""""""""""

* :code:`$`

    * As we automatically append :code:`$` to the end of each line, you should
      not use this character.

* :code:`\\`

    * As we automatically escape any given expression, you should not explicitly
      escape your regular expression when declaring an :code:`ALL` marker.

:code:`REG`
"""""""""""

The :code:`REG` marker will tell the system to explicitly check for the given
regex which follows the marker.

:code:`RZD`
"""""""""""

The :code:`RZD` marker will tell the system to explicitly check for the given
string plus all possible TDL.

Anti whitelist
--------------

Don't like some of our rule(s)? UHBW allows you to specify a file, which contain
a list of rule(s) you don't want to be applied.

Simply use the :code:`--anti-whitelist` flag to provide one or more anti whitelist
files and UHBW will obey your wishes!


Understanding how UHBW whitelist works
--------------------------------------

If you have your own whitelist, with the following lines:

::

    facebook.com
    ALL .gov
    REG face
    RZD example

UHBW will do as follows:

* Remove every line which match :code:`facebook.com` and :code:`www.facebook.com`
* In complementary convert all lines with :code:`ALL ` or :code:`REG` to the
  right format.
* Remove every line which match :code:`example.*`
* Check every line against the regular expression. More about this in next chapter.
* Print the results on screen or save to output file :code:`-o $output.file`.

The generated regular expression will from this example be:

::

    (\.gov$)|(face)|(example(.*))


.. note::
    The :code:`example` group is much longer, as we construct the list of TDL
    based on the Root Zone Database, of IANA and the Public Suffix List
    project.**

Which means UHBW actually will whitelist:

* all elements/lines which ends with :code:`.gov`
* all elements/lines which contain the word :code:`face`
* all possible TDL combination which starts with :code:`example`

File Formats
--------------

Your input files of domains / urls should be one domain / url per line and should also preferably be sorted.

::

    sort -u inputfile -o inputfile



Usage of the tool
-----------------

The script can be called by :code:`uhb-whitelist`, :code:`uhb_whitelist` or
:code:`ultimate-hosts-blacklist-whitelist`.

::

    usage: ultimate-hosts-blacklist-whitelist [-h]
                                                [-a ANTI_WHITELIST [ANTI_WHITELIST ...]]
                                                [--all ALL [ALL ...]] [-d] [-df]
                                                [-f FILE] [--hierachical-sorting]
                                                [-o OUTPUT] [-m] [--no-complement]
                                                [-p PROCESSES] [--reg REG [REG ...]]
                                                [--rzd RZD [RZD ...]]
                                                [--standard-sorting] [-v]
                                                [-w WHITELIST [WHITELIST ...]] [-wc]

    UHBW is a tool to clean up lists or hosts files with the hosted and/or your
    own whitelist.

    optional arguments:
        -h, --help            show this help message and exit
        -a ANTI_WHITELIST [ANTI_WHITELIST ...], --anti-whitelist ANTI_WHITELIST [ANTI_WHITELIST ...]
                                Read the given file override rules from the UHBW
                                hosted whitelist which is used by default. (See also
                                `-wc`)
        --all ALL [ALL ...]   Read the given file(s) and append its rules to the
                                whitelisting schema. Note: The rules injected
                                through this argument will be automatically prefixed
                                with the `ALL` marker.
        -d, --debug           Activate the debug mode. This mode will write the
                                whole processes to stdout.
        -df, --debug-into-file
                                Activate the logging into a file called
                                `uhb_whitelist_debug` at the current location.
        -f FILE, --file FILE  The file to whitelist/clean.
        --hierachical-sorting
                                Process a hierarchical sorting when outputing into a
                                file.
        -o OUTPUT, --output OUTPUT
                                Save the result to the given filename or path. (Can
                                not be the same as input file `-f`)
        -m, --multiprocessing
                                Activate the usage of multiple core processes.
        --no-complement       Forbid us the generation of complements while parsing
                                the whitelist list. Complements are `www.example.org`
                                if `example.org` is given and vice-versa.
        -p PROCESSES, --processes PROCESSES
                                The number of (maximal) processes core to use.
        --reg REG [REG ...]   Read the given file(s) and append its rules to the
                                whitelisting schema. Note: The rules injected
                                through this argument will be automatically prefixed
                                with the `REG` marker.
        --rzd RZD [RZD ...]   Read the given file(s) and append its rules to the
                                whitelisting schema. Note: The rules injected
                                through this argument will be automatically prefixed
                                with the `RZD` marker.
        --standard-sorting    Process a sorting when outputing into a file.
        -v, --version         Show the version end exist.
        -w WHITELIST [WHITELIST ...], --whitelist WHITELIST [WHITELIST ...]
                                Read the given file(s) and append its rules to the
                                whitelisting schema. Note: The rules injected
                                through this argument won't be changed. We follow what
                                you give us. That means that if you give any of our
                                supported rules, they will still be appended to the
                                whitelisting schema.
        -wc, --without-core   Disable the usage of the Ultimate Hosts Blacklist
                                whitelist hosted list.

    Crafted with ♥ by Nissar Chababy (Funilrys)

Contributors
------------

* Daniel - `@dnmTX`_
* Spirillen - `@spirillen`_

License
-------

::

    MIT License

    Copyright (c) 2018, 2019, 2020 Ultimate-Hosts-Blacklist
    Copyright (c) 2018, 2019, 2020 Nissar Chababy
    Copyright (c) 2019, 2020 Mitchell Krog

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.

.. _@dnmTX: https://github.com/dnmTX
.. _@spirillen: https://github.com/spirillen
.. _whitelist: https://github.com/Ultimate-Hosts-Blacklist/whitelist
.. _UHBW: https://github.com/Ultimate-Hosts-Blacklist/whitelist/tree/script



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/Ultimate-Hosts-Blacklist/whitelist/tree/script",
    "name": "ultimate-hosts-blacklist-whitelist-dev",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "Python,hosts,hosts file,whitelist",
    "author": "",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/10/53/1f6b211435e5508fb98f3afa259e77487212f1619cd638e2eed62888202d/ultimate-hosts-blacklist-whitelist-dev-3.23.1.tar.gz",
    "platform": "any",
    "description": "The whitelisting tool from the Ultimate Hosts Blacklist project\n===============================================================\n\nThis is the whitelisting tool provided by the Ultimate Hosts Blacklist project.\nThe script is mainly used to whitelist subject (domains, IP, URL) into our\ninfrastructure but it can also easily be used outside our infrastructure.\n\nThe ultimate hosts blacklist whitelist (**UHBW**) tool allows you:\n\n* to append your own whitelist as complementary to our whitelist.\n\n::\n\n    uhb_whitelist -f inputfile -o outputfile -w whitelistfile\n\n\n\n* to anti-whitelist (reverse) only particular domains while still using our core list.\n\n::\n\n    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile\n\n\n\n* to whitelist and anti-whitelist while still using our core list.\n\n::\n\n    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile -w whitelistfile\n\n\n\n* to override our core whitelist whilst still applying your own personal whitelist and anti-whitelist.\n\n::\n\n    uhb_whitelist -f inputfile -o outputfile --anti-whitelist antiwhitelistfile -w whitelistfile -wc\n\n\n* to have a whitelist tool ready to use as a Python module.\n\n\nInstallation\n------------\n\n::\n\n    $ pip3 install --user ultimate-hosts-blacklist-whitelist\n\n\n\nThe hosted whitelist\n--------------------\n\nThe hosted whitelist can be found at `whitelist`_\nThis white-list is maintained by the team of good peoples behind the `whitelist`_\nproject.\n\nComplementary whitelist\n-----------------------\n\n`UHBW`_ allows you to link one or more file(s) to the system which will be used as\ncomplementary to the hosted `whitelist`_, which is downloaded and used by default.\n\nSpecial markers\n---------------\n\nIf you already have tried to use a whitelist, you'll probably know, that in\ngenerally you can only add one domain or URL per line in a file, for which you\nwant to whitelist.\n\nWith UHBW you can do this, but in addition to that tedious way, UHBW allows you\nto use ``Regex``, :code:`RZD` and :code:`ALL`\n\n:code:`ALL`\n^^^^^^^^^^^\n\nThe :code:`ALL` marker will tell the system to escape and regex check against\nwhat follows.\n\nINVALID characters\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n* :code:`$`\n\n    * As we automatically append :code:`$` to the end of each line, you should\n      not use this character.\n\n* :code:`\\\\`\n\n    * As we automatically escape any given expression, you should not explicitly\n      escape your regular expression when declaring an :code:`ALL` marker.\n\n:code:`REG`\n\"\"\"\"\"\"\"\"\"\"\"\n\nThe :code:`REG` marker will tell the system to explicitly check for the given\nregex which follows the marker.\n\n:code:`RZD`\n\"\"\"\"\"\"\"\"\"\"\"\n\nThe :code:`RZD` marker will tell the system to explicitly check for the given\nstring plus all possible TDL.\n\nAnti whitelist\n--------------\n\nDon't like some of our rule(s)? UHBW allows you to specify a file, which contain\na list of rule(s) you don't want to be applied.\n\nSimply use the :code:`--anti-whitelist` flag to provide one or more anti whitelist\nfiles and UHBW will obey your wishes!\n\n\nUnderstanding how UHBW whitelist works\n--------------------------------------\n\nIf you have your own whitelist, with the following lines:\n\n::\n\n    facebook.com\n    ALL .gov\n    REG face\n    RZD example\n\nUHBW will do as follows:\n\n* Remove every line which match :code:`facebook.com` and :code:`www.facebook.com`\n* In complementary convert all lines with :code:`ALL ` or :code:`REG` to the\n  right format.\n* Remove every line which match :code:`example.*`\n* Check every line against the regular expression. More about this in next chapter.\n* Print the results on screen or save to output file :code:`-o $output.file`.\n\nThe generated regular expression will from this example be:\n\n::\n\n    (\\.gov$)|(face)|(example(.*))\n\n\n.. note::\n    The :code:`example` group is much longer, as we construct the list of TDL\n    based on the Root Zone Database, of IANA and the Public Suffix List\n    project.**\n\nWhich means UHBW actually will whitelist:\n\n* all elements/lines which ends with :code:`.gov`\n* all elements/lines which contain the word :code:`face`\n* all possible TDL combination which starts with :code:`example`\n\nFile Formats\n--------------\n\nYour input files of domains / urls should be one domain / url per line and should also preferably be sorted.\n\n::\n\n    sort -u inputfile -o inputfile\n\n\n\nUsage of the tool\n-----------------\n\nThe script can be called by :code:`uhb-whitelist`, :code:`uhb_whitelist` or\n:code:`ultimate-hosts-blacklist-whitelist`.\n\n::\n\n    usage: ultimate-hosts-blacklist-whitelist [-h]\n                                                [-a ANTI_WHITELIST [ANTI_WHITELIST ...]]\n                                                [--all ALL [ALL ...]] [-d] [-df]\n                                                [-f FILE] [--hierachical-sorting]\n                                                [-o OUTPUT] [-m] [--no-complement]\n                                                [-p PROCESSES] [--reg REG [REG ...]]\n                                                [--rzd RZD [RZD ...]]\n                                                [--standard-sorting] [-v]\n                                                [-w WHITELIST [WHITELIST ...]] [-wc]\n\n    UHBW is a tool to clean up lists or hosts files with the hosted and/or your\n    own whitelist.\n\n    optional arguments:\n        -h, --help            show this help message and exit\n        -a ANTI_WHITELIST [ANTI_WHITELIST ...], --anti-whitelist ANTI_WHITELIST [ANTI_WHITELIST ...]\n                                Read the given file override rules from the UHBW\n                                hosted whitelist which is used by default. (See also\n                                `-wc`)\n        --all ALL [ALL ...]   Read the given file(s) and append its rules to the\n                                whitelisting schema. Note: The rules injected\n                                through this argument will be automatically prefixed\n                                with the `ALL` marker.\n        -d, --debug           Activate the debug mode. This mode will write the\n                                whole processes to stdout.\n        -df, --debug-into-file\n                                Activate the logging into a file called\n                                `uhb_whitelist_debug` at the current location.\n        -f FILE, --file FILE  The file to whitelist/clean.\n        --hierachical-sorting\n                                Process a hierarchical sorting when outputing into a\n                                file.\n        -o OUTPUT, --output OUTPUT\n                                Save the result to the given filename or path. (Can\n                                not be the same as input file `-f`)\n        -m, --multiprocessing\n                                Activate the usage of multiple core processes.\n        --no-complement       Forbid us the generation of complements while parsing\n                                the whitelist list. Complements are `www.example.org`\n                                if `example.org` is given and vice-versa.\n        -p PROCESSES, --processes PROCESSES\n                                The number of (maximal) processes core to use.\n        --reg REG [REG ...]   Read the given file(s) and append its rules to the\n                                whitelisting schema. Note: The rules injected\n                                through this argument will be automatically prefixed\n                                with the `REG` marker.\n        --rzd RZD [RZD ...]   Read the given file(s) and append its rules to the\n                                whitelisting schema. Note: The rules injected\n                                through this argument will be automatically prefixed\n                                with the `RZD` marker.\n        --standard-sorting    Process a sorting when outputing into a file.\n        -v, --version         Show the version end exist.\n        -w WHITELIST [WHITELIST ...], --whitelist WHITELIST [WHITELIST ...]\n                                Read the given file(s) and append its rules to the\n                                whitelisting schema. Note: The rules injected\n                                through this argument won't be changed. We follow what\n                                you give us. That means that if you give any of our\n                                supported rules, they will still be appended to the\n                                whitelisting schema.\n        -wc, --without-core   Disable the usage of the Ultimate Hosts Blacklist\n                                whitelist hosted list.\n\n    Crafted with \u2665 by Nissar Chababy (Funilrys)\n\nContributors\n------------\n\n* Daniel - `@dnmTX`_\n* Spirillen - `@spirillen`_\n\nLicense\n-------\n\n::\n\n    MIT License\n\n    Copyright (c) 2018, 2019, 2020 Ultimate-Hosts-Blacklist\n    Copyright (c) 2018, 2019, 2020 Nissar Chababy\n    Copyright (c) 2019, 2020 Mitchell Krog\n\n    Permission is hereby granted, free of charge, to any person obtaining a copy\n    of this software and associated documentation files (the \"Software\"), to deal\n    in the Software without restriction, including without limitation the rights\n    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n    copies of the Software, and to permit persons to whom the Software is\n    furnished to do so, subject to the following conditions:\n\n    The above copyright notice and this permission notice shall be included in all\n    copies or substantial portions of the Software.\n\n    THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n    SOFTWARE.\n\n.. _@dnmTX: https://github.com/dnmTX\n.. _@spirillen: https://github.com/spirillen\n.. _whitelist: https://github.com/Ultimate-Hosts-Blacklist/whitelist\n.. _UHBW: https://github.com/Ultimate-Hosts-Blacklist/whitelist/tree/script\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "The whitelisting tool of the Ultimate Hosts Blacklist project.",
    "version": "3.23.1",
    "split_keywords": [
        "python",
        "hosts",
        "hosts file",
        "whitelist"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "a1697540ae3fcf4ee62a1f4bc29da039",
                "sha256": "c1ca26f1f3209f05a2ede1e37f839e827c9f8645ca90ddd08c1ac87caf867230"
            },
            "downloads": -1,
            "filename": "ultimate_hosts_blacklist_whitelist_dev-3.23.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a1697540ae3fcf4ee62a1f4bc29da039",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 19585,
            "upload_time": "2020-08-01T22:53:26",
            "upload_time_iso_8601": "2020-08-01T22:53:26.567804Z",
            "url": "https://files.pythonhosted.org/packages/5b/42/e7e5f47278ed006bcd7f41a560e069cda61cff076625b07f4c45af7bbe48/ultimate_hosts_blacklist_whitelist_dev-3.23.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "4790d7eb344d41da90f31a2d4f423276",
                "sha256": "0ddecea170c5fef844d1bc6d04bf658fb78c3c455148fddd95daeedb46482bcb"
            },
            "downloads": -1,
            "filename": "ultimate-hosts-blacklist-whitelist-dev-3.23.1.tar.gz",
            "has_sig": false,
            "md5_digest": "4790d7eb344d41da90f31a2d4f423276",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 19380,
            "upload_time": "2020-08-01T22:53:28",
            "upload_time_iso_8601": "2020-08-01T22:53:28.759640Z",
            "url": "https://files.pythonhosted.org/packages/10/53/1f6b211435e5508fb98f3afa259e77487212f1619cd638e2eed62888202d/ultimate-hosts-blacklist-whitelist-dev-3.23.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2020-08-01 22:53:28",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "ultimate-hosts-blacklist-whitelist-dev"
}
        
Elapsed time: 0.15971s