  
# Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
Create your own list from several sources.
See the **[blocklist-domains](https://github.com/dmachard/blocklist-domains)** repository for an implementation.
Default sources are defined on the [configuration file](../main/blocklist_aggregator/blocklist.conf)
## Table of contents
* [Installation](#installation)
* [Get Started](#get-started)
* [Custom Configuration](#custom-configuration)
* [Fetch and save-it to files](#fetch-and-save-it-to-files)
## Installation
    
If you want to generate your own unified blocklist,
install this module with the pip command.
```python
pip install blocklist_aggregator
```
## Get started
This basic example enable to get a unified list of domains.
You can save-it in a file or do what you want.
```python
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
```
## Custom configuration
See the default [configuration file](../main/blocklist_aggregator/blocklist.conf)
The configuration contains:
* the ads/tracking/malware URL lists with the pattern (regex) to use
* the domains list to exclude (whitelist)
* additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
```python
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
```
or loaded from external config file
```python
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
```
## Fetch and save-it to files
This module can be used to export the list in several format:
* text
* hosts
* CDB (key/value database)
```python
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
```
## For developpers
Run test units
```bash
python3 -m unittest discover tests/ -v
```
Raw data
{
"_id": null,
"home_page": "https://github.com/dmachard/blocklist-aggregator",
"name": "blocklist-aggregator",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "blocklist aggregator domains dns blacklist whitelist",
"author": "Denis MACHARD",
"author_email": "d.machard@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/78/6d/3ceb0e00edf3097639cc8ac4cb69f5df06323d1c36fec360ebfb98a035f4/blocklist_aggregator-1.2.0.tar.gz",
"platform": "any",
"description": "   \n\n# Blocklist aggregator\n\nThis python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.\nCreate your own list from several sources.\n\nSee the **[blocklist-domains](https://github.com/dmachard/blocklist-domains)** repository for an implementation.\n\nDefault sources are defined on the [configuration file](../main/blocklist_aggregator/blocklist.conf)\n\n## Table of contents\n\n* [Installation](#installation)\n* [Get Started](#get-started)\n* [Custom Configuration](#custom-configuration)\n* [Fetch and save-it to files](#fetch-and-save-it-to-files)\n\n## Installation\n\n    \n\nIf you want to generate your own unified blocklist, \ninstall this module with the pip command.\n\n```python\npip install blocklist_aggregator\n```\n\n## Get started\n\nThis basic example enable to get a unified list of domains.\nYou can save-it in a file or do what you want.\n\n```python\nimport blocklist_aggregator\n\nunified = blocklist_aggregator.fetch()\nprint(unified)\n[ \"doubleclick.net\", ..., \"telemetry.dropbox.com\" ]\n\nprint(len(unified))\n152978\n```\n\n## Custom configuration\n\nSee the default [configuration file](../main/blocklist_aggregator/blocklist.conf)\n\nThe configuration contains:\n\n* the ads/tracking/malware URL lists with the pattern (regex) to use\n* the domains list to exclude (whitelist)\n* additionnal domains list to block (blacklist)\n\nThe configuration can be overwritten at runtime.\n\n```python\ncfg_yaml = \"verbose: true\"\nunified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)\n```\n\nor loaded from external config file\n\n```python\nunified = blocklist_aggregator.fetch(cfg_filename=\"/home/custom-blocklist.conf\")\n```\n\n## Fetch and save-it to files\n\nThis module can be used to export the list in several format:\n\n* text\n* hosts\n* CDB (key/value database)\n\n```python\nimport blocklist_aggregator\n\n# fetch domains\nunified = blocklist_aggregator.fetch()\n\n# save to a text file\nblocklist_aggregator.save_raw(filename=\"/tmp/unified_list.txt\")\n\n# save to hosts file\nblocklist_aggregator.save_hosts(filename=\"/tmp/unified_hosts.txt\", ip=\"0.0.0.0\")\n\n# save to CDB\nblocklist_aggregator.save_cdb(filename=\"/tmp/unified_domains.cdb\")\n```\n\n## For developpers\n\nRun test units\n\n```bash\npython3 -m unittest discover tests/ -v\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "Domains blocklist aggregator",
"version": "1.2.0",
"project_urls": {
"Homepage": "https://github.com/dmachard/blocklist-aggregator"
},
"split_keywords": [
"blocklist",
"aggregator",
"domains",
"dns",
"blacklist",
"whitelist"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e0649ef6fa5037a3a42cc4555c08fa518b45ae21a43bd0eb349a93f89bcd1160",
"md5": "9b03bf8c785d85bb4a5fa75c9852eda7",
"sha256": "99e96ce8f804cfd8dcc32ee81b207200c89c3673151d7e50d36fabb78bde5a1e"
},
"downloads": -1,
"filename": "blocklist_aggregator-1.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9b03bf8c785d85bb4a5fa75c9852eda7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6017,
"upload_time": "2024-01-15T11:37:12",
"upload_time_iso_8601": "2024-01-15T11:37:12.136807Z",
"url": "https://files.pythonhosted.org/packages/e0/64/9ef6fa5037a3a42cc4555c08fa518b45ae21a43bd0eb349a93f89bcd1160/blocklist_aggregator-1.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "786d3ceb0e00edf3097639cc8ac4cb69f5df06323d1c36fec360ebfb98a035f4",
"md5": "77de295b49526f6698ddd9b3e6ccb0cd",
"sha256": "2b4b3b8f63e3c5125247aa78c6f7baf78432d4de9c594065457e01ec06d39c01"
},
"downloads": -1,
"filename": "blocklist_aggregator-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "77de295b49526f6698ddd9b3e6ccb0cd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5952,
"upload_time": "2024-01-15T11:37:13",
"upload_time_iso_8601": "2024-01-15T11:37:13.728329Z",
"url": "https://files.pythonhosted.org/packages/78/6d/3ceb0e00edf3097639cc8ac4cb69f5df06323d1c36fec360ebfb98a035f4/blocklist_aggregator-1.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-15 11:37:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dmachard",
"github_project": "blocklist-aggregator",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "blocklist-aggregator"
}
JSON
