  
# Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
Create your own list from several sources.
See the **[blocklist-domains](https://github.com/dmachard/blocklist-domains)** repository for an implementation.
Default sources are defined on the [configuration file](../main/blocklist_aggregator/blocklist.conf)
## Table of contents
* [Installation](#installation)
* [Get Started](#get-started)
* [Custom Configuration](#custom-configuration)
* [Fetch and save-it to files](#fetch-and-save-it-to-files)
## Installation
     
If you want to generate your own unified blocklist,
install this module with the pip command.
```python
pip install blocklist_aggregator
```
## Get started
This basic example enable to get a unified list of domains.
You can save-it in a file or do what you want.
```python
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
```
## Custom configuration
See the default [configuration file](../main/blocklist_aggregator/blocklist.conf)
The configuration contains:
* the ads/tracking/malware URL lists with the pattern (regex) to use
* the domains list to exclude (whitelist)
* additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
```python
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
```
or loaded from external config file
```python
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
```
## Fetch and save-it to files
This module can be used to export the list in several format:
* text
* hosts
* CDB (key/value database)
```python
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
```
## For developpers
INstall prerequisites
```bash
sudo apt install python3-venv python3.13-dev
python3 -m venv venv
source venv/bin/activate
python3 -m pip install -r requirements.txt
```
Run test units
```bash
python3 -m unittest discover tests/ -v
```
Raw data
{
"_id": null,
"home_page": "https://github.com/dmachard/blocklist-aggregator",
"name": "blocklist-aggregator",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "blocklist aggregator domains dns blacklist whitelist",
"author": "Denis MACHARD",
"author_email": "d.machard@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e8/4a/40430308ac08824f10cb99fe4707d3eb45e6ad8624d057fa7c56167d85a8/blocklist_aggregator-1.4.1.tar.gz",
"platform": "any",
"description": "   \n\n# Blocklist aggregator\n\nThis python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.\nCreate your own list from several sources.\n\nSee the **[blocklist-domains](https://github.com/dmachard/blocklist-domains)** repository for an implementation.\n\nDefault sources are defined on the [configuration file](../main/blocklist_aggregator/blocklist.conf)\n\n## Table of contents\n\n* [Installation](#installation)\n* [Get Started](#get-started)\n* [Custom Configuration](#custom-configuration)\n* [Fetch and save-it to files](#fetch-and-save-it-to-files)\n\n## Installation\n\n     \n\nIf you want to generate your own unified blocklist, \ninstall this module with the pip command.\n\n```python\npip install blocklist_aggregator\n```\n\n## Get started\n\nThis basic example enable to get a unified list of domains.\nYou can save-it in a file or do what you want.\n\n```python\nimport blocklist_aggregator\n\nunified = blocklist_aggregator.fetch()\nprint(unified)\n[ \"doubleclick.net\", ..., \"telemetry.dropbox.com\" ]\n\nprint(len(unified))\n152978\n```\n\n## Custom configuration\n\nSee the default [configuration file](../main/blocklist_aggregator/blocklist.conf)\n\nThe configuration contains:\n\n* the ads/tracking/malware URL lists with the pattern (regex) to use\n* the domains list to exclude (whitelist)\n* additionnal domains list to block (blacklist)\n\nThe configuration can be overwritten at runtime.\n\n```python\ncfg_yaml = \"verbose: true\"\nunified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)\n```\n\nor loaded from external config file\n\n```python\nunified = blocklist_aggregator.fetch(cfg_filename=\"/home/custom-blocklist.conf\")\n```\n\n## Fetch and save-it to files\n\nThis module can be used to export the list in several format:\n\n* text\n* hosts\n* CDB (key/value database)\n\n```python\nimport blocklist_aggregator\n\n# fetch domains\nunified = blocklist_aggregator.fetch()\n\n# save to a text file\nblocklist_aggregator.save_raw(filename=\"/tmp/unified_list.txt\")\n\n# save to hosts file\nblocklist_aggregator.save_hosts(filename=\"/tmp/unified_hosts.txt\", ip=\"0.0.0.0\")\n\n# save to CDB\nblocklist_aggregator.save_cdb(filename=\"/tmp/unified_domains.cdb\")\n```\n\n## For developpers\n\nINstall prerequisites\n\n```bash\nsudo apt install python3-venv python3.13-dev\npython3 -m venv venv\nsource venv/bin/activate\npython3 -m pip install -r requirements.txt\n```\n\nRun test units\n\n```bash\npython3 -m unittest discover tests/ -v\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Domains blocklist aggregator",
"version": "1.4.1",
"project_urls": {
"Homepage": "https://github.com/dmachard/blocklist-aggregator"
},
"split_keywords": [
"blocklist",
"aggregator",
"domains",
"dns",
"blacklist",
"whitelist"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a173fa0a29901f04465c40ea78260427d8a89551838d8d0e388886ed1c89ee69",
"md5": "64a10707b104f7a30e79edd19b8367cb",
"sha256": "2ff18e80c6f0ef9d05261f4da3810e25592afbcde9b17349e32fe49799d8d513"
},
"downloads": -1,
"filename": "blocklist_aggregator-1.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "64a10707b104f7a30e79edd19b8367cb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6594,
"upload_time": "2025-07-13T14:54:08",
"upload_time_iso_8601": "2025-07-13T14:54:08.431315Z",
"url": "https://files.pythonhosted.org/packages/a1/73/fa0a29901f04465c40ea78260427d8a89551838d8d0e388886ed1c89ee69/blocklist_aggregator-1.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e84a40430308ac08824f10cb99fe4707d3eb45e6ad8624d057fa7c56167d85a8",
"md5": "c6e242005bd2d5aac71b9ea315bc5474",
"sha256": "8e1d7a9d5b7c35047a8d146d590248207f749755dcc0ce18489fd5311bb3bf51"
},
"downloads": -1,
"filename": "blocklist_aggregator-1.4.1.tar.gz",
"has_sig": false,
"md5_digest": "c6e242005bd2d5aac71b9ea315bc5474",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6517,
"upload_time": "2025-07-13T14:54:09",
"upload_time_iso_8601": "2025-07-13T14:54:09.597171Z",
"url": "https://files.pythonhosted.org/packages/e8/4a/40430308ac08824f10cb99fe4707d3eb45e6ad8624d057fa7c56167d85a8/blocklist_aggregator-1.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-13 14:54:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dmachard",
"github_project": "blocklist-aggregator",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "pyyaml",
"specs": [
[
"==",
"6.0.1"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.4"
]
]
},
{
"name": "pure-cdb",
"specs": [
[
"==",
"4.0.0"
]
]
},
{
"name": "httpx",
"specs": [
[
"==",
"0.28.1"
]
]
}
],
"lcname": "blocklist-aggregator"
}
JSON
