dnsurlsearch


Namednsurlsearch JSON
Version 0.0.9 PyPI version JSON
download
home_pageNone
SummaryNone
upload_time2024-03-08 16:46:44
maintainerNone
docs_urlNone
authorNone
requires_python>=3
licenseNone
keywords pihole bind9 dns dns protocol sniffer tcpdump url
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # DNSurlSearch

[![PyPI - Version](https://img.shields.io/pypi/v/dnsurlsearch.svg)](https://pypi.org/project/dnsurlsearch)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dnsurlsearch.svg)](https://pypi.org/project/dnsurlsearch)

-----

**Table of Contents**

- [Installation](#Installation)
- [Getting started](#Getting_started)
- [Whitelist](#Whitelist)
- [Blacklist](#Blacklist)
- [Pihole](#Pihole)
- [License](#License)

# Installation

```console
pip install dnsurlsearch
```

# Getting_started
This package allow you to build a whitelist like that :
```
rr1.sn-cv0tb0xn-uane.googlevideo.com.
rr2.sn-cv0tb0xn-uane.googlevideo.com.
rr3.sn-cv1tb1xn-uane.googlevideo.com.
```
or a blacklist like that :
```
0.0.0.0 rr1.sn-cv0tb0xn-uane.googlevideo.com.
0.0.0.0 rr2.sn-cv0tb0xn-uane.googlevideo.com.
0.0.0.0 rr3.sn-cv1tb1xn-uane.googlevideo.com.
```
that you can save in a file and use with PiHole, for example, to filter urls.

The selection of urls is based on DNS protocol via exploration of cache file of DNS server like bind9 software
if you have installed bind9 on your localhost or via a sniffer in command line like tcpdump.

Initially the aim of this package was to filter advertisements on youTube video based on urls like *.googlevideo.com. Unfortunatedly,
some of these urls are not advertisements, so that we also need whitelist. 

# Whitelist
Whitelist generation is supposed to be manually. The script is launched and severals youTube video are played without any advertisement
to get url like *.googlevideo.com. without advertisement.

Here is an example of script example1.py (need to configure sudo to get rid of password) :

```python

from dnsurlsearch import DNSurlSearch

# First create url filter
whitelist = DNSurlSearch.UrlFilter()

# Create the cache
cache = DNSurlSearch.CacheDns()

# Set the SnifferCacheHandler to snif DNS protocol
h = DNSurlSearch.SnifferCacheHandler(DNSurlSearch.MANUAL)
h.set_cache_file_name('/tmp/log')

# The sniffer used is tcpdump
# The executable file tcpdump_cache_dns.sh contains the following command :
# "/usr/bin/tcpdump -n -s 0 port 53 > $1 &"
# to sniff DNS protocol
h.set_start_cmd('sudo /home/<user>/tcpdump_cache_dns.sh')
# The executable file k_tcpdump_cache_dns.sh contains the following command :
# "pkill tcpdump"
# to kill process
h.set_kill_cmd('sudo /home/<user>/k_tcpdump_cache_dns.sh')
cache.set_handler(h)
    
# Set patterns to search in the cache
cache.set_filter('.googlevideo.com.')
cache.set_filter('.other1.com.')
cache.set_filter('.other2.com.')
    
# Launch the sniffer to get url and search urls
new_cache = cache.get_cache_dns()
        
# Add urls found in the whitelist
whitelist.add(new_cache)
# Save the whitelist in a file
whitelist.write_url('whitelist')

```
You can launch this script with debug option (debug, info, warning, error, critical)
```bash
$ python3 example.py level=debug
2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 278 ----------------- CacheDns.set_handler()() -----------------
2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 282 ----------------- CacheDns.set_filter()() ------------------
2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 284 filters : ['.googlevideo.com.']
2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 526 --------------------- init_dns_cache() ---------------------
2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 528 sudo /home/<user>/tcpdump_cache_dns.sh /tmp/log
Type enter to stop tcpdump...
2024-02-26 10:02:32,922 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 537 sudo /home/<user>/k_tcpdump_cache_dns.sh
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 313 ---------------- CacheHandler._search_url() ----------------
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 314  pattern : ['.* (.+\\.googlevideo\\.com\\.).*'] - filename : Found_url.txt
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 328 Le cache est vide
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 160 --------------------- UrlFilter.add() ----------------------
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 161  Number of url to add : 0
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 143 ------------------ UrlFilter.write_url() -------------------
2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 144  Number of urls to write : 0 - file name : whitelist 
```

# Blacklist
Blacklist is supposed to be automatic. A script is launched by crontab every 10 minutes via crontab for example.

Here is an example of script that can be launched on a host 
where a DNS server like bind9 is installed:
```python
from dnsurlsearch import DNSurlSearch

# Read Whitelist if any
whitelist = DNSurlSearch.UrlFilter()
whitelist.read_url('whitelist')

# To get some measurements of urls found in the cache sent by mail
# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters
# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950
#        Number of new url found on the period  : 4
#        Total number of url found              : 4
stats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')

# First create url filter
# '0.0.0.0' parameter is the IP address to add in the file for each url
blacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')

# treatment  
#  - when reading blacklist file : remove ip_address to get url only
#  - when writting blacklist file : add ip_address 
blacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())

nb_url_start = blacklist.read_url('blacklist')

# Create the cache
cache = DNSurlSearch.CacheDns()

# Set the BindCacheHandler to snif cache of the DNS server bind9
h = DNSurlSearch.BindCacheHandler()

# Location of the file where rndc dump the cache of DNS server
h.set_cache_file_name('/var/cache/bind/named_dump.db')

# Set the command to dump the cache of DNS server
h.set_start_cmd('/usr/sbin/rndc dumpdb -cache')
cache.set_handler(h)

# Set patterns to search in the cache
cache.set_filter('.googlevideo.com.')

# Launch the update of cache and search urls
new_cache = cache.get_cache_dns()

# Add urls found in the blacklist
blacklist.add(new_cache)

# Check urls and remove url from blacklist if present in whitelist
blacklist.check(whitelist.get_url())

nb_url_end = blacklist.write_url('blacklist')

stats.save_stats(nb_url_start, nb_url_end, 144)
```

Here is an example of another script example3.py that can be launched on a host 
where DNS protocol can be sniffed with tcpdump. The handler is initialized in automatic mode during 600 seconds. 
After 600 seconds, the sniffer is automatically killed by the handler.
```python
from dnsurlsearch import DNSurlSearch

# Read Whitelist if any
whitelist = DNSurlSearch.UrlFilter()
whitelist.read_url('whitelist')

# To get some measurements of urls found in the cache sent by mail
# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters
# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950
#        Number of new url found on the period  : 4
#        Total number of url found              : 4
stats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')

# First create url filter
# '0.0.0.0' parameter is the IP address to add in the file for each url
blacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')

# treatment  
#  - when reading blacklist file : remove ip_address to get url only
#  - when writting blacklist file : add ip_address 
blacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())

nb_url_start = blacklist.read_url('blacklist')

# Create the cache
cache = DNSurlSearch.CacheDns()

# Set the SnifferCacheHandler to snif DNS protocol
h = DNSurlSearch.SnifferCacheHandler(AUTOMATIC, 600)
h.set_cache_file_name('/tmp/log')

# The sniffer used is tcpdump
# The executable file tcpdump_cache_dns.sh contains the following command :
# "/usr/bin/tcpdump -n -s 0 port 53 > $1 &"
# to sniff DNS protocol
h.set_start_cmd('sudo /home/<user>/tcpdump_cache_dns.sh')
# The executable file k_tcpdump_cache_dns.sh contains the following command :
# "pkill tcpdump"
# to kill process
h.set_kill_cmd('sudo /home/<user>/k_tcpdump_cache_dns.sh')
cache.set_handler(h)

# Set patterns to search in the cache
cache.set_filter('.googlevideo.com.')

# Launch the update of cache and search urls
new_cache = cache.get_cache_dns()

# Add urls found in the blacklist
blacklist.add(new_cache)

# Check urls and remove url from blacklist if present in whitelist
blacklist.check(whitelist.get_url())

nb_url_end = blacklist.write_url('blacklist')

stats.save_stats(nb_url_start, nb_url_end, 144)

```

```bash
$ python3 example3.py level=debug
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 140 ------------------- UrlFilter.read_url() -------------------
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 140 ------------------- UrlFilter.read_url() -------------------
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 313 ----------------- CacheDns.set_handler()() -----------------
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 317 ----------------- CacheDns.set_filter()() ------------------
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 319 filters : ['.googlevideo.com.']
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 561 --------------------- init_dns_cache() ---------------------
2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 563 sudo /home/<user>/tcpdump_cache_dns.sh /tmp/log
2024-03-03 11:55:05,122 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 569 Waiting for 20 seconds (0 minutes) before killing sniffer
2024-03-03 11:55:05,122 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 570 2024-03-03 11:55:05
2024-03-03 11:55:25,133 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 572 2024-03-03 11:55:25
2024-03-03 11:55:25,133 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 575 sudo /home/<user>/k_tcpdump_cache_dns.sh
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 343 Cache DNS : /tmp/log 03-03-2024 11:55 0.9453125 Ko
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 360 ---------------- CacheHandler._search_url() ----------------
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 361  pattern : ['.* (.+\\.googlevideo\\.com\\.).*'] - filename : Found_url.txt
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 375 Le cache est vide
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 184 --------------------- UrlFilter.add() ----------------------
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 185  Number of url to add : 0
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 192 0 New urls inserted
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 158 ------------------- UrlFilter.get_url() --------------------
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 196 -------------------- UrlFilter.check() ---------------------
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 204 0 Urls deleted (whitelist)
2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 167 ------------------ UrlFilter.write_url() -------------------
2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 168  Number of urls to write : 0 - file name : blacklist 
2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 267 ---------------- CacheDnsStat.save_stats() -----------------
2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 282 Start date: 2024-03-03 11:50:00.388145
 nb url orig: 0 total nb url: 0
2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 236 --------------- CacheDnsStat.send_message() ----------------
2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 237 Start date : 2024-03-03 11:50:00.388145
 nb new url : 0 total nb url : 0
2024-03-03 11:55:25,212 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 246 Message : 
Mesure period                          : 2024-03-03 11:50:00.388145 - 2024-03-03 11:55:25.174378
Number of new url found on the period  : 0
Total number of url found              : 0

```

# Pihole

Here is an example of another script example4.py that can be launched on a host 
where pihole server is installed (as root). In this example the log file of pihole is analyzed (have a look at /etc/dnsmasq.d/01-pihole.conf)

```python
from dnsurlsearch import DNSurlSearch

# Read Whitelist if any
whitelist = DNSurlSearch.UrlFilter()
whitelist.read_url('whitelist')

# To get some measurements of urls found in the cache sent by mail
# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters
# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950
#        Number of new url found on the period  : 4
#        Total number of url found              : 4
stats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')

# First create url filter
# '0.0.0.0' parameter is the IP address to add in the file for each url
blacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')

# treatment  
#  - when reading blacklist file : remove ip_address to get url only
#  - when writting blacklist file : add ip_address 
blacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())

nb_url_start = blacklist.read_url('blacklist')

# Create the cache
cache = DNSurlSearch.CacheDns()

# Set the SnifferCacheHandler to snif DNS protocol
h = DNSurlSearch.SnifferCacheHandler(DNSurlSearch.AUTOMATIC, 1)
h.set_cache_file_name('/var/log/pihole/pihole.log')

h.set_start_cmd('')
h.set_kill_cmd('')
cache.set_handler(h)

# Set patterns to search in the cache
cache.set_filter('.googlevideo.com')

# Launch the update of cache and search urls
new_cache = cache.get_cache_dns()

# Add urls found in the blacklist
blacklist.add(new_cache)

# Check urls and remove url from blacklist if present in whitelist
blacklist.check(whitelist.get_url())

nb_url_end = blacklist.write_url('blacklist')

stats.save_stats(nb_url_start, nb_url_end, 144)
```

# License

`dnsurlsearch` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "dnsurlsearch",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3",
    "maintainer_email": null,
    "keywords": "PiHole,bind9,dns,dns protocol,sniffer,tcpdump,url",
    "author": null,
    "author_email": "Nicolas Jeudy <nicola.jeudy@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/ec/b3/1d886a4df0c8c4116c01440cbc2b8e1e52bc908a33259b982de22d5c03dd/dnsurlsearch-0.0.9.tar.gz",
    "platform": null,
    "description": "# DNSurlSearch\n\n[![PyPI - Version](https://img.shields.io/pypi/v/dnsurlsearch.svg)](https://pypi.org/project/dnsurlsearch)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dnsurlsearch.svg)](https://pypi.org/project/dnsurlsearch)\n\n-----\n\n**Table of Contents**\n\n- [Installation](#Installation)\n- [Getting started](#Getting_started)\n- [Whitelist](#Whitelist)\n- [Blacklist](#Blacklist)\n- [Pihole](#Pihole)\n- [License](#License)\n\n# Installation\n\n```console\npip install dnsurlsearch\n```\n\n# Getting_started\nThis package allow you to build a whitelist like that :\n```\nrr1.sn-cv0tb0xn-uane.googlevideo.com.\nrr2.sn-cv0tb0xn-uane.googlevideo.com.\nrr3.sn-cv1tb1xn-uane.googlevideo.com.\n```\nor a blacklist like that :\n```\n0.0.0.0 rr1.sn-cv0tb0xn-uane.googlevideo.com.\n0.0.0.0 rr2.sn-cv0tb0xn-uane.googlevideo.com.\n0.0.0.0 rr3.sn-cv1tb1xn-uane.googlevideo.com.\n```\nthat you can save in a file and use with PiHole, for example, to filter urls.\n\nThe selection of urls is based on DNS protocol via exploration of cache file of DNS server like bind9 software\nif you have installed bind9 on your localhost or via a sniffer in command line like tcpdump.\n\nInitially the aim of this package was to filter advertisements on youTube video based on urls like *.googlevideo.com. Unfortunatedly,\nsome of these urls are not advertisements, so that we also need whitelist. \n\n# Whitelist\nWhitelist generation is supposed to be manually. The script is launched and severals youTube video are played without any advertisement\nto get url like *.googlevideo.com. without advertisement.\n\nHere is an example of script example1.py (need to configure sudo to get rid of password) :\n\n```python\n\nfrom dnsurlsearch import DNSurlSearch\n\n# First create url filter\nwhitelist = DNSurlSearch.UrlFilter()\n\n# Create the cache\ncache = DNSurlSearch.CacheDns()\n\n# Set the SnifferCacheHandler to snif DNS protocol\nh = DNSurlSearch.SnifferCacheHandler(DNSurlSearch.MANUAL)\nh.set_cache_file_name('/tmp/log')\n\n# The sniffer used is tcpdump\n# The executable file tcpdump_cache_dns.sh contains the following command :\n# \"/usr/bin/tcpdump -n -s 0 port 53 > $1 &\"\n# to sniff DNS protocol\nh.set_start_cmd('sudo /home/<user>/tcpdump_cache_dns.sh')\n# The executable file k_tcpdump_cache_dns.sh contains the following command :\n# \"pkill tcpdump\"\n# to kill process\nh.set_kill_cmd('sudo /home/<user>/k_tcpdump_cache_dns.sh')\ncache.set_handler(h)\n    \n# Set patterns to search in the cache\ncache.set_filter('.googlevideo.com.')\ncache.set_filter('.other1.com.')\ncache.set_filter('.other2.com.')\n    \n# Launch the sniffer to get url and search urls\nnew_cache = cache.get_cache_dns()\n        \n# Add urls found in the whitelist\nwhitelist.add(new_cache)\n# Save the whitelist in a file\nwhitelist.write_url('whitelist')\n\n```\nYou can launch this script with debug option (debug, info, warning, error, critical)\n```bash\n$ python3 example.py level=debug\n2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 278 ----------------- CacheDns.set_handler()() -----------------\n2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 282 ----------------- CacheDns.set_filter()() ------------------\n2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 284 filters : ['.googlevideo.com.']\n2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 526 --------------------- init_dns_cache() ---------------------\n2024-02-26 10:02:30,531 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 528 sudo /home/<user>/tcpdump_cache_dns.sh /tmp/log\nType enter to stop tcpdump...\n2024-02-26 10:02:32,922 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 537 sudo /home/<user>/k_tcpdump_cache_dns.sh\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 313 ---------------- CacheHandler._search_url() ----------------\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 314  pattern : ['.* (.+\\\\.googlevideo\\\\.com\\\\.).*'] - filename : Found_url.txt\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 328 Le cache est vide\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 160 --------------------- UrlFilter.add() ----------------------\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 161  Number of url to add : 0\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 143 ------------------ UrlFilter.write_url() -------------------\n2024-02-26 10:02:32,941 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 144  Number of urls to write : 0 - file name : whitelist \n```\n\n# Blacklist\nBlacklist is supposed to be automatic. A script is launched by crontab every 10 minutes via crontab for example.\n\nHere is an example of script that can be launched on a host \nwhere a DNS server like bind9 is installed:\n```python\nfrom dnsurlsearch import DNSurlSearch\n\n# Read Whitelist if any\nwhitelist = DNSurlSearch.UrlFilter()\nwhitelist.read_url('whitelist')\n\n# To get some measurements of urls found in the cache sent by mail\n# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters\n# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950\n#        Number of new url found on the period  : 4\n#        Total number of url found              : 4\nstats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')\n\n# First create url filter\n# '0.0.0.0' parameter is the IP address to add in the file for each url\nblacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')\n\n# treatment  \n#  - when reading blacklist file : remove ip_address to get url only\n#  - when writting blacklist file : add ip_address \nblacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())\n\nnb_url_start = blacklist.read_url('blacklist')\n\n# Create the cache\ncache = DNSurlSearch.CacheDns()\n\n# Set the BindCacheHandler to snif cache of the DNS server bind9\nh = DNSurlSearch.BindCacheHandler()\n\n# Location of the file where rndc dump the cache of DNS server\nh.set_cache_file_name('/var/cache/bind/named_dump.db')\n\n# Set the command to dump the cache of DNS server\nh.set_start_cmd('/usr/sbin/rndc dumpdb -cache')\ncache.set_handler(h)\n\n# Set patterns to search in the cache\ncache.set_filter('.googlevideo.com.')\n\n# Launch the update of cache and search urls\nnew_cache = cache.get_cache_dns()\n\n# Add urls found in the blacklist\nblacklist.add(new_cache)\n\n# Check urls and remove url from blacklist if present in whitelist\nblacklist.check(whitelist.get_url())\n\nnb_url_end = blacklist.write_url('blacklist')\n\nstats.save_stats(nb_url_start, nb_url_end, 144)\n```\n\nHere is an example of another script example3.py that can be launched on a host \nwhere DNS protocol can be sniffed with tcpdump. The handler is initialized in automatic mode during 600 seconds. \nAfter 600 seconds, the sniffer is automatically killed by the handler.\n```python\nfrom dnsurlsearch import DNSurlSearch\n\n# Read Whitelist if any\nwhitelist = DNSurlSearch.UrlFilter()\nwhitelist.read_url('whitelist')\n\n# To get some measurements of urls found in the cache sent by mail\n# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters\n# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950\n#        Number of new url found on the period  : 4\n#        Total number of url found              : 4\nstats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')\n\n# First create url filter\n# '0.0.0.0' parameter is the IP address to add in the file for each url\nblacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')\n\n# treatment  \n#  - when reading blacklist file : remove ip_address to get url only\n#  - when writting blacklist file : add ip_address \nblacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())\n\nnb_url_start = blacklist.read_url('blacklist')\n\n# Create the cache\ncache = DNSurlSearch.CacheDns()\n\n# Set the SnifferCacheHandler to snif DNS protocol\nh = DNSurlSearch.SnifferCacheHandler(AUTOMATIC, 600)\nh.set_cache_file_name('/tmp/log')\n\n# The sniffer used is tcpdump\n# The executable file tcpdump_cache_dns.sh contains the following command :\n# \"/usr/bin/tcpdump -n -s 0 port 53 > $1 &\"\n# to sniff DNS protocol\nh.set_start_cmd('sudo /home/<user>/tcpdump_cache_dns.sh')\n# The executable file k_tcpdump_cache_dns.sh contains the following command :\n# \"pkill tcpdump\"\n# to kill process\nh.set_kill_cmd('sudo /home/<user>/k_tcpdump_cache_dns.sh')\ncache.set_handler(h)\n\n# Set patterns to search in the cache\ncache.set_filter('.googlevideo.com.')\n\n# Launch the update of cache and search urls\nnew_cache = cache.get_cache_dns()\n\n# Add urls found in the blacklist\nblacklist.add(new_cache)\n\n# Check urls and remove url from blacklist if present in whitelist\nblacklist.check(whitelist.get_url())\n\nnb_url_end = blacklist.write_url('blacklist')\n\nstats.save_stats(nb_url_start, nb_url_end, 144)\n\n```\n\n```bash\n$ python3 example3.py level=debug\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 140 ------------------- UrlFilter.read_url() -------------------\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 140 ------------------- UrlFilter.read_url() -------------------\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 313 ----------------- CacheDns.set_handler()() -----------------\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 317 ----------------- CacheDns.set_filter()() ------------------\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 319 filters : ['.googlevideo.com.']\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 561 --------------------- init_dns_cache() ---------------------\n2024-03-03 11:55:05,115 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 563 sudo /home/<user>/tcpdump_cache_dns.sh /tmp/log\n2024-03-03 11:55:05,122 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 569 Waiting for 20 seconds (0 minutes) before killing sniffer\n2024-03-03 11:55:05,122 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 570 2024-03-03 11:55:05\n2024-03-03 11:55:25,133 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 572 2024-03-03 11:55:25\n2024-03-03 11:55:25,133 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 575 sudo /home/<user>/k_tcpdump_cache_dns.sh\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 343 Cache DNS : /tmp/log 03-03-2024 11:55 0.9453125 Ko\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 360 ---------------- CacheHandler._search_url() ----------------\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 361  pattern : ['.* (.+\\\\.googlevideo\\\\.com\\\\.).*'] - filename : Found_url.txt\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 375 Le cache est vide\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 184 --------------------- UrlFilter.add() ----------------------\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 185  Number of url to add : 0\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 192 0 New urls inserted\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 158 ------------------- UrlFilter.get_url() --------------------\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 196 -------------------- UrlFilter.check() ---------------------\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch INFO DNSurlSearch.py 204 0 Urls deleted (whitelist)\n2024-03-03 11:55:25,173 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 167 ------------------ UrlFilter.write_url() -------------------\n2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 168  Number of urls to write : 0 - file name : blacklist \n2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 267 ---------------- CacheDnsStat.save_stats() -----------------\n2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 282 Start date: 2024-03-03 11:50:00.388145\n nb url orig: 0 total nb url: 0\n2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 236 --------------- CacheDnsStat.send_message() ----------------\n2024-03-03 11:55:25,174 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 237 Start date : 2024-03-03 11:50:00.388145\n nb new url : 0 total nb url : 0\n2024-03-03 11:55:25,212 dnsurlsearch.DNSurlSearch DEBUG DNSurlSearch.py 246 Message : \nMesure period                          : 2024-03-03 11:50:00.388145 - 2024-03-03 11:55:25.174378\nNumber of new url found on the period  : 0\nTotal number of url found              : 0\n\n```\n\n# Pihole\n\nHere is an example of another script example4.py that can be launched on a host \nwhere pihole server is installed (as root). In this example the log file of pihole is analyzed (have a look at /etc/dnsmasq.d/01-pihole.conf)\n\n```python\nfrom dnsurlsearch import DNSurlSearch\n\n# Read Whitelist if any\nwhitelist = DNSurlSearch.UrlFilter()\nwhitelist.read_url('whitelist')\n\n# To get some measurements of urls found in the cache sent by mail\n# See https://docs.python.org/2.6/library/logging.html?highlight=logger#smtp-handler for more information on parameters\n# Mesure period                          : 2024-02-05 18:51:01.128016 - 2024-02-05 18:51:42.497950\n#        Number of new url found on the period  : 4\n#        Total number of url found              : 4\nstats = DNSurlSearch.CacheDnsStat(<mailhost>, <fromaddrmail>, <toaddrsmail>, 'Filtered url stats')\n\n# First create url filter\n# '0.0.0.0' parameter is the IP address to add in the file for each url\nblacklist = DNSurlSearch.UrlFilter('0.0.0.0', 'black_list')\n\n# treatment  \n#  - when reading blacklist file : remove ip_address to get url only\n#  - when writting blacklist file : add ip_address \nblacklist.set_treatment(DNSurlSearch.BlackListFilterTreatment())\n\nnb_url_start = blacklist.read_url('blacklist')\n\n# Create the cache\ncache = DNSurlSearch.CacheDns()\n\n# Set the SnifferCacheHandler to snif DNS protocol\nh = DNSurlSearch.SnifferCacheHandler(DNSurlSearch.AUTOMATIC, 1)\nh.set_cache_file_name('/var/log/pihole/pihole.log')\n\nh.set_start_cmd('')\nh.set_kill_cmd('')\ncache.set_handler(h)\n\n# Set patterns to search in the cache\ncache.set_filter('.googlevideo.com')\n\n# Launch the update of cache and search urls\nnew_cache = cache.get_cache_dns()\n\n# Add urls found in the blacklist\nblacklist.add(new_cache)\n\n# Check urls and remove url from blacklist if present in whitelist\nblacklist.check(whitelist.get_url())\n\nnb_url_end = blacklist.write_url('blacklist')\n\nstats.save_stats(nb_url_start, nb_url_end, 144)\n```\n\n# License\n\n`dnsurlsearch` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.\n",
    "bugtrack_url": null,
    "license": null,
    "summary": null,
    "version": "0.0.9",
    "project_urls": {
        "Documentation": "https://github.com/Nicola-31/dnsurlsearch#readme",
        "Issues": "https://github.com/Nicola-31/dnsurlsearch/issues",
        "Source": "https://github.com/Nicola-31/dnsurlsearch"
    },
    "split_keywords": [
        "pihole",
        "bind9",
        "dns",
        "dns protocol",
        "sniffer",
        "tcpdump",
        "url"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "01e6e4236718d264f13b062132b1a6814eafbdcc65055d0e5d37dc36abc5b236",
                "md5": "520295ad08b690094892e52e78d8ce54",
                "sha256": "20c4ad7a8d924d718c9811c136e1a8fff7863beb9ceda451e7efedcd9e4c01e7"
            },
            "downloads": -1,
            "filename": "dnsurlsearch-0.0.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "520295ad08b690094892e52e78d8ce54",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3",
            "size": 147707,
            "upload_time": "2024-03-08T16:46:35",
            "upload_time_iso_8601": "2024-03-08T16:46:35.265142Z",
            "url": "https://files.pythonhosted.org/packages/01/e6/e4236718d264f13b062132b1a6814eafbdcc65055d0e5d37dc36abc5b236/dnsurlsearch-0.0.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "ecb31d886a4df0c8c4116c01440cbc2b8e1e52bc908a33259b982de22d5c03dd",
                "md5": "06df2cd1f3e9a5661aad9aa65bd9fa7a",
                "sha256": "3f0b93de30ee35b63c2548a3f812e118137901e120a6f85a464ce9841f170dca"
            },
            "downloads": -1,
            "filename": "dnsurlsearch-0.0.9.tar.gz",
            "has_sig": false,
            "md5_digest": "06df2cd1f3e9a5661aad9aa65bd9fa7a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3",
            "size": 248727,
            "upload_time": "2024-03-08T16:46:44",
            "upload_time_iso_8601": "2024-03-08T16:46:44.675624Z",
            "url": "https://files.pythonhosted.org/packages/ec/b3/1d886a4df0c8c4116c01440cbc2b8e1e52bc908a33259b982de22d5c03dd/dnsurlsearch-0.0.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-03-08 16:46:44",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "Nicola-31",
    "github_project": "dnsurlsearch#readme",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "dnsurlsearch"
}
None
Elapsed time: 3.03589s