# pypdl
pypdl is a Python library for downloading files from the internet. It provides features such as multi-segmented downloads, retry download in case of failure, option to continue downloading using a different URL if necessary, progress tracking, pause/resume functionality, checksum and many more.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Usage](#usage)
- [Basic Usage](#basic-usage)
- [Advanced Usage](#advanced-usage)
- [Examples](#examples)
- [API Reference](#api-reference)
- [License](#license)
- [Contribution](#contribution)
- [Contact](#contact)
## Prerequisites
* Python 3.8 or later.
## Installation
To install the pypdl, run the following command:
```bash
pip install pypdl
```
## Usage
### Basic Usage
To download a file using the pypdl, simply create a new `Pypdl` object and call its `start` method, passing in the URL of the file to be downloaded:
```py
from pypdl import Pypdl
dl = Pypdl()
dl.start('http://example.com/file.txt')
```
### Advanced Usage
The `Pypdl` object provides additional options for advanced usage:
```py
from pypdl import Pypdl
dl = Pypdl(allow_reuse=False, logger=default_logger("Pypdl"))
dl.start(
url='http://example.com/file.txt',
file_path='file.txt',
multisegment=True,
segments=10,
overwrite=True,
etag=True,
retries=0,
mirror_func=None,
display=True,
clear_terminal=True,
block=True
)
```
Each option is explained below:
- `allow_reuse`: Whether to allow reuse of existing Pypdl object for the next download. The default value is `False`.
- `logger`: A logger object to log messages. The default value is a custom `Logger` with the name *Pypdl*.
- `url`: This can either be the URL of the file to download or a function that returns the URL.
- `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded into the given path.
- `multisegment`: Whether to use multi-segmented download. The default value is `True`.
- `segments`: The number of segments the file should be divided into for multi-segmented download. The default value is 10.
- `overwrite`: Whether to overwrite the file if it already exists. The default value is `True`.
- `etag`: Whether to validate the ETag before resuming downloads. The default value is `True`.
- `retries`: The number of times to retry the download in case of an error. The default value is 0.
- `mirror_func`: A function to get a new download URL in case of an error. The default value is `None`.
- `display`: Whether to display download progress and other optional messages. The default value is `True`.
- `clear_terminal`: Whether to clear the terminal before displaying the download progress. The default value is `True`.
- `block`: Whether to block until the download is complete. The default value is `True`.
### Examples
Here is an example that demonstrates how to use pypdl library to download a file using headers, proxy and timeout:
```py
import aiohttp
from pypdl import Pypdl
def main():
# Using headers
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"}
# Using proxy
proxy = "http://user:pass@some.proxy.com"
# Using timeout
timeout = aiohttp.ClientTimeout(sock_read=20)
# create a new pypdl object
dl = Pypdl(headers=headers, proxy=proxy, timeout=timeout)
# start the download
dl.start(
url='https://speed.hetzner.de/100MB.bin',
file_path='100MB.bin',
segments=10,
display=True,
multisegment=True,
block=True,
retries=3,
mirror_func=None,
etag=True,
)
if __name__ == '__main__':
main()
```
This example downloads a file from the internet using 10 segments and displays the download progress. If the download fails, it will retry up to 3 times. we are also using headers, proxy and timeout, For more info regarding these parameters refer [API reference](https://github.com/mjishnu/pypdl?tab=readme-ov-file#pypdl-1)
Another example of implementing pause resume functionality, printing the progress to console and changing log level to debug:
```py
from pypdl import Pypdl
# create a pypdl object
dl = Pypdl()
# changing log level to debug
dl.logger.setLevel('DEBUG')
# start the download process
# block=False so we can print the progress
# display=False so we can print the progress ourselves
dl.start('https://example.com/file.zip', segments=8,block=False,display=False)
# print the progress
while dl.progress != 70:
print(dl.progress)
# stop the download process
dl.stop()
#do something
#...
# resume the download process
dl.start('https://example.com/file.zip', segments=8,block=False,display=False)
# print rest of the progress
while not d.completed:
print(dl.progress)
```
This example we start the download process and print the progress to console. We then stop the download process and do something else. After that we resume the download process and print the rest of the progress to console. This can be used to create a pause/resume functionality.
Another example of using hash validation with dynamic url:
```py
from pypdl import Pypdl
# Generate the url dynamically
def dynamic_url():
return 'https://example.com/file.zip'
# create a pypdl object
dl = Pypdl()
# if block = True --> returns a FileValidator object
file = dl.start(dynamic_url, block=True)
# validate hash
if file.validate_hash(correct_hash,'sha256'):
print('Hash is valid')
else:
print('Hash is invalid')
# scenario where block = False --> returns a AutoShutdownFuture object
file = dl.start(dynamic_url, block=False)
# do something
# ...
# validate hash
if dl.completed:
if file.result().validate_hash(correct_hash,'sha256'):
print('Hash is valid')
else:
print('Hash is invalid')
```
An example of using Pypdl object to get size of the files with `allow_reuse` set to `True` and custom logger:
```py
import logging
import time
from pypdl import Pypdl
urls = [
'https://example.com/file1.zip',
'https://example.com/file2.zip',
'https://example.com/file3.zip',
'https://example.com/file4.zip',
'https://example.com/file5.zip',
]
# create a custom logger
logger = logging.getLogger('custom')
size = []
# create a pypdl object
dl = Pypdl(allow_reuse=True, logger=logger)
for url in urls:
dl.start(url, block=False)
# waiting for the size and other preliminary data to be retrived
while dl.wait:
time.sleep(0.1)
# get the size of the file and add it to size list
size.append(dl.size)
# do something
while not dl.completed:
print(dl.progress)
print(size)
# shutdown the downloader, this is essential when allow_reuse is enabled
dl.shutdown()
```
An example of using `PypdlFactory` to download multiple files concurrently:
```py
from pypdl import PypdlFactory
proxy = "http://user:pass@some.proxy.com"
# create a PypdlFactory object
factory = PypdlFactory(instances=5, allow_reuse=True, proxy=proxy)
# List of tasks to be downloaded. Each task is a tuple of (URL, {Pypdl arguments}).
# - URL: The download link (string).
# - {Pypdl arguments}: A dictionary of arguments supported by `Pypdl`.
tasks = [
('https://example.com/file1.zip', {'file_path': 'file1.zip'}),
('https://example.com/file2.zip', {'file_path': 'file2.zip'}),
('https://example.com/file3.zip', {'file_path': 'file3.zip'}),
('https://example.com/file4.zip', {'file_path': 'file4.zip'}),
('https://example.com/file5.zip', {'file_path': 'file5.zip'}),
]
# start the download process
results = factory.start(tasks, display=True, block=False)
# do something
# ...
# stop the download process
factory.stop()
# do something
# ...
# restart the download process
results = factory.start(tasks, display=True, block=True)
# print the results
for url, result in results:
# validate hash
if result.validate_hash(correct_hash,'sha256'):
print(f'{url} - Hash is valid')
else:
print(f'{url} - Hash is invalid')
task2 = [
('https://example.com/file6.zip', {'file_path': 'file6.zip'}),
('https://example.com/file7.zip', {'file_path': 'file7.zip'}),
('https://example.com/file8.zip', {'file_path': 'file8.zip'}),
('https://example.com/file9.zip', {'file_path': 'file9.zip'}),
('https://example.com/file10.zip', {'file_path': 'file10.zip'}),
]
# start the download process
factory.start(task2, display=True, block=True)
# shutdown the downloader, this is essential when allow_reuse is enabled
factory.shutdown()
```
For more detailed info about parameters refer [API reference](https://github.com/mjishnu/pypdl?tab=readme-ov-file#pypdlfactory)
## API Reference
### `Pypdl()`
The `Pypdl` class represents a file downloader that can download a file from a given URL to a specified file path. The class supports both single-segmented and multi-segmented downloads and many other features like retry download incase of failure and option to continue downloading using a different url if necessary, pause/resume functionality, progress tracking etc.
#### Arguments
- `allow_reuse`: (bool, Optional) Whether to allow reuse of existing `Pypdl` object for next download. The default value is `False`.It's essential to use `shutdown()` method when `allow_reuse` is enabled to ensure efficient resource management.
- `logger`: (logging.Logger, Optional) A logger object to log messages. The default value is custom `Logger` with the name *Pypdl*.
- Supported Keyword Arguments:
- `params`: Parameters to be sent in the query string of the new request. The default value is `None`.
- `data`: The data to send in the body of the request. The default value is `None`.
- `json`: A JSON-compatible Python object to send in the body of the request. The default value is `None`.
- `cookies`: HTTP Cookies to send with the request. The default value is `None`.
- `headers`: HTTP Headers to send with the request. The default value is `None`.
- `auth`: An object that represents HTTP Basic Authorization. The default value is `None`.
- `allow_redirects`: If set to False, do not follow redirects. The default value is `True`.
- `max_redirects`: Maximum number of redirects to follow. The default value is `10`.
- `proxy`: Proxy URL. The default value is `None`.
- `proxy_auth`: An object that represents proxy HTTP Basic Authorization. The default value is `None`.
- `timeout`: (default `aiohttp.ClientTimeout(sock_read=60)`): Override the session’s timeout. The default value is `aiohttp.ClientTimeout(sock_read=60)`.
- `ssl`: SSL validation mode. The default value is `None`.
- `proxy_headers`: HTTP headers to send to the proxy if the `proxy` parameter has been provided. The default value is `None`.
For detailed information on each parameter, refer the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request). Please ensure that only the *supported keyword arguments* are used. Using unsupported or irrelevant keyword arguments may lead to unexpected behavior or errors.
#### Attributes
- `size`: The total size of the file to be downloaded, in bytes.
- `progress`: The download progress percentage.
- `speed`: The download speed, in MB/s.
- `time_spent`: The time spent downloading, in seconds.
- `current_size`: The amount of data downloaded so far, in bytes.
- `eta`: The estimated time remaining for download completion, in the format "HH:MM:SS".
- `remaining`: The amount of data remaining to be downloaded, in bytes.
- `failed`: A flag that indicates if the download failed.
- `completed`: A flag that indicates if the download is complete.
- `wait`: A flag indicating whether preliminary information (e.g., file size) has been retrieved.
- `logger`: The logger object used for logging messages.
#### Methods
- `start(url,
file_path=None,
multisegment=True,
segments=10,
overwrite=True,
etag=True,
retries=0,
mirror_func=None,
display=True,
clear_terminal=True,
block=True)`: Starts the download process.
##### Parameters
- `url`: This can either be the URL of the file to download or a function that returns the URL.
- `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded into the given path.
- `multisegment`: Whether to use multi-segmented download. The default value is `True`.
- `segments`: The number of segments the file should be divided into for multi-segmented download. The default value is 10.
- `overwrite`: Whether to overwrite the file if it already exists. The default value is `True`.
- `etag`: Whether to validate the ETag before resuming downloads. The default value is `True`.
- `retries`: The number of times to retry the download in case of an error. The default value is 0.
- `mirror_func`: A function to get a new download URL in case of an error. The default value is `None`.
- `display`: Whether to display download progress and other optional messages. The default value is `True`.
- `clear_terminal`: Whether to clear the terminal before displaying the download progress. The default value is `True`.
- `block`: Whether to block until the download is complete. The default value is `True`.
##### Returns
- `AutoShutdownFuture`: If `block` and `allow_reuse` is set to `False`.
- `concurrent.futures.Future`: If `block` is `False` and `allow_reuse` is `True`.
- `FileValidator`: If `block` is `True` and the download is successful.
- `None`: If `block` is `True` and the download fails.
- `stop()`: Stops the download process.
- `shutdown()`: Shuts down the downloader.
### `PypdlFactory()`
The `PypdlFactory` class manages multiple instances of the `Pypdl` downloader. It allows for concurrent downloads and provides progress tracking across all active downloads.
#### Arguments
- `instances`: (int, Optional) The number of `Pypdl` instances to create. The default value is 5.
- `allow_reuse`: (bool, Optional) Whether to allow reuse of existing `PypdlFactory` objects for next download. The default value is `False`. It's essential to use `shutdown()` method when `allow_reuse` is enabled to ensure efficient resource management.
- `logger`: (logging.Logger, Optional) A logger object to log messages. The default value is custom `Logger` with the name *PypdlFactory*.
- Supported Keyword Arguments:
- `params`: Parameters to be sent in the query string of the new request. The default value is `None`.
- `data`: The data to send in the body of the request. The default value is `None`.
- `json`: A JSON-compatible Python object to send in the body of the request. The default value is `None`.
- `cookies`: HTTP Cookies to send with the request. The default value is `None`.
- `headers`: HTTP Headers to send with the request. The default value is `None`.
- `auth`: An object that represents HTTP Basic Authorization. The default value is `None`.
- `allow_redirects`: If set to False, do not follow redirects. The default value is `True`.
- `max_redirects`: Maximum number of redirects to follow. The default value is `10`.
- `proxy`: Proxy URL. The default value is `None`.
- `proxy_auth`: An object that represents proxy HTTP Basic Authorization. The default value is `None`.
- `timeout`: (default `aiohttp.ClientTimeout(sock_read=60)`): Override the session’s timeout. The default value is `aiohttp.ClientTimeout(sock_read=60)`.
- `ssl`: SSL validation mode. The default value is `None`.
- `proxy_headers`: HTTP headers to send to the proxy if the `proxy` parameter has been provided. The default value is `None`.
For detailed information on each parameter, refer the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request). Please ensure that only the *supported keyword arguments* are used. Using unsupported or irrelevant keyword arguments may lead to unexpected behavior or errors.
#### Attributes
- `progress`: The overall download progress percentage across all active downloads.
- `speed`: The average download speed across all active downloads, in MB/s.
- `time_spent`: The total time spent downloading across all active downloads, in seconds.
- `current_size`: The total amount of data downloaded so far across all active downloads, in bytes.
- `total`: The total number of download tasks.
- `success`: A list of tuples where each tuple contains the URL of the download and the `FileValidator` of the download.
- `failed`: A list of URLs for which the download failed.
- `remaining`: A list of remaining download tasks.
- `completed`: A flag to check if all tasks are completed.
- `logger`: The logger object used for logging messages.
#### Methods
- `start(tasks, display=True, clear_terminal=True, block=True)`: Starts the download process for multiple tasks.
##### Parameters
- `tasks`: (list) A list of tasks to be downloaded. Each task is a tuple where the first element is the URL and the second element is an optional dictionary with keyword arguments for `Pypdl` start method.
- `display`: (bool, Optional) Whether to display download progress and other messages. Default is True.
- `clear_terminal`: (bool, Optional) Whether to clear the terminal before displaying the download progress. Default is True.
- `block`: (bool, Optional) Whether to block the function until all downloads are complete. Default is True.
##### Returns
- `AutoShutdownFuture`: If `block` and `allow_reuse` is set to `False`.
- `concurrent.futures.Future`: If `block` is `False` and `allow_reuse` is `True`.
- `list`: If `block` is `True`. This is a list of tuples where each tuple contains the URL of the download and the `FileValidator` of the download.
- `stop()`: Stops all active downloads.
- `shutdown()`: Shuts down the factory.
### Helper Classes
#### `Basicdown()`
The `Basicdown` class is the base downloader class that provides the basic structure for downloading files.
##### Attributes
- `curr`: The current size of the downloaded file in bytes.
- `completed`: A flag that indicates if the download is complete.
- `interrupt`: A flag that indicates if the download was interrupted.
- `downloaded`: The total amount of data downloaded so far in bytes.
##### Methods
- `download(url, path, mode, session, **kwargs)`: Downloads data in chunks.
#### `Singledown()`
The `Singledown` class extends `Basicdown` and is responsible for downloading a whole file in a single segment.
##### Methods
- `worker(url, file_path, session, **kwargs)`: Downloads a whole file in a single segment.
#### `Multidown()`
The `Multidown` class extends `Basicdown` and is responsible for downloading a specific segment of a file.
##### Methods
- `worker(segment_table, id, session, **kwargs)`: Downloads a part of the file in multiple segments.
#### `FileValidator()`
The `FileValidator` class is used to validate the integrity of the downloaded file.
##### Parameters
- `path`: The path of the file to be validated.
##### Methods
- `calculate_hash(algorithm, **kwargs)`: Calculates the hash of the file using the specified algorithm. Returns the calculated hash as a string.
- `validate_hash(correct_hash, algorithm, **kwargs)`: Validates the hash of the file against the correct hash. Returns `True` if the hashes match, `False` otherwise.
`calculate_hash` and `validate_hash` can support additional keyword arguments from the [hashlib module](https://docs.python.org/3/library/hashlib.html#hashlib.new).
#### `AutoShutdownFuture()`
The `AutoShutdownFuture` class is a wrapper for concurrent.futures.Future object that shuts down a list of associated executors when the result is retrieved.
##### Parameters
- `future`: The Future object to be wrapped.
- `executors`: The list of executors to be shut down when the result is retrieved.
##### Methods
- `result(timeout=None)`: Retrieves the result of the Future object and shuts down the executor. If the download was successful, it returns a `FileValidator` object; otherwise, it returns `None`.
## License
pypdl is licensed under the MIT License. See the [LICENSE](https://github.com/mjishnu/pypdl/blob/main/LICENSE) file for more details.
## Contribution
Contributions to pypdl are always welcome. If you want to contribute to this project, please fork the repository and submit a pull request.
## Contact
If you have any questions, issues, or feedback about pypdl, please open an issue on the [GitHub repository](https://github.com/mjishnu/pypdl).
Raw data
{
"_id": null,
"home_page": "https://github.com/mjishnu/pypdl",
"name": "pypdl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "python, downloader, multi-threaded-downloader, concurrent-downloader, parallel-downloader, async-downloader, asyncronous-downloader, download-manager, fast-downloader, download-accelerator, download-optimizer, download-utility, download-tool, download-automation",
"author": "mjishnu",
"author_email": "<mjishnu@proton.me>",
"download_url": "https://files.pythonhosted.org/packages/05/eb/7761564a97d880f34358476b33f0410f664235c1069b980a02badea4d16b/pypdl-1.4.5.tar.gz",
"platform": null,
"description": "# pypdl\n\npypdl is a Python library for downloading files from the internet. It provides features such as multi-segmented downloads, retry download in case of failure, option to continue downloading using a different URL if necessary, progress tracking, pause/resume functionality, checksum and many more.\n\n## Table of Contents\n\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Usage](#usage)\n - [Basic Usage](#basic-usage)\n - [Advanced Usage](#advanced-usage)\n - [Examples](#examples)\n- [API Reference](#api-reference)\n- [License](#license)\n- [Contribution](#contribution)\n- [Contact](#contact)\n\n## Prerequisites\n\n* Python 3.8 or later.\n\n## Installation\n\nTo install the pypdl, run the following command:\n\n\n```bash\npip install pypdl\n```\n## Usage\n\n### Basic Usage\n\nTo download a file using the pypdl, simply create a new `Pypdl` object and call its `start` method, passing in the URL of the file to be downloaded:\n\n```py\nfrom pypdl import Pypdl\n\ndl = Pypdl()\ndl.start('http://example.com/file.txt')\n```\n\n### Advanced Usage\n\nThe `Pypdl` object provides additional options for advanced usage:\n\n```py\nfrom pypdl import Pypdl\n\ndl = Pypdl(allow_reuse=False, logger=default_logger(\"Pypdl\"))\ndl.start(\n url='http://example.com/file.txt',\n file_path='file.txt',\n multisegment=True,\n segments=10,\n overwrite=True,\n etag=True,\n retries=0,\n mirror_func=None,\n display=True,\n clear_terminal=True,\n block=True\n)\n```\n\nEach option is explained below:\n- `allow_reuse`: Whether to allow reuse of existing Pypdl object for the next download. The default value is `False`.\n- `logger`: A logger object to log messages. The default value is a custom `Logger` with the name *Pypdl*.\n- `url`: This can either be the URL of the file to download or a function that returns the URL.\n- `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded into the given path.\n- `multisegment`: Whether to use multi-segmented download. The default value is `True`.\n- `segments`: The number of segments the file should be divided into for multi-segmented download. The default value is 10.\n- `overwrite`: Whether to overwrite the file if it already exists. The default value is `True`.\n- `etag`: Whether to validate the ETag before resuming downloads. The default value is `True`.\n- `retries`: The number of times to retry the download in case of an error. The default value is 0.\n- `mirror_func`: A function to get a new download URL in case of an error. The default value is `None`.\n- `display`: Whether to display download progress and other optional messages. The default value is `True`.\n- `clear_terminal`: Whether to clear the terminal before displaying the download progress. The default value is `True`.\n- `block`: Whether to block until the download is complete. The default value is `True`.\n\n### Examples\n\nHere is an example that demonstrates how to use pypdl library to download a file using headers, proxy and timeout:\n\n```py\nimport aiohttp\nfrom pypdl import Pypdl\n\ndef main():\n # Using headers\n headers = {\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0\"}\n # Using proxy\n proxy = \"http://user:pass@some.proxy.com\"\n # Using timeout\n timeout = aiohttp.ClientTimeout(sock_read=20)\n\n # create a new pypdl object\n dl = Pypdl(headers=headers, proxy=proxy, timeout=timeout)\n\n # start the download\n dl.start(\n url='https://speed.hetzner.de/100MB.bin',\n file_path='100MB.bin',\n segments=10,\n display=True,\n multisegment=True,\n block=True,\n retries=3,\n mirror_func=None,\n etag=True,\n )\n\nif __name__ == '__main__':\n main()\n```\n\nThis example downloads a file from the internet using 10 segments and displays the download progress. If the download fails, it will retry up to 3 times. we are also using headers, proxy and timeout, For more info regarding these parameters refer [API reference](https://github.com/mjishnu/pypdl?tab=readme-ov-file#pypdl-1)\n\nAnother example of implementing pause resume functionality, printing the progress to console and changing log level to debug:\n\n```py\nfrom pypdl import Pypdl\n\n# create a pypdl object\ndl = Pypdl()\n\n# changing log level to debug\ndl.logger.setLevel('DEBUG')\n\n# start the download process\n# block=False so we can print the progress\n# display=False so we can print the progress ourselves\ndl.start('https://example.com/file.zip', segments=8,block=False,display=False)\n\n# print the progress\nwhile dl.progress != 70:\n print(dl.progress)\n\n# stop the download process\ndl.stop() \n\n#do something\n#...\n\n# resume the download process\ndl.start('https://example.com/file.zip', segments=8,block=False,display=False)\n\n# print rest of the progress\nwhile not d.completed:\n print(dl.progress)\n\n```\n\nThis example we start the download process and print the progress to console. We then stop the download process and do something else. After that we resume the download process and print the rest of the progress to console. This can be used to create a pause/resume functionality.\n\nAnother example of using hash validation with dynamic url:\n\n```py\nfrom pypdl import Pypdl\n\n# Generate the url dynamically\ndef dynamic_url():\n return 'https://example.com/file.zip'\n\n# create a pypdl object\ndl = Pypdl()\n\n# if block = True --> returns a FileValidator object\nfile = dl.start(dynamic_url, block=True) \n\n# validate hash\nif file.validate_hash(correct_hash,'sha256'):\n print('Hash is valid')\nelse:\n print('Hash is invalid')\n\n# scenario where block = False --> returns a AutoShutdownFuture object\nfile = dl.start(dynamic_url, block=False)\n\n# do something\n# ...\n\n# validate hash\nif dl.completed:\n if file.result().validate_hash(correct_hash,'sha256'):\n print('Hash is valid')\n else:\n print('Hash is invalid')\n```\nAn example of using Pypdl object to get size of the files with `allow_reuse` set to `True` and custom logger:\n\n```py\nimport logging\nimport time\nfrom pypdl import Pypdl\n\nurls = [\n 'https://example.com/file1.zip',\n 'https://example.com/file2.zip',\n 'https://example.com/file3.zip',\n 'https://example.com/file4.zip',\n 'https://example.com/file5.zip',\n]\n\n# create a custom logger\nlogger = logging.getLogger('custom')\n\nsize = []\n\n# create a pypdl object\ndl = Pypdl(allow_reuse=True, logger=logger)\n\nfor url in urls:\n dl.start(url, block=False)\n\n # waiting for the size and other preliminary data to be retrived\n while dl.wait:\n time.sleep(0.1)\n \n # get the size of the file and add it to size list\n size.append(dl.size)\n\n # do something \n\n while not dl.completed:\n print(dl.progress)\n\nprint(size)\n# shutdown the downloader, this is essential when allow_reuse is enabled\ndl.shutdown()\n\n```\n\n\nAn example of using `PypdlFactory` to download multiple files concurrently:\n\n```py\nfrom pypdl import PypdlFactory\n\nproxy = \"http://user:pass@some.proxy.com\"\n\n# create a PypdlFactory object\nfactory = PypdlFactory(instances=5, allow_reuse=True, proxy=proxy)\n\n# List of tasks to be downloaded. Each task is a tuple of (URL, {Pypdl arguments}).\n# - URL: The download link (string).\n# - {Pypdl arguments}: A dictionary of arguments supported by `Pypdl`.\ntasks = [\n ('https://example.com/file1.zip', {'file_path': 'file1.zip'}),\n ('https://example.com/file2.zip', {'file_path': 'file2.zip'}),\n ('https://example.com/file3.zip', {'file_path': 'file3.zip'}),\n ('https://example.com/file4.zip', {'file_path': 'file4.zip'}),\n ('https://example.com/file5.zip', {'file_path': 'file5.zip'}),\n]\n\n# start the download process\nresults = factory.start(tasks, display=True, block=False)\n\n# do something\n# ...\n\n# stop the download process\nfactory.stop()\n\n# do something\n# ...\n\n# restart the download process\nresults = factory.start(tasks, display=True, block=True)\n\n# print the results\nfor url, result in results:\n # validate hash\n if result.validate_hash(correct_hash,'sha256'):\n print(f'{url} - Hash is valid')\n else:\n print(f'{url} - Hash is invalid')\n\ntask2 = [\n ('https://example.com/file6.zip', {'file_path': 'file6.zip'}),\n ('https://example.com/file7.zip', {'file_path': 'file7.zip'}),\n ('https://example.com/file8.zip', {'file_path': 'file8.zip'}),\n ('https://example.com/file9.zip', {'file_path': 'file9.zip'}),\n ('https://example.com/file10.zip', {'file_path': 'file10.zip'}),\n]\n\n# start the download process\nfactory.start(task2, display=True, block=True)\n\n# shutdown the downloader, this is essential when allow_reuse is enabled\nfactory.shutdown()\n```\nFor more detailed info about parameters refer [API reference](https://github.com/mjishnu/pypdl?tab=readme-ov-file#pypdlfactory)\n## API Reference\n\n### `Pypdl()`\n\nThe `Pypdl` class represents a file downloader that can download a file from a given URL to a specified file path. The class supports both single-segmented and multi-segmented downloads and many other features like retry download incase of failure and option to continue downloading using a different url if necessary, pause/resume functionality, progress tracking etc.\n\n#### Arguments\n- `allow_reuse`: (bool, Optional) Whether to allow reuse of existing `Pypdl` object for next download. The default value is `False`.It's essential to use `shutdown()` method when `allow_reuse` is enabled to ensure efficient resource management.\n\n- `logger`: (logging.Logger, Optional) A logger object to log messages. The default value is custom `Logger` with the name *Pypdl*.\n\n- Supported Keyword Arguments:\n - `params`: Parameters to be sent in the query string of the new request. The default value is `None`.\n - `data`: The data to send in the body of the request. The default value is `None`.\n - `json`: A JSON-compatible Python object to send in the body of the request. The default value is `None`.\n - `cookies`: HTTP Cookies to send with the request. The default value is `None`.\n - `headers`: HTTP Headers to send with the request. The default value is `None`.\n - `auth`: An object that represents HTTP Basic Authorization. The default value is `None`.\n - `allow_redirects`: If set to False, do not follow redirects. The default value is `True`.\n - `max_redirects`: Maximum number of redirects to follow. The default value is `10`.\n - `proxy`: Proxy URL. The default value is `None`.\n - `proxy_auth`: An object that represents proxy HTTP Basic Authorization. The default value is `None`.\n - `timeout`: (default `aiohttp.ClientTimeout(sock_read=60)`): Override the session\u2019s timeout. The default value is `aiohttp.ClientTimeout(sock_read=60)`.\n - `ssl`: SSL validation mode. The default value is `None`.\n - `proxy_headers`: HTTP headers to send to the proxy if the `proxy` parameter has been provided. The default value is `None`.\n\n For detailed information on each parameter, refer the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request). Please ensure that only the *supported keyword arguments* are used. Using unsupported or irrelevant keyword arguments may lead to unexpected behavior or errors.\n \n#### Attributes\n\n- `size`: The total size of the file to be downloaded, in bytes.\n- `progress`: The download progress percentage.\n- `speed`: The download speed, in MB/s.\n- `time_spent`: The time spent downloading, in seconds.\n- `current_size`: The amount of data downloaded so far, in bytes.\n- `eta`: The estimated time remaining for download completion, in the format \"HH:MM:SS\".\n- `remaining`: The amount of data remaining to be downloaded, in bytes.\n- `failed`: A flag that indicates if the download failed.\n- `completed`: A flag that indicates if the download is complete.\n- `wait`: A flag indicating whether preliminary information (e.g., file size) has been retrieved.\n- `logger`: The logger object used for logging messages.\n\n#### Methods\n\n- `start(url,\nfile_path=None,\nmultisegment=True,\nsegments=10,\noverwrite=True,\netag=True,\nretries=0,\nmirror_func=None,\ndisplay=True,\nclear_terminal=True,\nblock=True)`: Starts the download process.\n\n ##### Parameters\n\n - `url`: This can either be the URL of the file to download or a function that returns the URL.\n - `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded into the given path.\n - `multisegment`: Whether to use multi-segmented download. The default value is `True`.\n - `segments`: The number of segments the file should be divided into for multi-segmented download. The default value is 10.\n - `overwrite`: Whether to overwrite the file if it already exists. The default value is `True`.\n - `etag`: Whether to validate the ETag before resuming downloads. The default value is `True`.\n - `retries`: The number of times to retry the download in case of an error. The default value is 0.\n - `mirror_func`: A function to get a new download URL in case of an error. The default value is `None`.\n - `display`: Whether to display download progress and other optional messages. The default value is `True`.\n - `clear_terminal`: Whether to clear the terminal before displaying the download progress. The default value is `True`.\n - `block`: Whether to block until the download is complete. The default value is `True`.\n\n ##### Returns\n \n - `AutoShutdownFuture`: If `block` and `allow_reuse` is set to `False`.\n - `concurrent.futures.Future`: If `block` is `False` and `allow_reuse` is `True`.\n - `FileValidator`: If `block` is `True` and the download is successful.\n - `None`: If `block` is `True` and the download fails.\n\n- `stop()`: Stops the download process.\n- `shutdown()`: Shuts down the downloader.\n\n### `PypdlFactory()`\n\nThe `PypdlFactory` class manages multiple instances of the `Pypdl` downloader. It allows for concurrent downloads and provides progress tracking across all active downloads.\n\n#### Arguments\n\n- `instances`: (int, Optional) The number of `Pypdl` instances to create. The default value is 5.\n- `allow_reuse`: (bool, Optional) Whether to allow reuse of existing `PypdlFactory` objects for next download. The default value is `False`. It's essential to use `shutdown()` method when `allow_reuse` is enabled to ensure efficient resource management.\n\n- `logger`: (logging.Logger, Optional) A logger object to log messages. The default value is custom `Logger` with the name *PypdlFactory*.\n\n- Supported Keyword Arguments:\n - `params`: Parameters to be sent in the query string of the new request. The default value is `None`.\n - `data`: The data to send in the body of the request. The default value is `None`.\n - `json`: A JSON-compatible Python object to send in the body of the request. The default value is `None`.\n - `cookies`: HTTP Cookies to send with the request. The default value is `None`.\n - `headers`: HTTP Headers to send with the request. The default value is `None`.\n - `auth`: An object that represents HTTP Basic Authorization. The default value is `None`.\n - `allow_redirects`: If set to False, do not follow redirects. The default value is `True`.\n - `max_redirects`: Maximum number of redirects to follow. The default value is `10`.\n - `proxy`: Proxy URL. The default value is `None`.\n - `proxy_auth`: An object that represents proxy HTTP Basic Authorization. The default value is `None`.\n - `timeout`: (default `aiohttp.ClientTimeout(sock_read=60)`): Override the session\u2019s timeout. The default value is `aiohttp.ClientTimeout(sock_read=60)`.\n - `ssl`: SSL validation mode. The default value is `None`.\n - `proxy_headers`: HTTP headers to send to the proxy if the `proxy` parameter has been provided. The default value is `None`.\n\n For detailed information on each parameter, refer the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request). Please ensure that only the *supported keyword arguments* are used. Using unsupported or irrelevant keyword arguments may lead to unexpected behavior or errors.\n\n#### Attributes\n\n- `progress`: The overall download progress percentage across all active downloads.\n- `speed`: The average download speed across all active downloads, in MB/s.\n- `time_spent`: The total time spent downloading across all active downloads, in seconds.\n- `current_size`: The total amount of data downloaded so far across all active downloads, in bytes.\n- `total`: The total number of download tasks.\n- `success`: A list of tuples where each tuple contains the URL of the download and the `FileValidator` of the download.\n- `failed`: A list of URLs for which the download failed.\n- `remaining`: A list of remaining download tasks.\n- `completed`: A flag to check if all tasks are completed.\n- `logger`: The logger object used for logging messages.\n\n#### Methods\n\n- `start(tasks, display=True, clear_terminal=True, block=True)`: Starts the download process for multiple tasks.\n\n ##### Parameters\n\n - `tasks`: (list) A list of tasks to be downloaded. Each task is a tuple where the first element is the URL and the second element is an optional dictionary with keyword arguments for `Pypdl` start method.\n - `display`: (bool, Optional) Whether to display download progress and other messages. Default is True.\n - `clear_terminal`: (bool, Optional) Whether to clear the terminal before displaying the download progress. Default is True.\n - `block`: (bool, Optional) Whether to block the function until all downloads are complete. Default is True.\n\n ##### Returns\n\n - `AutoShutdownFuture`: If `block` and `allow_reuse` is set to `False`.\n - `concurrent.futures.Future`: If `block` is `False` and `allow_reuse` is `True`.\n - `list`: If `block` is `True`. This is a list of tuples where each tuple contains the URL of the download and the `FileValidator` of the download.\n\n- `stop()`: Stops all active downloads.\n- `shutdown()`: Shuts down the factory.\n\n### Helper Classes\n\n#### `Basicdown()`\n\nThe `Basicdown` class is the base downloader class that provides the basic structure for downloading files.\n\n##### Attributes\n\n- `curr`: The current size of the downloaded file in bytes.\n- `completed`: A flag that indicates if the download is complete.\n- `interrupt`: A flag that indicates if the download was interrupted.\n- `downloaded`: The total amount of data downloaded so far in bytes.\n\n##### Methods\n\n- `download(url, path, mode, session, **kwargs)`: Downloads data in chunks.\n\n#### `Singledown()`\n\nThe `Singledown` class extends `Basicdown` and is responsible for downloading a whole file in a single segment.\n\n##### Methods\n\n- `worker(url, file_path, session, **kwargs)`: Downloads a whole file in a single segment.\n\n#### `Multidown()`\n\nThe `Multidown` class extends `Basicdown` and is responsible for downloading a specific segment of a file.\n\n##### Methods\n\n- `worker(segment_table, id, session, **kwargs)`: Downloads a part of the file in multiple segments.\n\n#### `FileValidator()`\n\nThe `FileValidator` class is used to validate the integrity of the downloaded file.\n\n##### Parameters\n\n- `path`: The path of the file to be validated.\n\n##### Methods\n\n- `calculate_hash(algorithm, **kwargs)`: Calculates the hash of the file using the specified algorithm. Returns the calculated hash as a string.\n\n- `validate_hash(correct_hash, algorithm, **kwargs)`: Validates the hash of the file against the correct hash. Returns `True` if the hashes match, `False` otherwise.\n\n `calculate_hash` and `validate_hash` can support additional keyword arguments from the [hashlib module](https://docs.python.org/3/library/hashlib.html#hashlib.new).\n\n#### `AutoShutdownFuture()`\n\nThe `AutoShutdownFuture` class is a wrapper for concurrent.futures.Future object that shuts down a list of associated executors when the result is retrieved.\n\n##### Parameters\n\n- `future`: The Future object to be wrapped.\n- `executors`: The list of executors to be shut down when the result is retrieved.\n\n##### Methods\n\n- `result(timeout=None)`: Retrieves the result of the Future object and shuts down the executor. If the download was successful, it returns a `FileValidator` object; otherwise, it returns `None`.\n\n## License\n\npypdl is licensed under the MIT License. See the [LICENSE](https://github.com/mjishnu/pypdl/blob/main/LICENSE) file for more details.\n\n## Contribution\n\nContributions to pypdl are always welcome. If you want to contribute to this project, please fork the repository and submit a pull request.\n\n## Contact\n\nIf you have any questions, issues, or feedback about pypdl, please open an issue on the [GitHub repository](https://github.com/mjishnu/pypdl).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A concurrent pure python download manager",
"version": "1.4.5",
"project_urls": {
"Homepage": "https://github.com/mjishnu/pypdl"
},
"split_keywords": [
"python",
" downloader",
" multi-threaded-downloader",
" concurrent-downloader",
" parallel-downloader",
" async-downloader",
" asyncronous-downloader",
" download-manager",
" fast-downloader",
" download-accelerator",
" download-optimizer",
" download-utility",
" download-tool",
" download-automation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "191d533560670b9acd870b7386b1b49d968d2483c74cba75dcde9f6db988a0ad",
"md5": "387e5a4e7694b9f88a94d8c3884a0916",
"sha256": "605e896ad50e26dd1fa3604cd97f692315263873255f1e8acfa4dfd93974cfe2"
},
"downloads": -1,
"filename": "pypdl-1.4.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "387e5a4e7694b9f88a94d8c3884a0916",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16736,
"upload_time": "2024-12-02T06:49:59",
"upload_time_iso_8601": "2024-12-02T06:49:59.057993Z",
"url": "https://files.pythonhosted.org/packages/19/1d/533560670b9acd870b7386b1b49d968d2483c74cba75dcde9f6db988a0ad/pypdl-1.4.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "05eb7761564a97d880f34358476b33f0410f664235c1069b980a02badea4d16b",
"md5": "5f17c4c8e77fbeff0e94debacd20b2f9",
"sha256": "403751ffddfaf25788b3fe86240fe71376aed571605d696f662f85b73eadf009"
},
"downloads": -1,
"filename": "pypdl-1.4.5.tar.gz",
"has_sig": false,
"md5_digest": "5f17c4c8e77fbeff0e94debacd20b2f9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 19575,
"upload_time": "2024-12-02T06:50:01",
"upload_time_iso_8601": "2024-12-02T06:50:01.266488Z",
"url": "https://files.pythonhosted.org/packages/05/eb/7761564a97d880f34358476b33f0410f664235c1069b980a02badea4d16b/pypdl-1.4.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-02 06:50:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mjishnu",
"github_project": "pypdl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pypdl"
}
JSON
