# pypdl
pypdl is a Python library for downloading files from the internet. It provides features such as multi-threaded downloads, retry download in case of failure, option to continue downloading using a different URL if necessary, progress tracking, pause/resume functionality, checksum and many more.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Usage](#usage)
- [Basic Usage](#basic-usage)
- [Advanced Usage](#advanced-usage)
- [Examples](#examples)
- [API Reference](#api-reference)
- [License](#license)
- [Contribution](#contribution)
- [Contact](#contact)
## Prerequisites
* Python 3.8 or later.
## Installation
To install the pypdl, run the following command:
```bash
pip install pypdl
```
## Usage
### Basic Usage
To download a file using the pypdl, simply create a new `Downloader` object and call its `start` method, passing in the URL of the file to be downloaded:
```py
from pypdl import Downloader
dl = Downloader()
dl.start('http://example.com/file.txt')
```
### Advanced Usage
The `Downloader` object provides additional options for advanced usage:
```py
dl.start(
url='http://example.com/file.txt',
file_path='file.txt',
segments=10,
display=True,
multithread=True,
block=True,
retries=0,
mirror_func=None,
etag=True
)
```
Each option is explained below:
- `url`: The URL of the file to download.
- `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it otherwise, the file is downloaded into the given path.
- `segments`: The number of segments the file should be divided in multi-threaded download. The default value is 10.
- `display`: Whether to display download progress and other optional messages. The default value is `True`.
- `multithread`: Whether to use multi-threaded download. The default value is `True`.
- `block`: Whether to block until the download is complete. The default value is `True`.
- `retries`: The number of times to retry the download in case of an error. The default value is 0.
- `mirror_func`: A function to get a new download URL in case of an error.
- `etag`: Whether to validate etag before resuming downloads. The default value is `True`.
### Examples
Here is an example that demonstrates how to use pypdl library to download a file using headers, proxies and authentication:
```py
from pypdl import Downloader
def main():
# Using headers
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"}
# Using proxies
proxies = {
"http": "http://10.10.1.10:3128",
"https": "https://10.10.1.10:1080",
}
# Using authentication
auth = ("user","pass")
# create a new downloader object
dl = Downloader(headers=headers, proxies=proxies, auth=auth)
# start the download
dl.start(
url='https://speed.hetzner.de/100MB.bin',
file_path='100MB.bin',
segments=10,
display=True,
multithread=True,
block=True,
retries=3,
mirror_func=None,
etag=True,
)
if __name__ == '__main__':
main()
```
This example downloads a file from the internet using 10 threads and displays the download progress. If the download fails, it will retry up to 3 times. we are also using headers, proxies and authentication.
Another example of implementing pause resume functionality and printing the progress to console:
```py
from pypdl import Downloader
from threading import Event
# create a downloader object
dl = Downloader()
# start the download process
# block=False so we can print the progress
# display=False so we can print the progress ourselves
dl.start('https://example.com/file.zip', segments=8,block=False,display=False)
# print the progress
while dl.progress != 70:
print(dl.progress)
# stop the download process
dl.stop()
#do something
#...
# resume the download process
dl.start('https://example.com/file.zip', segments=8,block=False,display=False)
# print rest of the progress
while not d.completed:
print(dl.progress)
```
This example we start the download process and print the progress to console. We then stop the download process and do something else. After that we resume the download process and print the rest of the progress to console. This can be used to create a pause/resume functionality.
Another example of using hash validation:
```py
from pypdl import Downloader
# create a downloader object
dl = Downloader()
# if block = True --> returns a FileValidator object
file = dl.start('https://example.com/file.zip', block=True)
# validate hash
if file.validate_hash(correct_hash,'sha256'):
print('Hash is valid')
else:
print('Hash is invalid')
# if block = False --> returns a AutoShutdownFuture object
file = dl.start('https://example.com/file.zip', block=False)
# do something
# ...
# validate hash
if dl.completed:
if file.result().validate_hash(correct_hash,'sha256'):
print('Hash is valid')
else:
print('Hash is invalid')
```
## API Reference
### `Downloader()`
The `Downloader` class represents a file downloader that can download a file from a given URL to a specified file path. The class supports both single-threaded and multi-threaded downloads and many other features like retry download incase of failure and option to continue downloading using a different url if necessary, pause/resume functionality, progress tracking etc.
#### Keyword Arguments
- `params`: (dict, Optional) A dictionary, list of tuples or bytes to send as a query string. Default is None.
- `allow_redirects`: (bool, Optional) A Boolean to enable/disable redirection. Default is True.
- `auth`: (tuple, Optional) A tuple to enable a certain HTTP authentication. Default is None.
- `cert`: (str or tuple, Optional) A String or Tuple specifying a cert file or key. Default is None.
- `cookies`: (dict, Optional) A dictionary of cookies to send to the specified url. Default is None.
- `headers`: (dict, Optional) A dictionary of HTTP headers to send to the specified url. Default is None.
- `proxies`: (dict, Optional) A dictionary of the protocol to the proxy url. Default is None.
- `timeout`: (number or tuple, Optional) A number, or a tuple, indicating how many seconds to wait for the client to make a connection and/or send a response. Default is 20 seconds.
- `verify`: (bool or str, Optional) A Boolean or a String indication to verify the servers TLS certificate or not. Default is True.
#### Attributes
- `size`: The total size of the file to be downloaded, in bytes.
- `progress`: The download progress percentage.
- `speed`: The download speed, in MB/s.
- `time_spent`: The time spent downloading, in seconds.
- `current_size`: The amount of data downloaded so far, in bytes.
- `eta`: The estimated time remaining for download completion, in the format "HH:MM:SS".
- `remaining`: The amount of data remaining to be downloaded, in bytes.
- `failed`: A flag that indicates if the download failed.
- `completed`: A flag that indicates if the download is complete.
#### Methods
- `start(url, file_path, segments=10, display=True, multithread=True, block=True, retries=0, mirror_func=None, etag=False)`: Starts the download process.
##### Parameters
- `url`: (str) The download URL.
- `file_path`: (str, Optional) The optional file path to save the download. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded with the given name.
- `segments`: (int, Optional) The number of segments the file should be divided into for multi-threaded download.
- `display`: (bool, Optional) Whether to display download progress and other optional messages.
- `multithread`: (bool, Optional) Whether to use multi-threaded download.
- `block`: (bool, Optional) Whether to block until the download is complete.
- `retries`: (int, Optional) The number of times to retry the download in case of an error.
- `mirror_func`: (function, Optional) A function to get a new download URL in case of an error.
- `etag`: (bool, Optional) Whether to validate etag before resuming downloads.
##### Returns
- `AutoShutdownFuture`: If `block` is `False`.
- `FileValidator`: If `block` is `True` and the download is successful.
- `None`: If `block` is `True` and the download fails.
- `stop()`: Stops the download process.
### Helper Classes
#### `Basicdown()`
The `Basicdown` class is the base downloader class that provides the basic structure for downloading files.
##### Attributes
- `curr`: The current size of the downloaded file in bytes.
- `completed`: A flag that indicates if the download is complete.
- `id`: The ID of the current instance.
- `stop`: An event that can be used to stop the download process.
- `error`: An event that can be used to signal an error.
##### Methods
- `download(url, path, mode, **kwargs)`: Downloads data in chunks.
#### `Simpledown()`
The `Simpledown` class extends `Basicdown` and is responsible for downloading a whole file in a single segment.
##### Parameters
- `url`: Url of the file.
- `file_path`: Path to save the file.
- `stop`: Stop event.
- `error`: Error event.
- `**kwargs`: Additional keyword arguments.
##### Methods
- `worker()`: Downloads a whole file in a single segment.
#### `Multidown()`
The `Multidown` class extends `Basicdown` and is responsible for downloading a specific segment of a file.
##### Parameters
- `segement_table`: Dictionary that contains the download information.
- `segment_id`: ID of the download part.
- `stop`: Stop event.
- `error`: Error event.
- `**kwargs`: Additional keyword arguments.
##### Methods
- `worker()`: Downloads a part of the file in multiple segments.
#### `FileValidator()`
The `FileValidator` class is used to validate the integrity of the downloaded file.
##### Parameters
- `path`: The path of the file to be validated.
##### Methods
- `calculate_hash(algorithm, **kwargs)`: Calculates the hash of the file using the specified algorithm. Returns the calculated hash as a string.
- `validate_hash(correct_hash, algorithm, **kwargs)`: Validates the hash of the file against the correct hash. Returns `True` if the hashes match, `False` otherwise.
`calculate_hash` and `validate_hash` can support additional keyword arguments from the [hashlib module](https://docs.python.org/3/library/hashlib.html#hashlib.new).
#### `AutoShutdownFuture()`
The `AutoShutdownFuture` class is a wrapper for concurrent.futures.Future object that shuts down the executor when the result is retrieved.
##### Parameters
- `future`: The Future object to be wrapped.
- `executor`: The executor to be shut down when the result is retrieved.
##### Methods
- `result(timeout=None)`: Retrieves the result of the Future object and shuts down the executor. If the download was successful, it returns a `FileValidator` object; otherwise, it returns `None`.
## License
pypdl is licensed under the MIT License. See the [LICENSE](https://github.com/mjishnu/pypdl/blob/main/LICENSE) file for more details.
## Contribution
Contributions to pypdl are always welcome. If you want to contribute to this project, please fork the repository and submit a pull request.
## Contact
If you have any questions, issues, or feedback about pypdl, please open an issue on the [GitHub repository](https://github.com/mjishnu/pypdl).
Raw data
{
"_id": null,
"home_page": "https://github.com/mjishnu/pypdl",
"name": "pypdl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "python, downloader, concurrent-downloader, parrel-downloader, download manager, fast-downloader",
"author": "mjishnu",
"author_email": "<mjishnu@skiff.com>",
"download_url": "https://files.pythonhosted.org/packages/9d/fc/4fc3e0a961563051ed1ec65bba62f627fb2f6e0c845298b033aae9d89b6f/pypdl-1.3.2.tar.gz",
"platform": null,
"description": "# pypdl\n\npypdl is a Python library for downloading files from the internet. It provides features such as multi-threaded downloads, retry download in case of failure, option to continue downloading using a different URL if necessary, progress tracking, pause/resume functionality, checksum and many more.\n\n## Table of Contents\n\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Usage](#usage)\n - [Basic Usage](#basic-usage)\n - [Advanced Usage](#advanced-usage)\n - [Examples](#examples)\n- [API Reference](#api-reference)\n- [License](#license)\n- [Contribution](#contribution)\n- [Contact](#contact)\n\n## Prerequisites\n\n* Python 3.8 or later.\n\n## Installation\n\nTo install the pypdl, run the following command:\n\n\n```bash\npip install pypdl\n```\n## Usage\n\n### Basic Usage\n\nTo download a file using the pypdl, simply create a new `Downloader` object and call its `start` method, passing in the URL of the file to be downloaded:\n\n```py\nfrom pypdl import Downloader\n\ndl = Downloader()\ndl.start('http://example.com/file.txt')\n```\n\n### Advanced Usage\n\nThe `Downloader` object provides additional options for advanced usage:\n\n```py\ndl.start(\n url='http://example.com/file.txt',\n file_path='file.txt',\n segments=10,\n display=True,\n multithread=True,\n block=True,\n retries=0,\n mirror_func=None,\n etag=True\n)\n```\n\nEach option is explained below:\n\n- `url`: The URL of the file to download.\n- `file_path`: An optional path to save the downloaded file. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it otherwise, the file is downloaded into the given path.\n- `segments`: The number of segments the file should be divided in multi-threaded download. The default value is 10.\n- `display`: Whether to display download progress and other optional messages. The default value is `True`.\n- `multithread`: Whether to use multi-threaded download. The default value is `True`.\n- `block`: Whether to block until the download is complete. The default value is `True`.\n- `retries`: The number of times to retry the download in case of an error. The default value is 0.\n- `mirror_func`: A function to get a new download URL in case of an error.\n- `etag`: Whether to validate etag before resuming downloads. The default value is `True`.\n\n### Examples\n\nHere is an example that demonstrates how to use pypdl library to download a file using headers, proxies and authentication:\n\n```py\nfrom pypdl import Downloader\n\ndef main():\n # Using headers\n headers = {\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0\"}\n # Using proxies\n proxies = {\n \"http\": \"http://10.10.1.10:3128\",\n \"https\": \"https://10.10.1.10:1080\",\n }\n # Using authentication\n auth = (\"user\",\"pass\")\n\n # create a new downloader object\n dl = Downloader(headers=headers, proxies=proxies, auth=auth)\n\n # start the download\n dl.start(\n url='https://speed.hetzner.de/100MB.bin',\n file_path='100MB.bin',\n segments=10,\n display=True,\n multithread=True,\n block=True,\n retries=3,\n mirror_func=None,\n etag=True,\n )\n\nif __name__ == '__main__':\n main()\n```\n\nThis example downloads a file from the internet using 10 threads and displays the download progress. If the download fails, it will retry up to 3 times. we are also using headers, proxies and authentication.\n\nAnother example of implementing pause resume functionality and printing the progress to console:\n\n```py\nfrom pypdl import Downloader\nfrom threading import Event\n\n# create a downloader object\ndl = Downloader()\n\n# start the download process\n# block=False so we can print the progress\n# display=False so we can print the progress ourselves\ndl.start('https://example.com/file.zip', segments=8,block=False,display=False)\n\n# print the progress\nwhile dl.progress != 70:\n print(dl.progress)\n\n# stop the download process\ndl.stop() \n\n#do something\n#...\n\n# resume the download process\ndl.start('https://example.com/file.zip', segments=8,block=False,display=False)\n\n# print rest of the progress\nwhile not d.completed:\n print(dl.progress)\n\n```\n\nThis example we start the download process and print the progress to console. We then stop the download process and do something else. After that we resume the download process and print the rest of the progress to console. This can be used to create a pause/resume functionality.\n\nAnother example of using hash validation:\n\n```py\nfrom pypdl import Downloader\n\n# create a downloader object\ndl = Downloader()\n\n# if block = True --> returns a FileValidator object\nfile = dl.start('https://example.com/file.zip', block=True) \n\n# validate hash\nif file.validate_hash(correct_hash,'sha256'):\n print('Hash is valid')\nelse:\n print('Hash is invalid')\n\n# if block = False --> returns a AutoShutdownFuture object\nfile = dl.start('https://example.com/file.zip', block=False)\n\n# do something\n# ...\n\n# validate hash\nif dl.completed:\n if file.result().validate_hash(correct_hash,'sha256'):\n print('Hash is valid')\n else:\n print('Hash is invalid')\n```\n\n## API Reference\n\n### `Downloader()`\n\nThe `Downloader` class represents a file downloader that can download a file from a given URL to a specified file path. The class supports both single-threaded and multi-threaded downloads and many other features like retry download incase of failure and option to continue downloading using a different url if necessary, pause/resume functionality, progress tracking etc.\n\n#### Keyword Arguments\n\n- `params`: (dict, Optional) A dictionary, list of tuples or bytes to send as a query string. Default is None.\n- `allow_redirects`: (bool, Optional) A Boolean to enable/disable redirection. Default is True.\n- `auth`: (tuple, Optional) A tuple to enable a certain HTTP authentication. Default is None.\n- `cert`: (str or tuple, Optional) A String or Tuple specifying a cert file or key. Default is None.\n- `cookies`: (dict, Optional) A dictionary of cookies to send to the specified url. Default is None.\n- `headers`: (dict, Optional) A dictionary of HTTP headers to send to the specified url. Default is None.\n- `proxies`: (dict, Optional) A dictionary of the protocol to the proxy url. Default is None.\n- `timeout`: (number or tuple, Optional) A number, or a tuple, indicating how many seconds to wait for the client to make a connection and/or send a response. Default is 20 seconds.\n- `verify`: (bool or str, Optional) A Boolean or a String indication to verify the servers TLS certificate or not. Default is True.\n\n#### Attributes\n\n- `size`: The total size of the file to be downloaded, in bytes.\n- `progress`: The download progress percentage.\n- `speed`: The download speed, in MB/s.\n- `time_spent`: The time spent downloading, in seconds.\n- `current_size`: The amount of data downloaded so far, in bytes.\n- `eta`: The estimated time remaining for download completion, in the format \"HH:MM:SS\".\n- `remaining`: The amount of data remaining to be downloaded, in bytes.\n- `failed`: A flag that indicates if the download failed.\n- `completed`: A flag that indicates if the download is complete.\n\n#### Methods\n\n- `start(url, file_path, segments=10, display=True, multithread=True, block=True, retries=0, mirror_func=None, etag=False)`: Starts the download process.\n\n ##### Parameters\n\n - `url`: (str) The download URL.\n - `file_path`: (str, Optional) The optional file path to save the download. By default, it uses the present working directory. If `file_path` is a directory, then the file is downloaded into it; otherwise, the file is downloaded with the given name.\n - `segments`: (int, Optional) The number of segments the file should be divided into for multi-threaded download.\n - `display`: (bool, Optional) Whether to display download progress and other optional messages.\n - `multithread`: (bool, Optional) Whether to use multi-threaded download.\n - `block`: (bool, Optional) Whether to block until the download is complete.\n - `retries`: (int, Optional) The number of times to retry the download in case of an error.\n - `mirror_func`: (function, Optional) A function to get a new download URL in case of an error.\n - `etag`: (bool, Optional) Whether to validate etag before resuming downloads.\n\n ##### Returns\n \n - `AutoShutdownFuture`: If `block` is `False`.\n - `FileValidator`: If `block` is `True` and the download is successful.\n - `None`: If `block` is `True` and the download fails.\n\n- `stop()`: Stops the download process.\n\n### Helper Classes\n\n#### `Basicdown()`\n\nThe `Basicdown` class is the base downloader class that provides the basic structure for downloading files.\n\n##### Attributes\n\n- `curr`: The current size of the downloaded file in bytes.\n- `completed`: A flag that indicates if the download is complete.\n- `id`: The ID of the current instance.\n- `stop`: An event that can be used to stop the download process.\n- `error`: An event that can be used to signal an error.\n\n##### Methods\n\n- `download(url, path, mode, **kwargs)`: Downloads data in chunks.\n\n#### `Simpledown()`\n\nThe `Simpledown` class extends `Basicdown` and is responsible for downloading a whole file in a single segment.\n\n##### Parameters\n\n- `url`: Url of the file.\n- `file_path`: Path to save the file.\n- `stop`: Stop event.\n- `error`: Error event.\n- `**kwargs`: Additional keyword arguments.\n\n##### Methods\n\n- `worker()`: Downloads a whole file in a single segment.\n\n#### `Multidown()`\n\nThe `Multidown` class extends `Basicdown` and is responsible for downloading a specific segment of a file.\n\n##### Parameters\n\n- `segement_table`: Dictionary that contains the download information.\n- `segment_id`: ID of the download part.\n- `stop`: Stop event.\n- `error`: Error event.\n- `**kwargs`: Additional keyword arguments.\n\n##### Methods\n\n- `worker()`: Downloads a part of the file in multiple segments.\n\n#### `FileValidator()`\n\nThe `FileValidator` class is used to validate the integrity of the downloaded file.\n\n##### Parameters\n\n- `path`: The path of the file to be validated.\n\n##### Methods\n\n- `calculate_hash(algorithm, **kwargs)`: Calculates the hash of the file using the specified algorithm. Returns the calculated hash as a string.\n\n- `validate_hash(correct_hash, algorithm, **kwargs)`: Validates the hash of the file against the correct hash. Returns `True` if the hashes match, `False` otherwise.\n\n `calculate_hash` and `validate_hash` can support additional keyword arguments from the [hashlib module](https://docs.python.org/3/library/hashlib.html#hashlib.new).\n\n#### `AutoShutdownFuture()`\n\nThe `AutoShutdownFuture` class is a wrapper for concurrent.futures.Future object that shuts down the executor when the result is retrieved.\n\n##### Parameters\n\n- `future`: The Future object to be wrapped.\n- `executor`: The executor to be shut down when the result is retrieved.\n\n##### Methods\n\n- `result(timeout=None)`: Retrieves the result of the Future object and shuts down the executor. If the download was successful, it returns a `FileValidator` object; otherwise, it returns `None`.\n\n## License\n\npypdl is licensed under the MIT License. See the [LICENSE](https://github.com/mjishnu/pypdl/blob/main/LICENSE) file for more details.\n\n## Contribution\n\nContributions to pypdl are always welcome. If you want to contribute to this project, please fork the repository and submit a pull request.\n\n## Contact\n\nIf you have any questions, issues, or feedback about pypdl, please open an issue on the [GitHub repository](https://github.com/mjishnu/pypdl).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A concurrent python download manager",
"version": "1.3.2",
"project_urls": {
"Homepage": "https://github.com/mjishnu/pypdl"
},
"split_keywords": [
"python",
" downloader",
" concurrent-downloader",
" parrel-downloader",
" download manager",
" fast-downloader"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87eca66d67c06d4974a1b9b4b4fa21af23eebcde3259b321da0b5c9e76f4fd9c",
"md5": "968808412936be096931d909883b4a6f",
"sha256": "f496eebc5016244046c67c1902ad42982fc8976b60c827f0fe1265b048d4c130"
},
"downloads": -1,
"filename": "pypdl-1.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "968808412936be096931d909883b4a6f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11657,
"upload_time": "2024-04-06T07:53:11",
"upload_time_iso_8601": "2024-04-06T07:53:11.365043Z",
"url": "https://files.pythonhosted.org/packages/87/ec/a66d67c06d4974a1b9b4b4fa21af23eebcde3259b321da0b5c9e76f4fd9c/pypdl-1.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9dfc4fc3e0a961563051ed1ec65bba62f627fb2f6e0c845298b033aae9d89b6f",
"md5": "648a537f660af0ad7d18c73eea556961",
"sha256": "21142cdeb4a91fb85803ab4cd50ebb0a356044a97b0a8251dbeb2b56d414e21b"
},
"downloads": -1,
"filename": "pypdl-1.3.2.tar.gz",
"has_sig": false,
"md5_digest": "648a537f660af0ad7d18c73eea556961",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 13394,
"upload_time": "2024-04-06T07:53:12",
"upload_time_iso_8601": "2024-04-06T07:53:12.996855Z",
"url": "https://files.pythonhosted.org/packages/9d/fc/4fc3e0a961563051ed1ec65bba62f627fb2f6e0c845298b033aae9d89b6f/pypdl-1.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-06 07:53:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mjishnu",
"github_project": "pypdl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pypdl"
}
JSON
