Parfive
=======
.. image:: https://img.shields.io/pypi/v/parfive.svg
:target: https://pypi.python.org/pypi/parfive
:alt: Latest PyPI version
A parallel file downloader using asyncio. parfive can handle downloading
multiple files in parallel as well as downloading each file in a number of
chunks.
Usage
-----
.. image:: https://asciinema.org/a/EuALahgkiicWHGmrfFsZSLz81.svg
:alt: asciicast demo of parfive
:target: https://asciinema.org/a/EuALahgkiicWHGmrfFsZSLz81
parfive works by creating a downloader object, appending files to it and then
running the download. parfive has a synchronous API, but uses asyncio to
paralellise downloading the files.
A simple example is::
from parfive import Downloader
dl = Downloader()
dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./")
files = dl.download()
Parfive also bundles a CLI. The following example will download the two files concurrently.::
$ parfive 'http://212.183.159.230/5MB.zip' 'http://212.183.159.230/10MB.zip'
$ parfive --help
usage: parfive [-h] [--max-conn MAX_CONN] [--overwrite] [--no-file-progress]
[--directory DIRECTORY] [--print-filenames]
URLS [URLS ...]
Parfive, the python asyncio based downloader
positional arguments:
URLS URLs of files to be downloaded.
optional arguments:
-h, --help show this help message and exit
--max-conn MAX_CONN Number of maximum connections.
--overwrite Overwrite if the file exists.
--no-file-progress Show progress bar for each file.
--directory DIRECTORY
Directory to which downloaded files are saved.
--print-filenames Print successfully downloaded files's names to stdout.
Results
^^^^^^^
``parfive.Downloader.download`` returns a ``parfive.Results`` object, which is a
list of the filenames that have been downloaded. It also tracks any files which
failed to download.
Handling Errors
^^^^^^^^^^^^^^^
If files fail to download, the urls and the response from the server are stored
in the ``Results`` object returned by ``parfive.Downloader``. These can be used to
inform users about the errors. (Note, the progress bar will finish in an
incomplete state if a download fails, i.e. it will show ``4/5 Files Downloaded``).
The ``Results`` object is a list with an extra attribute ``errors``, this property
returns a list of named tuples, where these named tuples contains the ``.url``
and the ``.response``, which is a ``aiohttp.ClientResponse`` or a
``aiohttp.ClientError`` object.
Installation
------------
parfive is available on PyPI, you can install it with pip::
pip install parfive
or if you want to use FTP downloads::
pip install parfive[ftp]
Requirements
^^^^^^^^^^^^
- Python 3.7 or above
- aiohttp
- tqdm
- aioftp (for downloads over FTP)
Licence
-------
MIT Licensed
Authors
-------
`parfive` was written by `Stuart Mumford <http://stuartmumford.uk>`__.
Raw data
{
"_id": null,
"home_page": "https://parfive.readthedocs.io/",
"name": "parfive",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "\"Stuart Mumford\"",
"author_email": "\"stuart@cadair.com\"",
"download_url": "https://files.pythonhosted.org/packages/a1/91/4747fa3f02ec4a2e7c29881d6fef09643d0386f261fb2f01145860426959/parfive-2.0.2.tar.gz",
"platform": null,
"description": "Parfive\n=======\n\n.. image:: https://img.shields.io/pypi/v/parfive.svg\n :target: https://pypi.python.org/pypi/parfive\n :alt: Latest PyPI version\n\nA parallel file downloader using asyncio. parfive can handle downloading\nmultiple files in parallel as well as downloading each file in a number of\nchunks.\n\nUsage\n-----\n\n.. image:: https://asciinema.org/a/EuALahgkiicWHGmrfFsZSLz81.svg\n :alt: asciicast demo of parfive\n :target: https://asciinema.org/a/EuALahgkiicWHGmrfFsZSLz81\n\nparfive works by creating a downloader object, appending files to it and then\nrunning the download. parfive has a synchronous API, but uses asyncio to\nparalellise downloading the files.\n\nA simple example is::\n\n from parfive import Downloader\n dl = Downloader()\n dl.enqueue_file(\"http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt\", path=\"./\")\n files = dl.download()\n\nParfive also bundles a CLI. The following example will download the two files concurrently.::\n\n $ parfive 'http://212.183.159.230/5MB.zip' 'http://212.183.159.230/10MB.zip'\n $ parfive --help\n usage: parfive [-h] [--max-conn MAX_CONN] [--overwrite] [--no-file-progress]\n [--directory DIRECTORY] [--print-filenames]\n URLS [URLS ...]\n\n Parfive, the python asyncio based downloader\n\n positional arguments:\n URLS URLs of files to be downloaded.\n\n optional arguments:\n -h, --help show this help message and exit\n --max-conn MAX_CONN Number of maximum connections.\n --overwrite Overwrite if the file exists.\n --no-file-progress Show progress bar for each file.\n --directory DIRECTORY\n Directory to which downloaded files are saved.\n --print-filenames Print successfully downloaded files's names to stdout.\n\n\nResults\n^^^^^^^\n\n``parfive.Downloader.download`` returns a ``parfive.Results`` object, which is a\nlist of the filenames that have been downloaded. It also tracks any files which\nfailed to download.\n\nHandling Errors\n^^^^^^^^^^^^^^^\n\nIf files fail to download, the urls and the response from the server are stored\nin the ``Results`` object returned by ``parfive.Downloader``. These can be used to\ninform users about the errors. (Note, the progress bar will finish in an\nincomplete state if a download fails, i.e. it will show ``4/5 Files Downloaded``).\n\nThe ``Results`` object is a list with an extra attribute ``errors``, this property\nreturns a list of named tuples, where these named tuples contains the ``.url``\nand the ``.response``, which is a ``aiohttp.ClientResponse`` or a\n``aiohttp.ClientError`` object.\n\nInstallation\n------------\n\nparfive is available on PyPI, you can install it with pip::\n\n pip install parfive\n\nor if you want to use FTP downloads::\n\n pip install parfive[ftp]\n\nRequirements\n^^^^^^^^^^^^\n\n- Python 3.7 or above\n- aiohttp\n- tqdm\n- aioftp (for downloads over FTP)\n\nLicence\n-------\n\nMIT Licensed\n\nAuthors\n-------\n\n`parfive` was written by `Stuart Mumford <http://stuartmumford.uk>`__.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A HTTP and FTP parallel file downloader.",
"version": "2.0.2",
"project_urls": {
"Homepage": "https://parfive.readthedocs.io/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5c41cf4dad1a42119d65daf2aa3fe50a91e96a474f49e6f3435deb1f8b3900d1",
"md5": "b165b0ce033bb6ba1ff15fcfe319cf1c",
"sha256": "c6b6de349078edcfb88f76ba63f16983ba30ecb6c2105dff578c1cbe3bf15dd4"
},
"downloads": -1,
"filename": "parfive-2.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b165b0ce033bb6ba1ff15fcfe319cf1c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 26131,
"upload_time": "2022-10-27T13:53:14",
"upload_time_iso_8601": "2022-10-27T13:53:14.406077Z",
"url": "https://files.pythonhosted.org/packages/5c/41/cf4dad1a42119d65daf2aa3fe50a91e96a474f49e6f3435deb1f8b3900d1/parfive-2.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a1914747fa3f02ec4a2e7c29881d6fef09643d0386f261fb2f01145860426959",
"md5": "87a9048a02e1c1386d1c092fb9f646f8",
"sha256": "908211f9c5cb52dc8b279626872095f3cce15da86893f53b4176deceddcfc85d"
},
"downloads": -1,
"filename": "parfive-2.0.2.tar.gz",
"has_sig": false,
"md5_digest": "87a9048a02e1c1386d1c092fb9f646f8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 31686,
"upload_time": "2022-10-27T13:53:16",
"upload_time_iso_8601": "2022-10-27T13:53:16.854626Z",
"url": "https://files.pythonhosted.org/packages/a1/91/4747fa3f02ec4a2e7c29881d6fef09643d0386f261fb2f01145860426959/parfive-2.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-10-27 13:53:16",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "parfive"
}
JSON