# Django Async Downloads
Asynchronous downloads scaffolding for Django projects.
- TODO: describe cache mechanism
- TODO: how to customise the front end
## Installation
```
# Without websocket support
pip install django-async-downloads
# With Websocket support
pip install django-async-downloads[ws]
```
Note at the present time on Windows, if you require websocket support,
you will need "Microsoft Visual C++ 14.0" installed (https://visualstudio.microsoft.com/visual-cpp-build-tools/)
due to `hiredis` from the dependency chain (`channels_redis` -> `aioredis-py` -> `hiredis`).
Add to your `INSTALLED_APPS`:
```
INSTALLED_APPS = [
...
"async_downloads",
]
```
Add to your project's `url_patterns`:
```
path("async_downloads/", include("async_downloads.urls"))
```
Add the CSS:
```
<link rel="stylesheet" type="text/css" href="{% static 'css/async_downloads.css' %}" />
```
Add the JS:
```
<script src="{% static "js/async_downloads.js" %}" id="async-downloads-script"
data-url="{% url 'async_downloads:ajax_update' %}"
data-clear-url="{% url 'async_downloads:ajax_clear_download' %}"></script>
```
Include the download centre nav-menu:
```
<ul class="navbar-nav">
...
{% include 'async_downloads/download_centre.html' %}
...
```
## WS mode
Package can be used with websockets to provide asynchronous communication
between frontend and backend.
If package will be used in websockets mode additional settings must be applied.
* Inside common settings `WS_MODE` must be toggled on.
```
ASYNC_DOWNLOADS_WS_MODE = True
```
* Application needs to be configured as a `ASGI` and proper WS urls need to be configured:
```
# ws/urls.py
from async_downloads.ws_consumers import DownloadsConsumer
urlpatterns += [
re_path(r"ws/downloads/(?P<username>[\w.]+)/$", DownloadsConsumer.as_asgi()),
]
```
```
# example asgi.py
from django.core.asgi import get_asgi_application
from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
import ws.urls
application = ProtocolTypeRouter(
{
"http": get_asgi_application(),
"websocket": AuthMiddlewareStack(URLRouter(ws.urls.urlpatterns)),
}
)
```
* Change in application template async-downloads section to use WS js version.
```
# example: FULL_WS_URL = "http://app.com/ws/downloads/"
<script src="{% static "js/ws_async_downloads.js" %}" id="async-downloads-script"
data-url="{{ FULL_WS_URL }}"
</script>
```
- `data-url` must be an absolute URL because this is required for a WebSocket connection,
and it must include the protocol because the `ws_async_downloads.js` script will
inspect it to determine which WebSockets protocol to use - `ws` if `http` or `wss`
if `https`.
* Configure `CHANNEL_LAYERS` inside common settings. Example config:
```
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("127.0.0.1", 6379)],
},
},
}
```
## Usage
TODO: using the JS
### Cache Functions
#### `cache.init_download`
Initialise a download by preparing the cache entries. Returns a tuple of keys, the collection key
and specific download key. You would typically want to call this within the web process, and pass
the download key (and possibly the collection key) into the asynchronous function call so that the
status can be updated.
Arguments:
- `user`: the unique identifier for a collection of downloads - this will typically be a user
object but can be user PK or username.
- `filename`: the name of the file being downloaded (does not need to be unique)
- `name`: (optional) the name to associate with this download - defaults to `filename`
#### `cache.save_download`
The asynchronous process should call this function when the iterable or file is prepared in order to save
the output.
Arguments:
- `download_key`: the cache key of this particular download
- `iterable`: (optional) an iterable of data rows to be written
- `file`: (optional) a BytesIO object
#### `cache.update_percentage`
The asynchronous process can call this function to calculate and update the completion percentage.
(download_key, total, cur, resolution=10):
Arguments:
- `download_key`: the cache key of this particular download
- `total`: total to compare current progress against
- `cur`: current progress index
- `resolution`: resolution of the percentage calculation (make smaller for fewer updates, larger for
more precision); default is `10`, meaning it will increase in steps of 10%. The value is capped between
`1` and `100`.
#### `cache.set_percentage`
The asynchronous process can call this function to directly set the completion percentage.
Arguments:
- `download_key`: the cache key of this particular download
- `percentage`: an number between 0 and 100 (inclusive)
#### `cache.cleanup_collection`
There can be a build up of expired download keys in a collection so it can be worth periodically
removing them using this function. (Note that the collection cache itself will expire if not touched
for `ASYNC_DOWNLOADS_TIMEOUT` seconds, so it may not be critical to use this.)
Arguments:
- `collection_key`: the cache key of a collection of downloads
### `cache.set_error`
The asynchronous process can call this function to set an error message on the download.
Arguments:
- `download_key`: the cache key of this particular download
- `error`: the error message to set
#### `tasks.cleanup_expired_downloads`
Delete expired downloads (where the download no longer exists in the cache).
This is a clean up operation to prevent downloads that weren't manually deleted from building up,
and should be run periodically to avoid bloating the server with files.
This is best setup as a periodic task in your project, which can be done by adding the following
to your project's `celery.py`:
```
app.conf.beat_schedule = {
"async_downloads_cleanup": {
"task": "async_downloads.tasks.cleanup_expired_downloads",
"schedule": crontab(hour=0, minute=0, day_of_week=1)
}
}
```
## Configurable Settings
### `ASYNC_DOWNLOADS_TIMEOUT`
Default: `60 * 60 * 24` (1 day)
This is the cache timeout used for cache operations.
### `ASYNC_DOWNLOADS_DOWNLOAD_TEMPLATE`
Default: `"async_downloads/downloads.html"`
This is the template that will be used by the `ajax_update` view. You can override it by creating
a template in `<project>/templates/async_downloads/downloads.html`, or else putting the template
wherever you choose and changing this setting.
### `ASYNC_DOWNLOADS_PATH_PREFIX`
Default: `"downloads"`
The parent directory for all downloads in the `MEDIA_ROOT` directory.
### `ASYNC_DOWNLOADS_COLLECTION_KEY_FORMAT`
Default: `"async_downloads/{}"`
The collection key keeps track of the cache keys of a grouped collection of downloads. In the
unlikely event that this key format clashes with something in your project, you can change it.
The expectation is for the string to have a user primary key inserted with `str.format`, so `{}`
is required to be present.
### `ASYNC_DOWNLOADS_WS_CHANNEL_NAME`
Default: `"downloads"`
The channel name for all shared information about download in channels cache layer.
### `ASYNC_DOWNLOADS_CACHE_NAME`
If this settings will be added default cache will be changed into new with provided name.
### `ASYNC_DOWNLOADS_WS_MODE`
Default: `False`
If this flag will be set to `True` package will be set to work with WebSockets in [WS_MODE](#ws-mode).
### `ASYNC_DOWNLOADS_CSS_CLASS`
Default: `None`
Additional CSS class can be added to `download-content` div.
Raw data
{
"_id": null,
"home_page": "https://github.com/QuickRelease/django-async-downloads.git",
"name": "django-async-downloads",
"maintainer": "Quick Release (Automotive) Ltd.",
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "django download asynchronous async celery",
"author": "David Vaughan",
"author_email": "david.vaughan@quickrelease.co.uk",
"download_url": "https://files.pythonhosted.org/packages/68/84/a36e95b4e45bda60c2ebd797a78f6e4b8812086d7bb8cfd8ca46ed265abc/django-async-downloads-0.5.2.tar.gz",
"platform": null,
"description": "# Django Async Downloads\n\nAsynchronous downloads scaffolding for Django projects.\n\n- TODO: describe cache mechanism\n- TODO: how to customise the front end\n\n## Installation\n\n```\n# Without websocket support\npip install django-async-downloads\n\n# With Websocket support\npip install django-async-downloads[ws]\n```\n\nNote at the present time on Windows, if you require websocket support,\nyou will need \"Microsoft Visual C++ 14.0\" installed (https://visualstudio.microsoft.com/visual-cpp-build-tools/) \ndue to `hiredis` from the dependency chain (`channels_redis` -> `aioredis-py` -> `hiredis`).\n\nAdd to your `INSTALLED_APPS`:\n```\nINSTALLED_APPS = [\n ...\n \"async_downloads\",\n]\n```\n\nAdd to your project's `url_patterns`:\n```\npath(\"async_downloads/\", include(\"async_downloads.urls\"))\n```\n\nAdd the CSS:\n```\n<link rel=\"stylesheet\" type=\"text/css\" href=\"{% static 'css/async_downloads.css' %}\" />\n```\n\nAdd the JS:\n```\n<script src=\"{% static \"js/async_downloads.js\" %}\" id=\"async-downloads-script\"\n data-url=\"{% url 'async_downloads:ajax_update' %}\"\n data-clear-url=\"{% url 'async_downloads:ajax_clear_download' %}\"></script>\n```\n\nInclude the download centre nav-menu:\n```\n<ul class=\"navbar-nav\">\n ...\n {% include 'async_downloads/download_centre.html' %}\n ...\n```\n\n## WS mode\nPackage can be used with websockets to provide asynchronous communication\nbetween frontend and backend.\nIf package will be used in websockets mode additional settings must be applied.\n* Inside common settings `WS_MODE` must be toggled on.\n ```\n ASYNC_DOWNLOADS_WS_MODE = True\n ```\n* Application needs to be configured as a `ASGI` and proper WS urls need to be configured:\n ```\n # ws/urls.py\n from async_downloads.ws_consumers import DownloadsConsumer\n \n urlpatterns += [\n re_path(r\"ws/downloads/(?P<username>[\\w.]+)/$\", DownloadsConsumer.as_asgi()),\n ]\n ```\n ```\n # example asgi.py\n from django.core.asgi import get_asgi_application\n\n from channels.auth import AuthMiddlewareStack\n from channels.routing import ProtocolTypeRouter, URLRouter\n\n import ws.urls\n\n application = ProtocolTypeRouter(\n {\n \"http\": get_asgi_application(),\n \"websocket\": AuthMiddlewareStack(URLRouter(ws.urls.urlpatterns)),\n }\n )\n ```\n\n* Change in application template async-downloads section to use WS js version.\n ```\n # example: FULL_WS_URL = \"http://app.com/ws/downloads/\"\n <script src=\"{% static \"js/ws_async_downloads.js\" %}\" id=\"async-downloads-script\"\n data-url=\"{{ FULL_WS_URL }}\"\n </script>\n ```\n - `data-url` must be an absolute URL because this is required for a WebSocket connection,\n and it must include the protocol because the `ws_async_downloads.js` script will \n inspect it to determine which WebSockets protocol to use - `ws` if `http` or `wss` \n if `https`.\n\n* Configure `CHANNEL_LAYERS` inside common settings. Example config:\n ```\n CHANNEL_LAYERS = {\n \"default\": {\n \"BACKEND\": \"channels_redis.core.RedisChannelLayer\",\n \"CONFIG\": {\n \"hosts\": [(\"127.0.0.1\", 6379)],\n },\n },\n }\n ```\n\n## Usage\n\nTODO: using the JS\n\n### Cache Functions\n\n#### `cache.init_download`\nInitialise a download by preparing the cache entries. Returns a tuple of keys, the collection key\nand specific download key. You would typically want to call this within the web process, and pass\nthe download key (and possibly the collection key) into the asynchronous function call so that the\nstatus can be updated.\n\nArguments:\n- `user`: the unique identifier for a collection of downloads - this will typically be a user\n object but can be user PK or username.\n- `filename`: the name of the file being downloaded (does not need to be unique)\n- `name`: (optional) the name to associate with this download - defaults to `filename`\n\n\n#### `cache.save_download`\nThe asynchronous process should call this function when the iterable or file is prepared in order to save\nthe output.\n\nArguments:\n- `download_key`: the cache key of this particular download\n- `iterable`: (optional) an iterable of data rows to be written\n- `file`: (optional) a BytesIO object\n\n\n#### `cache.update_percentage`\nThe asynchronous process can call this function to calculate and update the completion percentage.\n(download_key, total, cur, resolution=10):\nArguments:\n- `download_key`: the cache key of this particular download\n- `total`: total to compare current progress against\n- `cur`: current progress index\n- `resolution`: resolution of the percentage calculation (make smaller for fewer updates, larger for\nmore precision); default is `10`, meaning it will increase in steps of 10%. The value is capped between\n`1` and `100`.\n\n\n#### `cache.set_percentage`\nThe asynchronous process can call this function to directly set the completion percentage.\n\nArguments:\n- `download_key`: the cache key of this particular download\n- `percentage`: an number between 0 and 100 (inclusive)\n\n\n#### `cache.cleanup_collection`\nThere can be a build up of expired download keys in a collection so it can be worth periodically\nremoving them using this function. (Note that the collection cache itself will expire if not touched\nfor `ASYNC_DOWNLOADS_TIMEOUT` seconds, so it may not be critical to use this.)\n\nArguments:\n- `collection_key`: the cache key of a collection of downloads\n\n\n### `cache.set_error`\nThe asynchronous process can call this function to set an error message on the download.\n\nArguments:\n- `download_key`: the cache key of this particular download\n- `error`: the error message to set\n\n#### `tasks.cleanup_expired_downloads`\nDelete expired downloads (where the download no longer exists in the cache).\nThis is a clean up operation to prevent downloads that weren't manually deleted from building up,\nand should be run periodically to avoid bloating the server with files.\n\nThis is best setup as a periodic task in your project, which can be done by adding the following\nto your project's `celery.py`:\n```\napp.conf.beat_schedule = {\n \"async_downloads_cleanup\": {\n \"task\": \"async_downloads.tasks.cleanup_expired_downloads\",\n \"schedule\": crontab(hour=0, minute=0, day_of_week=1)\n }\n}\n```\n\n## Configurable Settings\n\n### `ASYNC_DOWNLOADS_TIMEOUT`\nDefault: `60 * 60 * 24` (1 day)\n\nThis is the cache timeout used for cache operations. \n\n### `ASYNC_DOWNLOADS_DOWNLOAD_TEMPLATE`\nDefault: `\"async_downloads/downloads.html\"`\n\nThis is the template that will be used by the `ajax_update` view. You can override it by creating\na template in `<project>/templates/async_downloads/downloads.html`, or else putting the template\nwherever you choose and changing this setting.\n\n### `ASYNC_DOWNLOADS_PATH_PREFIX`\nDefault: `\"downloads\"`\n\nThe parent directory for all downloads in the `MEDIA_ROOT` directory.\n\n### `ASYNC_DOWNLOADS_COLLECTION_KEY_FORMAT`\nDefault: `\"async_downloads/{}\"`\n\nThe collection key keeps track of the cache keys of a grouped collection of downloads. In the\nunlikely event that this key format clashes with something in your project, you can change it.\nThe expectation is for the string to have a user primary key inserted with `str.format`, so `{}`\nis required to be present.\n\n### `ASYNC_DOWNLOADS_WS_CHANNEL_NAME`\nDefault: `\"downloads\"`\n\nThe channel name for all shared information about download in channels cache layer.\n\n### `ASYNC_DOWNLOADS_CACHE_NAME`\n\nIf this settings will be added default cache will be changed into new with provided name.\n\n### `ASYNC_DOWNLOADS_WS_MODE`\nDefault: `False`\n\nIf this flag will be set to `True` package will be set to work with WebSockets in [WS_MODE](#ws-mode).\n\n### `ASYNC_DOWNLOADS_CSS_CLASS`\nDefault: `None`\n\nAdditional CSS class can be added to `download-content` div.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Asynchronous downloads scaffolding for Django projects",
"version": "0.5.2",
"project_urls": {
"Homepage": "https://github.com/QuickRelease/django-async-downloads.git"
},
"split_keywords": [
"django",
"download",
"asynchronous",
"async",
"celery"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c7e1840a8799ca3e8ab6216fcbbb36ae412778fccba995fe8720efb9300103b2",
"md5": "106368070d0250af420519d437ee8206",
"sha256": "e0920278baf835c1b32e1b5fc35493eaf607de595a7a9d07ea110c7bf5737549"
},
"downloads": -1,
"filename": "django_async_downloads-0.5.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "106368070d0250af420519d437ee8206",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 16974,
"upload_time": "2024-03-20T15:46:55",
"upload_time_iso_8601": "2024-03-20T15:46:55.515649Z",
"url": "https://files.pythonhosted.org/packages/c7/e1/840a8799ca3e8ab6216fcbbb36ae412778fccba995fe8720efb9300103b2/django_async_downloads-0.5.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6884a36e95b4e45bda60c2ebd797a78f6e4b8812086d7bb8cfd8ca46ed265abc",
"md5": "0f8500efb76db360e0f035a94f96fc04",
"sha256": "2188bc3852f8c5de40850523c7738aa9463c44b75a131247086dbbb58216db88"
},
"downloads": -1,
"filename": "django-async-downloads-0.5.2.tar.gz",
"has_sig": false,
"md5_digest": "0f8500efb76db360e0f035a94f96fc04",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 16541,
"upload_time": "2024-03-20T15:46:56",
"upload_time_iso_8601": "2024-03-20T15:46:56.726373Z",
"url": "https://files.pythonhosted.org/packages/68/84/a36e95b4e45bda60c2ebd797a78f6e4b8812086d7bb8cfd8ca46ed265abc/django-async-downloads-0.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-20 15:46:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "QuickRelease",
"github_project": "django-async-downloads",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "django-async-downloads"
}
JSON
