# picsellia-s3manifestcollectstatic
This is a fork of https://github.com/dduong42/s3manifestcollectstatic
Optimized collectstatic for S3ManifestStaticStorage.
With max_workers parameters and custom handler of exception while uploading files
## Installation
1. Install the package
```
pip install picsellia-s3manifestcollectstatic
```
2. Add `picsellias3manifestcollectstatic` to `INSTALLED_APPS`
## Description
`collectstatic` can take a long time. When used with
`storages.backends.s3boto3.S3ManifestStaticStorage`, `collectstatic` uploads
the files twice, once without the hash at the end of the file name, and once
with the hash. Also, it doesn't use multiple threads to upload to s3.
`s3manifestcollectstatic` uploads the files only once, uses threads to speed
things up, and doesn't upload the files that are already on S3.
collectstatic: (Around 20 minutes)
```
$ time ./manage.py collectstatic --noinput
604 static files copied, 646 post-processed.
./manage.py collectstatic --noinput 29,94s user 2,27s system 2% cpu 20:25,06 total
```
s3manifestcollectstatic: (Around 30 seconds)
```
$ time ./manage.py s3manifestcollectstatic
604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.
Start the upload of 604 files
Uploading the manifest
./manage.py s3manifestcollectstatic 10,95s user 1,92s system 49% cpu 26,269 total
```
If you want to reupload the files use `-f`:
```
./manage.py s3manifestcollectstatic -f
```
Tested with Python 3.9, Django 3.2, django-storages 1.11
Raw data
{
"_id": null,
"home_page": "https://github.com/picselliahq/s3manifestcollectstatic",
"name": "picsellia-collectstatic",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "django,collectstatic,S3,manifest",
"author": "Thomas Darget",
"author_email": "thomas.darget@picsellia.com",
"download_url": "https://files.pythonhosted.org/packages/fc/93/1e3f8b4149c64d40d5365ac290cfb2d65766794ecd641a2d3fe4f5123ce9/picsellia_collectstatic-0.0.4.tar.gz",
"platform": null,
"description": "# picsellia-s3manifestcollectstatic\nThis is a fork of https://github.com/dduong42/s3manifestcollectstatic\n\nOptimized collectstatic for S3ManifestStaticStorage.\nWith max_workers parameters and custom handler of exception while uploading files\n\n## Installation\n\n1. Install the package\n```\npip install picsellia-s3manifestcollectstatic\n```\n2. Add `picsellias3manifestcollectstatic` to `INSTALLED_APPS`\n\n## Description\n\n`collectstatic` can take a long time. When used with\n`storages.backends.s3boto3.S3ManifestStaticStorage`, `collectstatic` uploads\nthe files twice, once without the hash at the end of the file name, and once\nwith the hash. Also, it doesn't use multiple threads to upload to s3.\n\n`s3manifestcollectstatic` uploads the files only once, uses threads to speed\nthings up, and doesn't upload the files that are already on S3.\n\ncollectstatic: (Around 20 minutes)\n\n```\n$ time ./manage.py collectstatic --noinput\n\n604 static files copied, 646 post-processed.\n./manage.py collectstatic --noinput 29,94s user 2,27s system 2% cpu 20:25,06 total\n```\n\ns3manifestcollectstatic: (Around 30 seconds)\n```\n$ time ./manage.py s3manifestcollectstatic\n\n604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.\nStart the upload of 604 files\nUploading the manifest\n./manage.py s3manifestcollectstatic 10,95s user 1,92s system 49% cpu 26,269 total\n```\n\nIf you want to reupload the files use `-f`:\n```\n./manage.py s3manifestcollectstatic -f\n```\n\nTested with Python 3.9, Django 3.2, django-storages 1.11\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Optimized collectstatic for S3ManifestStaticStorage",
"version": "0.0.4",
"project_urls": {
"Homepage": "https://github.com/picselliahq/s3manifestcollectstatic",
"Repository": "https://github.com/picselliahq/s3manifestcollectstatic"
},
"split_keywords": [
"django",
"collectstatic",
"s3",
"manifest"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e4afb3c3d4392fbee68df31f9376dc049a72c7c88a8d6e2a3ba9f186a5c28436",
"md5": "4a76f9d9c7bc66e005e284722281d819",
"sha256": "04ed43ae060c25e45ab8d33dd5470d1dee8400afb0358d30cace4992eaf7b7b0"
},
"downloads": -1,
"filename": "picsellia_collectstatic-0.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4a76f9d9c7bc66e005e284722281d819",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 5500,
"upload_time": "2023-09-26T09:28:33",
"upload_time_iso_8601": "2023-09-26T09:28:33.053740Z",
"url": "https://files.pythonhosted.org/packages/e4/af/b3c3d4392fbee68df31f9376dc049a72c7c88a8d6e2a3ba9f186a5c28436/picsellia_collectstatic-0.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fc931e3f8b4149c64d40d5365ac290cfb2d65766794ecd641a2d3fe4f5123ce9",
"md5": "1b38d80036e61cfa2ff730d5735627ba",
"sha256": "efd6a8e496d12b500adf0a8b6693ef4123b6a0e77cec6aa05436f501c83c1390"
},
"downloads": -1,
"filename": "picsellia_collectstatic-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "1b38d80036e61cfa2ff730d5735627ba",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 4284,
"upload_time": "2023-09-26T09:28:34",
"upload_time_iso_8601": "2023-09-26T09:28:34.498226Z",
"url": "https://files.pythonhosted.org/packages/fc/93/1e3f8b4149c64d40d5365ac290cfb2d65766794ecd641a2d3fe4f5123ce9/picsellia_collectstatic-0.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-26 09:28:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "picselliahq",
"github_project": "s3manifestcollectstatic",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "picsellia-collectstatic"
}
JSON
