# s3manifestcollectstatic
Optimized collectstatic for S3ManifestStaticStorage
## Installation
1. Install the package
```
pip install s3manifestcollectstatic
```
2. Add `s3manifestcollectstatic` to `INSTALLED_APPS`
## Description
`collectstatic` can take a long time. When used with
`storages.backends.s3boto3.S3ManifestStaticStorage`, `collectstatic` uploads
the files twice, once without the hash at the end of the file name, and once
with the hash. Also, it doesn't use multiple threads to upload to s3.
`s3manifestcollectstatic` uploads the files only once, uses threads to speed
things up, and doesn't upload the files that are already on S3.
collectstatic: (Around 20 minutes)
```
$ time ./manage.py collectstatic --noinput
604 static files copied, 646 post-processed.
./manage.py collectstatic --noinput 29,94s user 2,27s system 2% cpu 20:25,06 total
```
s3manifestcollectstatic: (Around 30 seconds)
```
$ time ./manage.py s3manifestcollectstatic
604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.
Start the upload of 604 files
Uploading the manifest
./manage.py s3manifestcollectstatic 10,95s user 1,92s system 49% cpu 26,269 total
```
If you want to reupload the files use `-f`:
```
./manage.py s3manifestcollectstatic -f
```
You can limit the maximum number of workers using `-w`:
```
./manage.py s3manifestcollectstatic -w 4
```
Tested with:
- Python 3.9, Django 3.2, django-storages 1.11
- Python 3.11, Django 4.2, django-storages 1.13
Raw data
{
"_id": null,
"home_page": "https://github.com/dduong42/s3manifestcollectstatic",
"name": "s3manifestcollectstatic",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6.1,<4.0.0",
"maintainer_email": "",
"keywords": "django,collectstatic,S3,manifest",
"author": "Daniel D\u01b0\u01a1ng",
"author_email": "daniel.duong@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/bb/c5/8291f5c1fd7fce0a7f9de0ec92663de1af58b580df69937e4e8021462010/s3manifestcollectstatic-0.3.2.tar.gz",
"platform": null,
"description": "# s3manifestcollectstatic\n\nOptimized collectstatic for S3ManifestStaticStorage\n\n## Installation\n\n1. Install the package\n```\npip install s3manifestcollectstatic\n```\n2. Add `s3manifestcollectstatic` to `INSTALLED_APPS`\n\n## Description\n\n`collectstatic` can take a long time. When used with\n`storages.backends.s3boto3.S3ManifestStaticStorage`, `collectstatic` uploads\nthe files twice, once without the hash at the end of the file name, and once\nwith the hash. Also, it doesn't use multiple threads to upload to s3.\n\n`s3manifestcollectstatic` uploads the files only once, uses threads to speed\nthings up, and doesn't upload the files that are already on S3.\n\ncollectstatic: (Around 20 minutes)\n\n```\n$ time ./manage.py collectstatic --noinput\n\n604 static files copied, 646 post-processed.\n./manage.py collectstatic --noinput 29,94s user 2,27s system 2% cpu 20:25,06 total\n```\n\ns3manifestcollectstatic: (Around 30 seconds)\n```\n$ time ./manage.py s3manifestcollectstatic\n\n604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.\nStart the upload of 604 files\nUploading the manifest\n./manage.py s3manifestcollectstatic 10,95s user 1,92s system 49% cpu 26,269 total\n```\n\nIf you want to reupload the files use `-f`:\n```\n./manage.py s3manifestcollectstatic -f\n```\n\nYou can limit the maximum number of workers using `-w`:\n```\n./manage.py s3manifestcollectstatic -w 4\n```\n\nTested with:\n- Python 3.9, Django 3.2, django-storages 1.11\n- Python 3.11, Django 4.2, django-storages 1.13\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Optimized collectstatic for S3ManifestStaticStorage",
"version": "0.3.2",
"project_urls": {
"Homepage": "https://github.com/dduong42/s3manifestcollectstatic",
"Repository": "https://github.com/dduong42/s3manifestcollectstatic"
},
"split_keywords": [
"django",
"collectstatic",
"s3",
"manifest"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4e81c09d2c97191f699e50d5970a7adc6509d704bf0add095e86c5947f5cdaed",
"md5": "e74f113edecc99468eac215eebfe4d58",
"sha256": "d3850b5f226eddc949daa38a07fcede657e521afa2192e67fedfab08ab5a846e"
},
"downloads": -1,
"filename": "s3manifestcollectstatic-0.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e74f113edecc99468eac215eebfe4d58",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6.1,<4.0.0",
"size": 4732,
"upload_time": "2023-08-21T01:35:23",
"upload_time_iso_8601": "2023-08-21T01:35:23.971135Z",
"url": "https://files.pythonhosted.org/packages/4e/81/c09d2c97191f699e50d5970a7adc6509d704bf0add095e86c5947f5cdaed/s3manifestcollectstatic-0.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bbc58291f5c1fd7fce0a7f9de0ec92663de1af58b580df69937e4e8021462010",
"md5": "51f4e4c02f38ef00a38993ea54c750d0",
"sha256": "c742b07e36261e898db22bf031925053d7b0bfa17084783190e7fb6408cea59a"
},
"downloads": -1,
"filename": "s3manifestcollectstatic-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "51f4e4c02f38ef00a38993ea54c750d0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6.1,<4.0.0",
"size": 3601,
"upload_time": "2023-08-21T01:35:25",
"upload_time_iso_8601": "2023-08-21T01:35:25.128228Z",
"url": "https://files.pythonhosted.org/packages/bb/c5/8291f5c1fd7fce0a7f9de0ec92663de1af58b580df69937e4e8021462010/s3manifestcollectstatic-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-21 01:35:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dduong42",
"github_project": "s3manifestcollectstatic",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "s3manifestcollectstatic"
}
JSON
