bucket-dir


Namebucket-dir JSON
Version 3.0.1 PyPI version JSON
download
home_pagehttps://github.com/hmrc/bucket-dir
SummaryGenerate directory listings for S3 statically hosted content.
upload_time2021-04-16 07:50:18
maintainer
docs_urlNone
authorDave Randall
requires_python>=3.7,<4.0
licenseApache-2.0
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # bucket-dir

<a href="https://github.com/hmrc"><img alt="HMRC: Digital" src="https://img.shields.io/badge/HMRC-Digital-FFA500?style=flat&labelColor=000000&logo=gov.uk"></a>
<a href="https://pypi.org/project/bucket-dir/"><img alt="PyPI" src="https://img.shields.io/pypi/v/bucket-dir"></a>
<a href="https://pypi.org/project/bucket-dir/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/bucket-dir"></a>
<a href="https://github.com/hmrc/bucket-dir/blob/master/LICENSE"><img alt="License: Apache 2.0" src="https://img.shields.io/github/license/hmrc/bucket-dir"></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>

**bucket-dir** is a utility for generating a browsable directory tree for an AWS S3 bucket.

!["Sample image"](/docs/sample.png "A sample of bucket-dir output.")

It was built in order to host Maven and Ivy repositories in S3 and serve them via CloudFront, but it could meet other needs too.

## Installation

```
pip install bucket-dir
```
## Usage

Run `bucket-dir` with the name of the bucket you wish to index as a parameter:

```
bucket-dir foo-bucket
```

If you only want to upload indexes for a particular part of the bucket, use `--target-path`. This will generate indexes for folders that lead to the path, and everything under the path:

```
# These all update the root index, foo-folder's index, and everything underneath foo-folder
bucket-dir foo-bucket --target-path '/foo-folder/foo-object'
bucket-dir foo-bucket --target-path '/foo-folder/'
bucket-dir foo-bucket --target-path 'foo-folder/foo-object'
bucket-dir foo-bucket --target-path 'foo-folder/'
```

If you need to exclude objects with certain names from the index use `--exclude-object`. This will hide any objects that match this name. `index.html` objects are ignored for free:

```
bucket-dir foo-bucket --exclude-object 'error.html' --exclude-object 'foo-object'
```

Use `bucket-dir --help` for all arguments.

Be sure to provide the command with credentials that allow it to perform ListBucket and PutObject calls against the bucket. E.g. with [aws-vault](https://github.com/99designs/aws-vault):

```
aws-vault exec foo-profile -- bucket-dir foo-bucket
```
### IAM requirements

This example demonstrates the most restrictive policy you can apply to the principal (e.g. an IAM user or role) that is going to run `bucket-dir`. Substitute `foo-bucket` for the name of your bucket:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::foo-bucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::foo-bucket/index.html",
                "arn:aws:s3:::foo-bucket/*/index.html"
            ]
        }
    ]
}
```

* `s3:ListBucket` is required for `bucket-dir` to be able to map out the folders and objects that the bucket contains.
* `s3:PutObject` is required for `bucket-dir` to be able to upload generated `index.html` documents.
* `s3:DeleteObject` is required for `bucket-dir` to be able to remove redundant `index.html` documents.


### Example AWS configuration

For examples on how you can configure an S3 bucket to serve static site content indexed by `bucket-dir`, see:

* [Configuring a public S3 Bucket for use with bucket-dir.](docs/s3_public.md)

Examples of how you can front public and private buckets with CloudFront, and how bucket-dir can be run in a lambda, will be added in due course.

### Using bucket-dir as a library

`bucket-dir` can also be used as a dependency of your own python applications.

```
from bucket_dir import BucketDirGenerator

BucketDirGenerator(bucket_name="foo-bucket", site_name="my static site").generate()
```

### Character support

`bucket-dir` supports objects using any of the _Safe characters_ listed in the S3 [object key naming guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines).

The exception to the above rule is using forward slashes consecutively (e.g. `my-folder//my-object`). This results in a folder called `/`, which breaks hyperlinks.

Use of characters in the _Characters that might require special handling_ list is currently unsupported but is theoretically ok.

Some characters in _Characters to avoid_ may also work, but you're on your own.

## Development

Start with `make init`. This will install prerequisties and set up a poetry managed virtual environment containing all the required runtime and development dependencies.

Unit testing can be performed with `make test`. If you want to run pytest with other options, use `poetry run pytest ...`.

You can execute the source code directly with `poetry run bucket-dir`.

Finally, you can build with `make build`. This will update dependencies, run security checks and analysis and finally package the code into a wheel and archive.

Publishing can be performed with `make publish`, but this is only intended to run in CI on commit to the main branch. If running locally, you need to have PyPI credentials set as env vars.

For other rules, see the [Makefile](Makefile).

If you are a collaborator, feel free to make changes directly to the main branch. Otherwise, please raise a PR. Don't forget to bump the version in [pyproject.toml](pyproject.toml).

### Profiling

To get a performance profile, use:

```
make profile
```

You must have the `graphviz` library installed.

A `combined.svg` image will be generated in the `prof` directory which you can use to find bottlenecks and potential enhancements.

## License

This code is open source software licensed under the [Apache 2.0 License]("http://www.apache.org/licenses/LICENSE-2.0.html").

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/hmrc/bucket-dir",
    "name": "bucket-dir",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.7,<4.0",
    "maintainer_email": "",
    "keywords": "",
    "author": "Dave Randall",
    "author_email": "19395688+daveygit2050@users.noreply.github.com",
    "download_url": "https://files.pythonhosted.org/packages/18/6d/b594904a9e00ec08a4e70b927dea4677ec784ef35635e2ab29c70b6355fb/bucket-dir-3.0.1.tar.gz",
    "platform": "",
    "description": "# bucket-dir\n\n<a href=\"https://github.com/hmrc\"><img alt=\"HMRC: Digital\" src=\"https://img.shields.io/badge/HMRC-Digital-FFA500?style=flat&labelColor=000000&logo=gov.uk\"></a>\n<a href=\"https://pypi.org/project/bucket-dir/\"><img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/bucket-dir\"></a>\n<a href=\"https://pypi.org/project/bucket-dir/\"><img alt=\"Python\" src=\"https://img.shields.io/pypi/pyversions/bucket-dir\"></a>\n<a href=\"https://github.com/hmrc/bucket-dir/blob/master/LICENSE\"><img alt=\"License: Apache 2.0\" src=\"https://img.shields.io/github/license/hmrc/bucket-dir\"></a>\n<a href=\"https://github.com/psf/black\"><img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"></a>\n\n**bucket-dir** is a utility for generating a browsable directory tree for an AWS S3 bucket.\n\n![\"Sample image\"](/docs/sample.png \"A sample of bucket-dir output.\")\n\nIt was built in order to host Maven and Ivy repositories in S3 and serve them via CloudFront, but it could meet other needs too.\n\n## Installation\n\n```\npip install bucket-dir\n```\n## Usage\n\nRun `bucket-dir` with the name of the bucket you wish to index as a parameter:\n\n```\nbucket-dir foo-bucket\n```\n\nIf you only want to upload indexes for a particular part of the bucket, use `--target-path`. This will generate indexes for folders that lead to the path, and everything under the path:\n\n```\n# These all update the root index, foo-folder's index, and everything underneath foo-folder\nbucket-dir foo-bucket --target-path '/foo-folder/foo-object'\nbucket-dir foo-bucket --target-path '/foo-folder/'\nbucket-dir foo-bucket --target-path 'foo-folder/foo-object'\nbucket-dir foo-bucket --target-path 'foo-folder/'\n```\n\nIf you need to exclude objects with certain names from the index use `--exclude-object`. This will hide any objects that match this name. `index.html` objects are ignored for free:\n\n```\nbucket-dir foo-bucket --exclude-object 'error.html' --exclude-object 'foo-object'\n```\n\nUse `bucket-dir --help` for all arguments.\n\nBe sure to provide the command with credentials that allow it to perform ListBucket and PutObject calls against the bucket. E.g. with [aws-vault](https://github.com/99designs/aws-vault):\n\n```\naws-vault exec foo-profile -- bucket-dir foo-bucket\n```\n### IAM requirements\n\nThis example demonstrates the most restrictive policy you can apply to the principal (e.g. an IAM user or role) that is going to run `bucket-dir`. Substitute `foo-bucket` for the name of your bucket:\n\n```\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:ListBucket\",\n            \"Resource\": \"arn:aws:s3:::foo-bucket\"\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:PutObject\",\n                \"s3:DeleteObject\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::foo-bucket/index.html\",\n                \"arn:aws:s3:::foo-bucket/*/index.html\"\n            ]\n        }\n    ]\n}\n```\n\n* `s3:ListBucket` is required for `bucket-dir` to be able to map out the folders and objects that the bucket contains.\n* `s3:PutObject` is required for `bucket-dir` to be able to upload generated `index.html` documents.\n* `s3:DeleteObject` is required for `bucket-dir` to be able to remove redundant `index.html` documents.\n\n\n### Example AWS configuration\n\nFor examples on how you can configure an S3 bucket to serve static site content indexed by `bucket-dir`, see:\n\n* [Configuring a public S3 Bucket for use with bucket-dir.](docs/s3_public.md)\n\nExamples of how you can front public and private buckets with CloudFront, and how bucket-dir can be run in a lambda, will be added in due course.\n\n### Using bucket-dir as a library\n\n`bucket-dir` can also be used as a dependency of your own python applications.\n\n```\nfrom bucket_dir import BucketDirGenerator\n\nBucketDirGenerator(bucket_name=\"foo-bucket\", site_name=\"my static site\").generate()\n```\n\n### Character support\n\n`bucket-dir` supports objects using any of the _Safe characters_ listed in the S3 [object key naming guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines).\n\nThe exception to the above rule is using forward slashes consecutively (e.g. `my-folder//my-object`). This results in a folder called `/`, which breaks hyperlinks.\n\nUse of characters in the _Characters that might require special handling_ list is currently unsupported but is theoretically ok.\n\nSome characters in _Characters to avoid_ may also work, but you're on your own.\n\n## Development\n\nStart with `make init`. This will install prerequisties and set up a poetry managed virtual environment containing all the required runtime and development dependencies.\n\nUnit testing can be performed with `make test`. If you want to run pytest with other options, use `poetry run pytest ...`.\n\nYou can execute the source code directly with `poetry run bucket-dir`.\n\nFinally, you can build with `make build`. This will update dependencies, run security checks and analysis and finally package the code into a wheel and archive.\n\nPublishing can be performed with `make publish`, but this is only intended to run in CI on commit to the main branch. If running locally, you need to have PyPI credentials set as env vars.\n\nFor other rules, see the [Makefile](Makefile).\n\nIf you are a collaborator, feel free to make changes directly to the main branch. Otherwise, please raise a PR. Don't forget to bump the version in [pyproject.toml](pyproject.toml).\n\n### Profiling\n\nTo get a performance profile, use:\n\n```\nmake profile\n```\n\nYou must have the `graphviz` library installed.\n\nA `combined.svg` image will be generated in the `prof` directory which you can use to find bottlenecks and potential enhancements.\n\n## License\n\nThis code is open source software licensed under the [Apache 2.0 License](\"http://www.apache.org/licenses/LICENSE-2.0.html\").\n",
    "bugtrack_url": null,
    "license": "Apache-2.0",
    "summary": "Generate directory listings for S3 statically hosted content.",
    "version": "3.0.1",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "984abaf71859b164590c1ad11111b2e0",
                "sha256": "ec2a4314cfdaa4e392f8157650bab9564661ad4b6d7c0813a25ced8d4d1c8bb3"
            },
            "downloads": -1,
            "filename": "bucket_dir-3.0.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "984abaf71859b164590c1ad11111b2e0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.7,<4.0",
            "size": 13299,
            "upload_time": "2021-04-16T07:50:19",
            "upload_time_iso_8601": "2021-04-16T07:50:19.877866Z",
            "url": "https://files.pythonhosted.org/packages/5e/58/58025b935230549d56c9b30c7212907410a079815ebe12043e9b3fca7aff/bucket_dir-3.0.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "8e0aae150422872836cfbd254c608ff2",
                "sha256": "d019b80d20dd8351b6a5edb6c392a46e217173474e85c9559e612e893e4a60c1"
            },
            "downloads": -1,
            "filename": "bucket-dir-3.0.1.tar.gz",
            "has_sig": false,
            "md5_digest": "8e0aae150422872836cfbd254c608ff2",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.7,<4.0",
            "size": 13986,
            "upload_time": "2021-04-16T07:50:18",
            "upload_time_iso_8601": "2021-04-16T07:50:18.924251Z",
            "url": "https://files.pythonhosted.org/packages/18/6d/b594904a9e00ec08a4e70b927dea4677ec784ef35635e2ab29c70b6355fb/bucket-dir-3.0.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-04-16 07:50:18",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "hmrc",
    "error": "Could not fetch GitHub repository",
    "lcname": "bucket-dir"
}
        
Elapsed time: 0.27690s