============
AWS Utils S3
============
Tools to list and fetch objects from versioned AWS_ S3_ bucket:
* `s3lsvers`: List object versions, see versioning_
* `s3getvers`: Fetch specified object versions
* `s3tmpgen`: Generate temporary url links to objects
.. contents:: Table of Contents
Installation
============
Using `pip`::
$ pip install ttr.aws.utils.s3
Using `pipx`::
$ pipx install ttr.aws.utils.s3
Quick start
===========
We want to fetch versions of feed in bucket `mybucket` named `my/versioned/feed.xml`
1. Configure AWSCLI credentials to allow access to your buckets and objects. E.g. using `AWS_DEFAULT_PROFILE`. See AWS_config_.
2. create csv file for given feed and time period::
$ s3lsvers -from 2012-05-24T00:15 -to 2012-05-24T01:15 -list-file list.csv mybucket/my/versioned/feed.xml
You shall then find file `list.csv` on your disk.
3. Review records in `list.csv` and delete all lines with version, which are not of your interest.
4. Using `list.csv`, ask s3getvers to fetch all versions specified in the file. Be sure to run it on empty directory::
$ s3getvers mybucket list.csv
You will see, how is each version downloaded and saved to your current directory.
5. Finally, you can try generating temorary url to your feed (showing the latest existing)::
$ s3tmpgen 2014-09-30T00:00:00Z mybucket my/versioned/feed.xml
https://mybucket.s3.amazonaws.com/my/versioned/feed.xml?Signature=kOCwz%2FkanVWX8O15dlXhy4jrbwY%3D&Expires=1412031600&AWSAccessKeyId=AKIAxyzxyzxyzEQA
Note, that the url does not include VersionId, so it will always point to the most up todate version (in case the key happens to be on versioned bucket).
Provided commands
=================
s3lsvers
--------
List versions of some feed. Could output into CSV file (-list-file) and/or html chart (-html-file).::
$ s3lsvers -h
usage: s3lsvers [-h] [-from None] [-to None] [-list-file None]
[-html-file None] [-version-id None] [-profile-name None]
[-aws-access-key-id None] [-aws-secret-access-key None]
bucket_key
List object versions stored on versioned S3 bucket, create CSV and/or HTML file.
CSV file can be used e.g. by `s3getvers` command.
HTML file allows showing feed size and update period in chart.
Version can be limited by time range `from` - `to`.
`version-id` allow starting from specific version (back to the past,
excluding given version).
Object key is defined either as {bucket_name}/{key_name} or as alias from .s3lsvers file.
Times are expressed in RFC 3339 format using Zulu (UTC) timezone, possibly truncated.
For truncated time strings, maximal time extent is used.
Listing has records with structure:
`{key_name};{version_id};{size};{last_modified};{age}`
- key_name: name of the key (excluding bucket name).
- version_id: unique identifier for given version on given bucket.
- size: size of key object in bytes
- last_modified: RFC 3339 formated object modification time
- age: update interval [s] for given version
Examples:
Lists all versions of given `keyname` on `bucket`::
$ s3lsvers bucketname/keyname
Lists all versions in period betwen `from` and `to` time::
$ s3lsvers -from 2010-01-01 -to 2011-07-19T12:00:00 bucket/key
Lists all versions and writes them into csv file named `versions.csv`::
$ s3lsvers -list-file versions.csv bucketname/keyname
Lists all versions and write them into html chart file `chart.html`::
$ s3lsvers -html-file chart.html bucketname/keyname
Using bucket/key_name aliases in .s3lsvers file
Aliases are specified in file .s3lsvers, which may be located in
currect directory, home directory or /etc/s3lsvers"
`.s3lsvers` example::
#.s3lsversrc - definition of some preconfigured bucket/key values
[DEFAULT]
pl-base: pl-base.dp.tamtamresearch.com
cz-base: cz-base.dp.tamtamresearch.com
# alias name must not contain "/"
[aliases]
plcsr: %(pl-base)s/region/pl/ConsumerServiceReady.xml
czcsr: %(cz-base)s/region/cz/ConsumerServiceReady.xml
The format follows SafeConfigParser rules, see
http://docs.python.org/2/library/configparser.html#safeconfigparser-objects
To list all versions of czcsr alias::
$ s3lsvers czcsr
positional arguments:
bucket_key {bucket_name}/{key_name} for the key to list
optional arguments:
-h, --help show this help message and exit
-from None, --from-time None
start of version modification time range (default:
oldest version)
-to None, --to-time None
end of version modification time range (default: now)
-list-file None Name of output CSV file.
-html-file None Name of output HTML file.
-version-id None version-id to start after
-profile-name None AWSCLI profile name
-aws-access-key-id None
AWS Access Key ID
-aws-secret-access-key None
AWS Secret Access Key
s3getvers
---------
::
$ s3getvers -h
usage: s3getvers [-h] [-output-version-id-names] [-no-decompression]
[-profile-name None] [-aws-access-key-id None]
[-aws-secret-access-key None]
bucket_name csv_version_file
Fetch S3 object versions as listed in a csv file
Typical csv file (as by default produced by s3lsvers) is:
m/y.xml;OrUr6XO8KSKEHbd8mQ.MloGcGlsh7Sir;191;2012-05-23T20:45:10.000Z;39
m/y.xml;xhkVOy.dJfjSfUwse8tsieqjDicp0owq;192;2012-05-23T20:44:31.000Z;62
m/y.xml;oKneK.N2wS8pW8.EmLqjldYlgcFwxN3V;193;2012-05-23T20:43:29.000Z;58
for `s3getvers` only the first two columns are significant:
:key_name: name of the object (not containing the bucket name itself)
:version_id: string, identifying unique version.
Typical use (assuming, above csv file is available under name verlist.csv)::
$ s3getvers yourbucketname verlist.csv
What will create following files in current directory:
* f.2012-05-23T20_45_10.xml
* f.2012-05-23T20_44_31.xml
* f.2012-05-23T20_43_29.xml
Files are (by default) saved decompressed (even if gzipped on the bucket)
positional arguments:
bucket_name bucket name (default: None)
csv_version_file name of CSV file with version_id
optional arguments:
-h, --help show this help message and exit
-output-version-id-names
Resulting file names shall use version_id to become
distinguished (default is to use timestamp of file
creation)
-no-decompression Keeps the files as they come, do not decompress, if
they come compressed
-profile-name None Name of AWSCLI profile to use for credentials
-aws-access-key-id None
Your AWS Access Key ID
-aws-secret-access-key None
Your AWS Secret Access Key
s3tmpgen
--------
::
$ s3tmpgen -h
usage: s3tmpgen [-h] [-profile-name None] [-aws-access-key-id None] [-aws-secret-access-key None] [-validate-bucket] [-validate-key] [-http] expire_dt bucket_name [key_names [key_names ...]]
Generate temporary url for accessing content of AWS S3 key.
Temporary url includes expiration time, after which it rejects serving the
content.
Urls are printed one per line to stdout.
For missing key names empty line is printed and error goes to stderr.
If the bucket is versioned, tmp url will serve the latest version
at the moment of request (version_id is not part of generated url).
By default, bucket and key name existnence is not verified.
Url is using https, unless `-http` is used.
positional arguments:
expire_dt ISO formatted time of expiration, full seconds, 'Z' is obligatory, e.g. '2014-02-14T21:47:16Z'
bucket_name name of bucket
key_names key names to generate tmpurl for
optional arguments:
-h, --help show this help message and exit
-profile-name None Name of AWSCLI profile to use for credentials
-aws-access-key-id None
Your AWS Access Key ID
-aws-secret-access-key None
Your AWS Secret Access Key
-validate-bucket Make sure, the bucket really exists
-validate-key Make sure, the key really exists
-http Force the url to use http and not https
Configuring AWS S3 credentials
==============================
Configure the credentials as you would do for using AWS CLI.
If you configure profiles, you may use switch `-profile` when calling the commands.
.. _AWS: http://aws.amazon.com/
.. _S3: http://aws.amazon.com/s3/
.. _versioning: http://aws.amazon.com/about-aws/whats-new/2010/02/08/versioning-feature-for-amazon-s3-now-available/
.. _AWS_config: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/tamtamresearch-public/pypi/ttr.aws.utils.s3",
"name": "ttr-aws-utils-s3",
"maintainer": "",
"docs_url": "https://pythonhosted.org/ttr-aws-utils-s3/",
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "aws s3",
"author": "Jan Vlcinsky",
"author_email": "Unknown <jan.vlcinsky@tamtamresearch.com>",
"download_url": "https://files.pythonhosted.org/packages/84/03/ad422886d350f57508b6d861b71e48432d9c2f630a25a1b6d4e6d6ca69c3/ttr_aws_utils_s3-0.6.1.tar.gz",
"platform": null,
"description": "============\nAWS Utils S3\n============\n\nTools to list and fetch objects from versioned AWS_ S3_ bucket:\n\n* `s3lsvers`: List object versions, see versioning_\n* `s3getvers`: Fetch specified object versions\n* `s3tmpgen`: Generate temporary url links to objects\n\n.. contents:: Table of Contents\n\nInstallation\n============\nUsing `pip`::\n\n $ pip install ttr.aws.utils.s3\n\nUsing `pipx`::\n\n $ pipx install ttr.aws.utils.s3\n \nQuick start\n===========\nWe want to fetch versions of feed in bucket `mybucket` named `my/versioned/feed.xml`\n\n1. Configure AWSCLI credentials to allow access to your buckets and objects. E.g. using `AWS_DEFAULT_PROFILE`. See AWS_config_.\n\n2. create csv file for given feed and time period::\n\n $ s3lsvers -from 2012-05-24T00:15 -to 2012-05-24T01:15 -list-file list.csv mybucket/my/versioned/feed.xml\n\n You shall then find file `list.csv` on your disk.\n\n3. Review records in `list.csv` and delete all lines with version, which are not of your interest.\n\n4. Using `list.csv`, ask s3getvers to fetch all versions specified in the file. Be sure to run it on empty directory::\n\n $ s3getvers mybucket list.csv\n\n You will see, how is each version downloaded and saved to your current directory.\n\n5. Finally, you can try generating temorary url to your feed (showing the latest existing)::\n\n $ s3tmpgen 2014-09-30T00:00:00Z mybucket my/versioned/feed.xml\n https://mybucket.s3.amazonaws.com/my/versioned/feed.xml?Signature=kOCwz%2FkanVWX8O15dlXhy4jrbwY%3D&Expires=1412031600&AWSAccessKeyId=AKIAxyzxyzxyzEQA\n\n Note, that the url does not include VersionId, so it will always point to the most up todate version (in case the key happens to be on versioned bucket).\n\nProvided commands\n=================\n\ns3lsvers\n--------\nList versions of some feed. Could output into CSV file (-list-file) and/or html chart (-html-file).::\n\n $ s3lsvers -h\n usage: s3lsvers [-h] [-from None] [-to None] [-list-file None]\n [-html-file None] [-version-id None] [-profile-name None]\n [-aws-access-key-id None] [-aws-secret-access-key None]\n bucket_key\n\n List object versions stored on versioned S3 bucket, create CSV and/or HTML file.\n CSV file can be used e.g. by `s3getvers` command.\n HTML file allows showing feed size and update period in chart.\n\n Version can be limited by time range `from` - `to`.\n `version-id` allow starting from specific version (back to the past,\n excluding given version).\n\n Object key is defined either as {bucket_name}/{key_name} or as alias from .s3lsvers file.\n\n Times are expressed in RFC 3339 format using Zulu (UTC) timezone, possibly truncated.\n For truncated time strings, maximal time extent is used.\n\n Listing has records with structure:\n `{key_name};{version_id};{size};{last_modified};{age}`\n - key_name: name of the key (excluding bucket name).\n - version_id: unique identifier for given version on given bucket.\n - size: size of key object in bytes\n - last_modified: RFC 3339 formated object modification time\n - age: update interval [s] for given version\n\n Examples:\n\n Lists all versions of given `keyname` on `bucket`::\n\n $ s3lsvers bucketname/keyname\n\n Lists all versions in period betwen `from` and `to` time::\n\n $ s3lsvers -from 2010-01-01 -to 2011-07-19T12:00:00 bucket/key\n\n Lists all versions and writes them into csv file named `versions.csv`::\n\n $ s3lsvers -list-file versions.csv bucketname/keyname\n\n Lists all versions and write them into html chart file `chart.html`::\n\n $ s3lsvers -html-file chart.html bucketname/keyname\n\n Using bucket/key_name aliases in .s3lsvers file\n\n Aliases are specified in file .s3lsvers, which may be located in\n currect directory, home directory or /etc/s3lsvers\"\n\n `.s3lsvers` example::\n\n #.s3lsversrc - definition of some preconfigured bucket/key values\n [DEFAULT]\n pl-base: pl-base.dp.tamtamresearch.com\n cz-base: cz-base.dp.tamtamresearch.com\n\n # alias name must not contain \"/\"\n [aliases]\n plcsr: %(pl-base)s/region/pl/ConsumerServiceReady.xml\n czcsr: %(cz-base)s/region/cz/ConsumerServiceReady.xml\n\n The format follows SafeConfigParser rules, see\n http://docs.python.org/2/library/configparser.html#safeconfigparser-objects\n\n To list all versions of czcsr alias::\n\n $ s3lsvers czcsr\n\n\n positional arguments:\n bucket_key {bucket_name}/{key_name} for the key to list\n\n optional arguments:\n -h, --help show this help message and exit\n -from None, --from-time None\n start of version modification time range (default:\n oldest version)\n -to None, --to-time None\n end of version modification time range (default: now)\n -list-file None Name of output CSV file.\n -html-file None Name of output HTML file.\n -version-id None version-id to start after\n -profile-name None AWSCLI profile name\n -aws-access-key-id None\n AWS Access Key ID\n -aws-secret-access-key None\n AWS Secret Access Key\n\ns3getvers\n---------\n::\n\n $ s3getvers -h\n usage: s3getvers [-h] [-output-version-id-names] [-no-decompression]\n [-profile-name None] [-aws-access-key-id None]\n [-aws-secret-access-key None]\n bucket_name csv_version_file\n\n Fetch S3 object versions as listed in a csv file\n\n Typical csv file (as by default produced by s3lsvers) is:\n\n m/y.xml;OrUr6XO8KSKEHbd8mQ.MloGcGlsh7Sir;191;2012-05-23T20:45:10.000Z;39\n m/y.xml;xhkVOy.dJfjSfUwse8tsieqjDicp0owq;192;2012-05-23T20:44:31.000Z;62\n m/y.xml;oKneK.N2wS8pW8.EmLqjldYlgcFwxN3V;193;2012-05-23T20:43:29.000Z;58\n\n for `s3getvers` only the first two columns are significant:\n :key_name: name of the object (not containing the bucket name itself)\n :version_id: string, identifying unique version.\n\n Typical use (assuming, above csv file is available under name verlist.csv)::\n\n $ s3getvers yourbucketname verlist.csv\n\n What will create following files in current directory:\n\n * f.2012-05-23T20_45_10.xml\n * f.2012-05-23T20_44_31.xml\n * f.2012-05-23T20_43_29.xml\n\n Files are (by default) saved decompressed (even if gzipped on the bucket)\n\n\n positional arguments:\n bucket_name bucket name (default: None)\n csv_version_file name of CSV file with version_id\n\n optional arguments:\n -h, --help show this help message and exit\n -output-version-id-names\n Resulting file names shall use version_id to become\n distinguished (default is to use timestamp of file\n creation)\n -no-decompression Keeps the files as they come, do not decompress, if\n they come compressed\n -profile-name None Name of AWSCLI profile to use for credentials\n -aws-access-key-id None\n Your AWS Access Key ID\n -aws-secret-access-key None\n Your AWS Secret Access Key\n\ns3tmpgen\n--------\n\n\n::\n\n $ s3tmpgen -h\n usage: s3tmpgen [-h] [-profile-name None] [-aws-access-key-id None] [-aws-secret-access-key None] [-validate-bucket] [-validate-key] [-http] expire_dt bucket_name [key_names [key_names ...]]\n\n Generate temporary url for accessing content of AWS S3 key.\n\n Temporary url includes expiration time, after which it rejects serving the\n content.\n\n Urls are printed one per line to stdout.\n\n For missing key names empty line is printed and error goes to stderr.\n\n If the bucket is versioned, tmp url will serve the latest version\n at the moment of request (version_id is not part of generated url).\n\n By default, bucket and key name existnence is not verified.\n\n Url is using https, unless `-http` is used.\n \n\n positional arguments:\n expire_dt ISO formatted time of expiration, full seconds, 'Z' is obligatory, e.g. '2014-02-14T21:47:16Z'\n bucket_name name of bucket\n key_names key names to generate tmpurl for\n\n optional arguments:\n -h, --help show this help message and exit\n -profile-name None Name of AWSCLI profile to use for credentials\n -aws-access-key-id None\n Your AWS Access Key ID\n -aws-secret-access-key None\n Your AWS Secret Access Key\n -validate-bucket Make sure, the bucket really exists\n -validate-key Make sure, the key really exists\n -http Force the url to use http and not https\n \n\nConfiguring AWS S3 credentials\n==============================\n\nConfigure the credentials as you would do for using AWS CLI.\n\nIf you configure profiles, you may use switch `-profile` when calling the commands.\n\n\n.. _AWS: http://aws.amazon.com/\n.. _S3: http://aws.amazon.com/s3/\n.. _versioning: http://aws.amazon.com/about-aws/whats-new/2010/02/08/versioning-feature-for-amazon-s3-now-available/\n.. _AWS_config: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "CLI to list and fetch objects from versioned S3 buckets. Plus get tmp url.",
"version": "0.6.1",
"project_urls": {
"Homepage": "https://gitlab.com/tamtamresearch-public/pypi/ttr.aws.utils.s3"
},
"split_keywords": [
"aws",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dd4b9af32608164879024d3d3eb4fc5dbbc5ae5e2622e4f77304f5330c14e186",
"md5": "f22fc9fb41ea8f09b4741d3608c0b857",
"sha256": "2d257b3054c89c8481ddce61a1b67831f62274684e4f4ed974f0a58276e86363"
},
"downloads": -1,
"filename": "ttr_aws_utils_s3-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f22fc9fb41ea8f09b4741d3608c0b857",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16250,
"upload_time": "2023-07-10T15:15:07",
"upload_time_iso_8601": "2023-07-10T15:15:07.095874Z",
"url": "https://files.pythonhosted.org/packages/dd/4b/9af32608164879024d3d3eb4fc5dbbc5ae5e2622e4f77304f5330c14e186/ttr_aws_utils_s3-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8403ad422886d350f57508b6d861b71e48432d9c2f630a25a1b6d4e6d6ca69c3",
"md5": "df468426ae3e5097a79e1b23b7c48422",
"sha256": "eecc606d32dc6a2d69b196f8175ed778db55bbdf15784db38b2e141f3de7c54e"
},
"downloads": -1,
"filename": "ttr_aws_utils_s3-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "df468426ae3e5097a79e1b23b7c48422",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 16034,
"upload_time": "2023-07-10T15:15:08",
"upload_time_iso_8601": "2023-07-10T15:15:08.977338Z",
"url": "https://files.pythonhosted.org/packages/84/03/ad422886d350f57508b6d861b71e48432d9c2f630a25a1b6d4e6d6ca69c3/ttr_aws_utils_s3-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-10 15:15:08",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "tamtamresearch-public",
"gitlab_project": "pypi",
"lcname": "ttr-aws-utils-s3"
}
JSON