| Name | merges3logs JSON |
| Version |
1.0.1
JSON |
| download |
| home_page | |
| Summary | Download cloudfront logs from S3 and merge them into a single log file for the day. |
| upload_time | 2023-11-10 23:31:15 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | Creative Commons Legal Code CC0 1.0 Universal CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER. Statement of Purpose The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an "owner") of an original work of authorship and/or a database (each, a "Work"). Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works ("Commons") that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others. For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the "Affirmer"), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights. 1. Copyright and Related Rights. A Work made available under CC0 may be protected by copyright and related or neighboring rights ("Copyright and Related Rights"). Copyright and Related Rights include, but are not limited to, the following: i. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work; ii. moral rights retained by the original author(s) and/or performer(s); iii. publicity and privacy rights pertaining to a person's image or likeness depicted in a Work; iv. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below; v. rights protecting the extraction, dissemination, use and reuse of data in a Work; vi. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and vii. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof. 2. Waiver. To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose. 3. Public License Fallback. Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "License"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose. 4. Limitations and Disclaimers. a. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document. b. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law. c. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work. d. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work. |
| keywords |
aws
logfile
s3
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# merges3logs
Download cloudfront logs from S3 and merge them into a single log file for the day.
## Installation
```bash
pip install merges3logs
```
## Usage
Set up the config file as documented below.
Run:
```bash
merges3logs path/to/config.ini
```
This will pick up the logs from yesterday (UTC). To run for a different date, run:
```bash
merges3logs path/to/config.ini --date YYYY-MM-DD
```
## When to Run
Some log lines for a given date will show up in the log files dated the following day.
This is because the last bit of one day doesn't get flushed until the following day, and
the file written will have that days date. These dates are all in UTC. Depending on
your logging configuration, it could take hours for all the last logs for a day to get
flushed to S3.
So, you probably want to run this program at least a few hours after midnight, UTC.
Any logs you download for the following day are cached and not re-downloaded the next day.
## Configuration
The bulk of the configuration is done via a ".ini"-style configuration file. Here is
an example:
```ini
[AWS]
AccessKey = XXX
SecretKey = XXX
[S3]
# Bucket to download log files from
BucketName = mylogsbucket
# The prefix of the logfiles to download.
# The "%" must be doubled, strftime format specifiers may be used
Prefix = path/to/logfileprefix_log-%%Y-%%m-%%d-
# Number of parallel downloads to do
MaxWorkers = 10
[Local]
# Directory to write files downloaded from S3
CacheDir = /path/to/mylogsbucketcache
# Directory to write merged logfiles to
DestDir = /path/to/merged-logs
# .gz is added to this logfile name
# The "%" must be doubled, strftime format specifiers may be used
DestFilename = webworkers-cloudfront-%%Y-%%m-%%d.log
# Remove the day's cached logfiles after a successful run?
RemoveFiles = False
```
Details:
- AWS specifies your access and secret keys.
- S3.BucketName is the name of your bucket.
- S3.Prefix is the "prefix" of your log file names for a certain date. An S3 prefix
is everything after the bucket name up to and including the date, with the date
encoded using "strftime()" format, however the "%"s need to be doubled (because
INI format otherwise interprets them).
- S3.MaxWorkers is the number of download jobs that will run to get logs. Depending
on your logging configuration in Cloudfront and how widely your services are accessed,
this can be tens or hundreds of thousands of log files a day. So running downloads
in parallel can really speed it up.
- Local.CacheDir is the path to a directory to store the downloaded log files.
This directory will need to have a cleanup job set up to prevent it from growing
unbounded. See also "Local.RemoveFiles".
- Local.DestDir is the directory that the merged log files will be written to.
- Local.DestFilename is the name of the file that will be written in the DestDir
with "strftime()" format to specify the date.
- Local.RemoveFiles, if "True" will delete the days files from the cache directory
after a successful run. If "False", they are kept and you will need to set up a
cron job or similar to delete them. Probably most useful for testing, so
repeated downloads are unnecessary. Default is "True".
## Cleanup
merges3logs will download the log files into a cache directory, and then work from
the files there. You can use "Local.RemoveFiles" to delete them after the run, or
set up a cron job for example:
```bash
find /path/to/cachedir -type f -mtime +3 -exec rm {} +
```
It is probably worthwhile to set up cleaning of the cache anyway, as it can be
large and may accumulate files if the program fails for any reason.
You will also need to clean up the destination log directory, though it does grow
much more slowly (logs are compressed and only one file per day). Something like
using logrotate or:
```bash
find /path/to/merged-logs -type f -mtime +60 -exec rm {} +
```
## Author
Written by Sean Reifschneider, Oct 2023.
## License
CC0 1.0 Universal, see LICENSE file for more information.
<!-- vim: ts=4 sw=4 ai et tw=85
-->
Raw data
{
"_id": null,
"home_page": "",
"name": "merges3logs",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Sean Reifschneider <jafo00@gmail.com>",
"keywords": "aws,logfile,s3",
"author": "",
"author_email": "Sean Reifschneider <jafo00@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/88/53/09b5128a211f77f026aae7ad7b7c829563ddffe5858c018160f707bfea65/merges3logs-1.0.1.tar.gz",
"platform": null,
"description": "# merges3logs\n\nDownload cloudfront logs from S3 and merge them into a single log file for the day.\n\n## Installation\n\n```bash\npip install merges3logs\n```\n\n## Usage\n\nSet up the config file as documented below.\n\nRun:\n\n```bash\nmerges3logs path/to/config.ini\n```\n\nThis will pick up the logs from yesterday (UTC). To run for a different date, run:\n\n```bash\nmerges3logs path/to/config.ini --date YYYY-MM-DD\n```\n\n## When to Run\n\nSome log lines for a given date will show up in the log files dated the following day.\nThis is because the last bit of one day doesn't get flushed until the following day, and\nthe file written will have that days date. These dates are all in UTC. Depending on\nyour logging configuration, it could take hours for all the last logs for a day to get\nflushed to S3.\n\nSo, you probably want to run this program at least a few hours after midnight, UTC.\n\nAny logs you download for the following day are cached and not re-downloaded the next day.\n\n## Configuration\n\nThe bulk of the configuration is done via a \".ini\"-style configuration file. Here is\nan example:\n\n```ini\n[AWS]\nAccessKey = XXX\nSecretKey = XXX\n\n[S3]\n# Bucket to download log files from\nBucketName = mylogsbucket\n# The prefix of the logfiles to download.\n# The \"%\" must be doubled, strftime format specifiers may be used\nPrefix = path/to/logfileprefix_log-%%Y-%%m-%%d-\n# Number of parallel downloads to do\nMaxWorkers = 10\n\n[Local]\n# Directory to write files downloaded from S3\nCacheDir = /path/to/mylogsbucketcache\n# Directory to write merged logfiles to\nDestDir = /path/to/merged-logs\n# .gz is added to this logfile name\n# The \"%\" must be doubled, strftime format specifiers may be used\nDestFilename = webworkers-cloudfront-%%Y-%%m-%%d.log\n# Remove the day's cached logfiles after a successful run?\nRemoveFiles = False\n```\n\nDetails:\n\n- AWS specifies your access and secret keys.\n- S3.BucketName is the name of your bucket.\n- S3.Prefix is the \"prefix\" of your log file names for a certain date. An S3 prefix\n is everything after the bucket name up to and including the date, with the date\n encoded using \"strftime()\" format, however the \"%\"s need to be doubled (because\n INI format otherwise interprets them).\n- S3.MaxWorkers is the number of download jobs that will run to get logs. Depending\n on your logging configuration in Cloudfront and how widely your services are accessed,\n this can be tens or hundreds of thousands of log files a day. So running downloads\n in parallel can really speed it up.\n- Local.CacheDir is the path to a directory to store the downloaded log files.\n This directory will need to have a cleanup job set up to prevent it from growing\n unbounded. See also \"Local.RemoveFiles\".\n- Local.DestDir is the directory that the merged log files will be written to.\n- Local.DestFilename is the name of the file that will be written in the DestDir\n with \"strftime()\" format to specify the date.\n- Local.RemoveFiles, if \"True\" will delete the days files from the cache directory\n after a successful run. If \"False\", they are kept and you will need to set up a\n cron job or similar to delete them. Probably most useful for testing, so\n repeated downloads are unnecessary. Default is \"True\".\n\n## Cleanup\n\nmerges3logs will download the log files into a cache directory, and then work from\nthe files there. You can use \"Local.RemoveFiles\" to delete them after the run, or\nset up a cron job for example:\n\n```bash\nfind /path/to/cachedir -type f -mtime +3 -exec rm {} +\n```\n\nIt is probably worthwhile to set up cleaning of the cache anyway, as it can be\nlarge and may accumulate files if the program fails for any reason.\n\nYou will also need to clean up the destination log directory, though it does grow\nmuch more slowly (logs are compressed and only one file per day). Something like\nusing logrotate or:\n\n```bash\nfind /path/to/merged-logs -type f -mtime +60 -exec rm {} +\n```\n\n## Author\n\nWritten by Sean Reifschneider, Oct 2023.\n\n## License\n\nCC0 1.0 Universal, see LICENSE file for more information.\n\n<!-- vim: ts=4 sw=4 ai et tw=85\n-->\n",
"bugtrack_url": null,
"license": "Creative Commons Legal Code CC0 1.0 Universal CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN \"AS-IS\" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER. Statement of Purpose The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an \"owner\") of an original work of authorship and/or a database (each, a \"Work\"). Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works (\"Commons\") that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others. For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the \"Affirmer\"), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights. 1. Copyright and Related Rights. A Work made available under CC0 may be protected by copyright and related or neighboring rights (\"Copyright and Related Rights\"). Copyright and Related Rights include, but are not limited to, the following: i. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work; ii. moral rights retained by the original author(s) and/or performer(s); iii. publicity and privacy rights pertaining to a person's image or likeness depicted in a Work; iv. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below; v. rights protecting the extraction, dissemination, use and reuse of data in a Work; vi. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and vii. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof. 2. Waiver. To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the \"Waiver\"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose. 3. Public License Fallback. Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the \"License\"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose. 4. Limitations and Disclaimers. a. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document. b. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law. c. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work. d. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work.",
"summary": "Download cloudfront logs from S3 and merge them into a single log file for the day.",
"version": "1.0.1",
"project_urls": {
"Source": "https://github.com/linsomniac/merges3logs"
},
"split_keywords": [
"aws",
"logfile",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "698e8e279800c1baee1697b2e5519c71d704778f2e1eb6486126d5259b4757d5",
"md5": "c9fbb7e9f1d071be8854d968d34a0243",
"sha256": "92283441937a336d0294ed87478761da2e409a375d42ef8bf4fea60c38f03241"
},
"downloads": -1,
"filename": "merges3logs-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c9fbb7e9f1d071be8854d968d34a0243",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11572,
"upload_time": "2023-11-10T23:31:13",
"upload_time_iso_8601": "2023-11-10T23:31:13.859735Z",
"url": "https://files.pythonhosted.org/packages/69/8e/8e279800c1baee1697b2e5519c71d704778f2e1eb6486126d5259b4757d5/merges3logs-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "885309b5128a211f77f026aae7ad7b7c829563ddffe5858c018160f707bfea65",
"md5": "02b4fdcdd19a590c06e45820e4217d09",
"sha256": "bea61c0b27d8cddcbf1080813044539c216160a409dfa79e710d137884fdfdcc"
},
"downloads": -1,
"filename": "merges3logs-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "02b4fdcdd19a590c06e45820e4217d09",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 9193,
"upload_time": "2023-11-10T23:31:15",
"upload_time_iso_8601": "2023-11-10T23:31:15.451887Z",
"url": "https://files.pythonhosted.org/packages/88/53/09b5128a211f77f026aae7ad7b7c829563ddffe5858c018160f707bfea65/merges3logs-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-10 23:31:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "linsomniac",
"github_project": "merges3logs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "merges3logs"
}
