TrailScraper
============
|PyPi Release| |Build Status|
A command-line tool to get valuable information out of AWS CloudTrail
and a general purpose toolbox for working with IAM policies
Installation
------------
OSX
~~~
.. code:: bash
$ brew install trailscraper
Installation using pip
~~~~~~~~~~~~~~~~~~~~~~
Requirements:
- Python >= 3.5
- pip
.. code:: bash
$ pip install trailscraper
Run directly using docker
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code:: bash
$ docker run --rm --env-file <(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latest
Current Versions starting from 0.7.0 are found on GitHub Container
Registry (``ghcr.io``), older versions on
`DockerHub <https://hub.docker.com/r/flosell/trailscraper/>`__
Usage
-----
- `Get CloudTrail events matching a filter from CloudTrail
API <#get-cloudtrail-events-matching-a-filter-from-cloudtrail-api>`__
- `Download some logs <#download-some-logs>`__
- `Download some logs in organisational
trails <#download-some-logs-in-organisational-trails>`__
- `Find CloudTrail events matching a filter in downloaded
logs <#find-cloudtrail-events-matching-a-filter-in-downloaded-logs>`__
- `Generate Policy from some CloudTrail
records <#generate-policy-from-some-cloudtrail-records>`__
- `Extend existing policy by guessing matching
actions <#extend-existing-policy-by-guessing-matching-actions>`__
- `Find CloudTrail events and generate an IAM
Policy <#find-cloudtrail-events-and-generate-an-iam-policy>`__
Get CloudTrail events matching a filter from CloudTrail API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ trailscraper select --use-cloudtrail-api \
--filter-assumed-role-arn some-arn \
--from 'one hour ago' \
--to 'now'
{
"Records": [
{
"eventTime": "2017-12-11T15:01:51Z",
"eventSource": "autoscaling.amazonaws.com",
"eventName": "DescribeLaunchConfigurations",
...
Download some logs
~~~~~~~~~~~~~~~~~~
::
$ trailscraper download --bucket some-bucket \
--account-id some-account-id \
--region some-other-region \
--region us-east-1 \
--from 'two days ago' \
--to 'now' \
*Note: Include us-east-1 to download logs for global services. See*
`below <#why-is-trailscraper-missing-some-events>`__ *for details*
Download some logs in organisational trails
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ trailscraper download --bucket some-bucket \
--account-id some-account-id \
--region us-east-1 \
--org-id o-someorgid \
--from 'two days ago' \
--to 'now'
Find CloudTrail events matching a filter in downloaded logs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ trailscraper select --filter-assumed-role-arn some-arn \
--from 'one hour ago' \
--to 'now'
{
"Records": [
{
"eventTime": "2017-12-11T15:01:51Z",
"eventSource": "autoscaling.amazonaws.com",
"eventName": "DescribeLaunchConfigurations",
...
Generate Policy from some CloudTrail records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ gzcat some-records.json.gz | trailscraper generate
{
"Statement": [
{
"Action": [
"ec2:DescribeInstances"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
Extend existing policy by guessing matching actions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CloudTrail logs might not always contain all relevant actions. For
example, your logs might only contain the ``Create`` actions after a
terraform run when you really want the delete and update permissions as
well. TrailScraper can try to guess additional statements that might be
relevant:
::
$ cat minimal-policy.json | trailscraper guess
{
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:ListObjects"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
$ cat minimal-policy.json | ./go trailscraper guess --only Get
{
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
Find CloudTrail events and generate an IAM Policy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
$ trailscraper select | trailscraper generate
{
"Statement": [
{
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:DescribeVpcs",
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:iam::1111111111:role/someRole"
]
}
],
"Version": "2012-10-17"
}
FAQ
---
How can I generate policies in CloudFormation YAML instead of JSON?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TrailScraper doesn’t provide this. But you can use
`cfn-flip <https://github.com/awslabs/aws-cfn-template-flip>`__ to do
it:
::
$ trailscraper select | trailscraper generate | cfn-flip
Statement:
- Action:
- ec2:DescribeInstances
Effect: Allow
Resource:
- '*'
How can I generate policies in Terraform HCL instead of JSON?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TrailScraper doesn’t provide this. But you can use
`iam-policy-json-to-terraform <https://github.com/flosell/iam-policy-json-to-terraform>`__
to do it:
::
$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform
data "aws_iam_policy_document" "policy" {
statement {
sid = ""
effect = "Allow"
resources = ["*"]
actions = [
"ec2:DescribeInstances",
]
}
}
Why is TrailScraper missing some events?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Make sure you have logs for the ``us-east-1`` region. Some global AWS
services (e.g. Route53, IAM, STS, CloudFront) use this region. For
details, check the `CloudTrail
Documentation <http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-global-service-events>`__
Why are some TrailScraper-generated actions not real IAM actions?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is totally possible. Unfortunately, there is no good,
machine-readable documentation on how CloudTrail events map to IAM
actions so TrailScraper is using heuristics to figure out the right
actions. These heuristics likely don’t cover all special cases of the
AWS world.
This is where you come in: If you find a special case that’s not covered
by TrailScraper, please `open a new
issue <https://github.com/flosell/trailscraper/issues/new>`__ or, even
better, submit a pull request.
For more details, check out the `contribution
guide <./CONTRIBUTING.md>`__
Why does click think I am in an ASCII environment?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.``
Set environment variables that describe your locale, e.g. :
::
export LC_ALL=de_DE.utf-8
export LANG=de_DE.utf-8
or
::
LC_ALL=C.UTF-8
LANG=C.UTF-8
For details, see
http://click.pocoo.org/5/python3/#python-3-surrogate-handling
Development
-----------
.. code:: bash
$ ./go setup # set up venv, dependencies and tools
$ ./go test # run some tests
$ ./go check # run some style checks
$ ./go # let's see what we can do here
.. |PyPi Release| image:: https://img.shields.io/pypi/v/trailscraper.svg
:target: https://pypi.python.org/pypi/trailscraper
.. |Build Status| image:: https://github.com/flosell/trailscraper/actions/workflows/check.yml/badge.svg
:target: https://github.com/flosell/trailscraper/actions/workflows/check.yml
Changelog
=========
This changelog contains a loose collection of changes in every release
including breaking changes to the API.
The format is based on `Keep a Changelog <http://keepachangelog.com/>`__
0.8.1
-----
Changed
~~~~~~~
- Updated supporting dependencies
.. _section-1:
0.8.0
-----
Added
~~~~~
- Support for Python 3.10
- Support for Python 3.11
Removed
~~~~~~~
- **Removed official support for EOL Python 3.6**. TrailScraper might
still run but we no longer actively test for it.
.. _section-2:
0.7.0
-----
.. _added-1:
Added
~~~~~
- Support for Python 3.9
.. _removed-1:
Removed
~~~~~~~
- **Removed official support for Python 3.5**. TrailScraper might still
run but we no longer actively test for it
.. _changed-1:
Changed
~~~~~~~
- Moving from DockerHub to GitHub Container registry for Docker images
(since DockerHub is dropping support for autobuilds and restricting
it in other ways)
- Updated supporting dependencies
.. _section-3:
0.6.4
-----
Fixed
~~~~~
- Fixed Docker images that threw a ``ModuleNotFoundError``
.. _changed-2:
Changed
~~~~~~~
- Performance tweaks
- ``trailscraper download`` uses smarter directory listing to
improve performance with large date ranges and little new data
- ``trailscraper download`` now downloads files in parallel
- Minor performance improvements in ``trailscraper select``
0.6.2 and 0.6.3
---------------
(skipeed because of continuing release-script issues)
.. _section-4:
0.6.1
-----
(same as 0.6.1, just fixing inconsistent release)
.. _section-5:
0.6.0
-----
.. _added-2:
Added
~~~~~
- Support for Python 3.7 and 3.8
- Support for org-level trails (#101)
.. _fixed-1:
Fixed
~~~~~
- ``trailscraper guess`` was not working when installed through
homebrew or pip (#110)
.. _removed-2:
Removed
~~~~~~~
- **Removed official support for Python 2.7 and 3.4**. TrailScraper
might still run but we no longer actively test for it
.. _section-6:
0.5.1
-----
.. _added-3:
Added
~~~~~
- New command ``guess`` to extend existing policy by guessing matching
actions #22
.. _fixed-2:
Fixed
~~~~~
- Fixed parsing events that contain resources without an ARN
(e.g. ``s3:ListObjects``) #51
.. _section-7:
0.5.0
-----
**Breaking CLI changes**: split up ``generate-policy`` into ``select``
and ``generate`` (#38)
.. _added-4:
Added
~~~~~
- New command ``select`` to print all CloudTrail records matching a
filter to stdout
- New command ``generate`` to take CloudTrail records from stdin and
generate a policy for it
.. _changed-3:
Changed
~~~~~~~
- New command ``select`` defaults to not filtering at all whereas
``generate-policy`` filtered for recent events by default. Changed to
make filtering more explicit and predictable instead of surprising
users who wonder why their events don’t show up
.. _removed-3:
Removed
~~~~~~~
- Removed command ``generate-policy``, replaced with ``select`` and
``generate``. Use pipes to produce the same behavior:
.. code:: bash
$ trailscraper select | trailscraper generate
.. _section-8:
0.4.4
-----
.. _fixed-3:
Fixed
~~~~~
- Made trailscraper timezone-aware. Until now, trailscraper implicitly
treated everything as UTC, meaning relative timestamps (e.g. ``now``,
``two hours ago``) didn’t work properly when filtering logfiles to
download or records to generate from. (#39)
.. _added-5:
Added
~~~~~
- New command ``trailscraper last-event-timestamp`` to get the last
known event timestamp.
- New flag ``trailscraper download --wait`` to wait until events for
the specified timeframe are found. Useful if you are waiting for
CloudTrail to ship logs for a recent operation.
.. _section-9:
0.4.3
-----
*skipped because of release-problems*
.. _section-10:
0.4.2
-----
.. _fixed-4:
Fixed
~~~~~
- Fixed various special cases in mapping CloudTrail to IAM Actions:
- API Gateway
- App Stream 2
- DynamoDB Streams
- Lex
- Mechanical Turk
- S3
- STS
- Tagging
.. _section-11:
0.4.1
-----
.. _fixed-5:
Fixed
~~~~~
- Ignore record files that can’t be read (e.g. not valid GZIP) in
Python 2.7 (was only working in Python 3.\* before)
- Fixed permissions generated for services that include the API version
date (e.g. Lambda, CloudFront) (#20)
.. _section-12:
0.4.0
-----
.. _added-6:
Added
~~~~~
- Support for CloudTrail ``lookup_events`` API that allows users to
generate a policy without downloading logs from an S3 bucket. Note
that this API only returns `“create, modify, and delete API
calls” <https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-supported-services.html>`__
- ``trailscraper download`` now supports ``--from`` and ``--to`` flags
to specify the timeframe that should be downloaded. Accepts precise
(e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.
- ``trailscraper generate-policy`` now supports ``--from`` and ``--to``
to filter events to consider for the generated policy. Accepts
precise (e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.
- Performance optimizations: ``generate-policy`` only reads logfiles
for the timeframe requested
- Added ``--version`` command line argument
.. _changed-4:
Changed
~~~~~~~
- Set more flexible dependencies
.. _removed-4:
Removed
~~~~~~~
- Removed ``--past-days`` parameter in ``trailscraper download``. Was
replaced by ``--from`` and ``--to`` (see above)
.. _fixed-6:
Fixed
~~~~~
- Ignore record files that can’t be read (e.g. not valid GZIP)
.. _section-13:
0.3.0
-----
.. _added-7:
Added
~~~~~
- Support for Python >= 2.7
.. _changed-5:
Changed
~~~~~~~
- Do not download CloudTrail Logs from S3 if they already exist in the
target folder (#9)
- Removed dependency on fork of the awacs-library to simplify
installation and development
.. _fixed-7:
Fixed
~~~~~
- Bug that led to policy-statements with the same set of actions not
being combined properly in some cases (#7)
.. _section-14:
0.2.0
-----
.. _added-8:
Added
~~~~~
- Basic filtering for role-arns when generating policy (#3)
.. _section-15:
0.1.0
-----
*Initial Release*
.. _added-9:
Added
~~~~~
- Basic feature to download CloudTrail Logs from S3 for certain
accounts and timeframe
- Basic feature to generate IAM Policies from a set of downloaded
CloudTrail logs
Raw data
{
"_id": null,
"home_page": "http://github.com/flosell/trailscraper",
"name": "trailscraper",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "aws cloud iam cloudtrail trailscraper",
"author": "Florian Sellmayr",
"author_email": "florian.sellmayr@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/bc/9b/f425ff02c84a16e434526d3ffe7abfc50589f46a5efe9b02cfd09bec698e/trailscraper-0.8.1.tar.gz",
"platform": null,
"description": "TrailScraper\n============\n\n|PyPi Release| |Build Status|\n\nA command-line tool to get valuable information out of AWS CloudTrail\nand a general purpose toolbox for working with IAM policies\n\nInstallation\n------------\n\nOSX\n~~~\n\n.. code:: bash\n\n $ brew install trailscraper\n\nInstallation using pip\n~~~~~~~~~~~~~~~~~~~~~~\n\nRequirements:\n\n- Python >= 3.5\n- pip\n\n.. code:: bash\n\n $ pip install trailscraper\n\nRun directly using docker\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: bash\n\n $ docker run --rm --env-file <(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latest\n\nCurrent Versions starting from 0.7.0 are found on GitHub Container\nRegistry (``ghcr.io``), older versions on\n`DockerHub <https://hub.docker.com/r/flosell/trailscraper/>`__\n\nUsage\n-----\n\n- `Get CloudTrail events matching a filter from CloudTrail\n API <#get-cloudtrail-events-matching-a-filter-from-cloudtrail-api>`__\n- `Download some logs <#download-some-logs>`__\n- `Download some logs in organisational\n trails <#download-some-logs-in-organisational-trails>`__\n- `Find CloudTrail events matching a filter in downloaded\n logs <#find-cloudtrail-events-matching-a-filter-in-downloaded-logs>`__\n- `Generate Policy from some CloudTrail\n records <#generate-policy-from-some-cloudtrail-records>`__\n- `Extend existing policy by guessing matching\n actions <#extend-existing-policy-by-guessing-matching-actions>`__\n- `Find CloudTrail events and generate an IAM\n Policy <#find-cloudtrail-events-and-generate-an-iam-policy>`__\n\nGet CloudTrail events matching a filter from CloudTrail API\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ trailscraper select --use-cloudtrail-api \\ \n --filter-assumed-role-arn some-arn \\ \n --from 'one hour ago' \\ \n --to 'now'\n {\n \"Records\": [\n {\n \"eventTime\": \"2017-12-11T15:01:51Z\",\n \"eventSource\": \"autoscaling.amazonaws.com\",\n \"eventName\": \"DescribeLaunchConfigurations\",\n ...\n\nDownload some logs\n~~~~~~~~~~~~~~~~~~\n\n::\n\n $ trailscraper download --bucket some-bucket \\\n --account-id some-account-id \\\n --region some-other-region \\ \n --region us-east-1 \\\n --from 'two days ago' \\\n --to 'now' \\\n\n*Note: Include us-east-1 to download logs for global services. See*\n`below <#why-is-trailscraper-missing-some-events>`__ *for details*\n\nDownload some logs in organisational trails\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ trailscraper download --bucket some-bucket \\\n --account-id some-account-id \\\n --region us-east-1 \\\n --org-id o-someorgid \\\n --from 'two days ago' \\\n --to 'now'\n\nFind CloudTrail events matching a filter in downloaded logs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ trailscraper select --filter-assumed-role-arn some-arn \\ \n --from 'one hour ago' \\ \n --to 'now'\n {\n \"Records\": [\n {\n \"eventTime\": \"2017-12-11T15:01:51Z\",\n \"eventSource\": \"autoscaling.amazonaws.com\",\n \"eventName\": \"DescribeLaunchConfigurations\",\n ...\n\nGenerate Policy from some CloudTrail records\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ gzcat some-records.json.gz | trailscraper generate\n {\n \"Statement\": [\n {\n \"Action\": [\n \"ec2:DescribeInstances\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n }\n ],\n \"Version\": \"2012-10-17\"\n } \n\nExtend existing policy by guessing matching actions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCloudTrail logs might not always contain all relevant actions. For\nexample, your logs might only contain the ``Create`` actions after a\nterraform run when you really want the delete and update permissions as\nwell. TrailScraper can try to guess additional statements that might be\nrelevant:\n\n::\n\n $ cat minimal-policy.json | trailscraper guess\n {\n \"Statement\": [\n {\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n },\n {\n \"Action\": [\n \"s3:DeleteObject\",\n \"s3:GetObject\",\n \"s3:ListObjects\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n }\n ],\n \"Version\": \"2012-10-17\"\n }\n $ cat minimal-policy.json | ./go trailscraper guess --only Get\n {\n \"Statement\": [\n {\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n },\n {\n \"Action\": [\n \"s3:GetObject\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n }\n ],\n \"Version\": \"2012-10-17\"\n }\n\nFind CloudTrail events and generate an IAM Policy\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ trailscraper select | trailscraper generate\n {\n \"Statement\": [\n {\n \"Action\": [\n \"ec2:DescribeInstances\",\n \"ec2:DescribeSecurityGroups\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeVolumes\",\n \"ec2:DescribeVpcs\",\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"*\"\n ]\n },\n {\n \"Action\": [\n \"sts:AssumeRole\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": [\n \"arn:aws:iam::1111111111:role/someRole\"\n ]\n }\n ],\n \"Version\": \"2012-10-17\"\n } \n\nFAQ\n---\n\nHow can I generate policies in CloudFormation YAML instead of JSON?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTrailScraper doesn\u2019t provide this. But you can use\n`cfn-flip <https://github.com/awslabs/aws-cfn-template-flip>`__ to do\nit:\n\n::\n\n $ trailscraper select | trailscraper generate | cfn-flip\n Statement:\n - Action:\n - ec2:DescribeInstances\n Effect: Allow\n Resource:\n - '*'\n\nHow can I generate policies in Terraform HCL instead of JSON?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTrailScraper doesn\u2019t provide this. But you can use\n`iam-policy-json-to-terraform <https://github.com/flosell/iam-policy-json-to-terraform>`__\nto do it:\n\n::\n\n $ trailscraper select | trailscraper generate | iam-policy-json-to-terraform\n data \"aws_iam_policy_document\" \"policy\" {\n statement {\n sid = \"\"\n effect = \"Allow\"\n resources = [\"*\"]\n\n actions = [\n \"ec2:DescribeInstances\",\n ]\n }\n }\n\nWhy is TrailScraper missing some events?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Make sure you have logs for the ``us-east-1`` region. Some global AWS\n services (e.g.\u00a0Route53, IAM, STS, CloudFront) use this region. For\n details, check the `CloudTrail\n Documentation <http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-global-service-events>`__\n\nWhy are some TrailScraper-generated actions not real IAM actions?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis is totally possible. Unfortunately, there is no good,\nmachine-readable documentation on how CloudTrail events map to IAM\nactions so TrailScraper is using heuristics to figure out the right\nactions. These heuristics likely don\u2019t cover all special cases of the\nAWS world.\n\nThis is where you come in: If you find a special case that\u2019s not covered\nby TrailScraper, please `open a new\nissue <https://github.com/flosell/trailscraper/issues/new>`__ or, even\nbetter, submit a pull request.\n\nFor more details, check out the `contribution\nguide <./CONTRIBUTING.md>`__\n\nWhy does click think I am in an ASCII environment?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n``Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.``\n\nSet environment variables that describe your locale, e.g.\u00a0:\n\n::\n\n export LC_ALL=de_DE.utf-8\n export LANG=de_DE.utf-8\n\nor\n\n::\n\n LC_ALL=C.UTF-8\n LANG=C.UTF-8\n\nFor details, see\nhttp://click.pocoo.org/5/python3/#python-3-surrogate-handling\n\nDevelopment\n-----------\n\n.. code:: bash\n\n $ ./go setup # set up venv, dependencies and tools\n $ ./go test # run some tests\n $ ./go check # run some style checks\n $ ./go # let's see what we can do here\n\n.. |PyPi Release| image:: https://img.shields.io/pypi/v/trailscraper.svg\n :target: https://pypi.python.org/pypi/trailscraper\n.. |Build Status| image:: https://github.com/flosell/trailscraper/actions/workflows/check.yml/badge.svg\n :target: https://github.com/flosell/trailscraper/actions/workflows/check.yml\n\n\nChangelog\n=========\n\nThis changelog contains a loose collection of changes in every release\nincluding breaking changes to the API.\n\nThe format is based on `Keep a Changelog <http://keepachangelog.com/>`__\n\n0.8.1\n-----\n\nChanged\n~~~~~~~\n\n- Updated supporting dependencies\n\n.. _section-1:\n\n0.8.0\n-----\n\nAdded\n~~~~~\n\n- Support for Python 3.10\n- Support for Python 3.11\n\nRemoved\n~~~~~~~\n\n- **Removed official support for EOL Python 3.6**. TrailScraper might\n still run but we no longer actively test for it.\n\n.. _section-2:\n\n0.7.0\n-----\n\n.. _added-1:\n\nAdded\n~~~~~\n\n- Support for Python 3.9\n\n.. _removed-1:\n\nRemoved\n~~~~~~~\n\n- **Removed official support for Python 3.5**. TrailScraper might still\n run but we no longer actively test for it\n\n.. _changed-1:\n\nChanged\n~~~~~~~\n\n- Moving from DockerHub to GitHub Container registry for Docker images\n (since DockerHub is dropping support for autobuilds and restricting\n it in other ways)\n- Updated supporting dependencies\n\n.. _section-3:\n\n0.6.4\n-----\n\nFixed\n~~~~~\n\n- Fixed Docker images that threw a ``ModuleNotFoundError``\n\n.. _changed-2:\n\nChanged\n~~~~~~~\n\n- Performance tweaks\n\n - ``trailscraper download`` uses smarter directory listing to\n improve performance with large date ranges and little new data\n - ``trailscraper download`` now downloads files in parallel\n - Minor performance improvements in ``trailscraper select``\n\n0.6.2 and 0.6.3\n---------------\n\n(skipeed because of continuing release-script issues)\n\n.. _section-4:\n\n0.6.1\n-----\n\n(same as 0.6.1, just fixing inconsistent release)\n\n.. _section-5:\n\n0.6.0\n-----\n\n.. _added-2:\n\nAdded\n~~~~~\n\n- Support for Python 3.7 and 3.8\n- Support for org-level trails (#101)\n\n.. _fixed-1:\n\nFixed\n~~~~~\n\n- ``trailscraper guess`` was not working when installed through\n homebrew or pip (#110)\n\n.. _removed-2:\n\nRemoved\n~~~~~~~\n\n- **Removed official support for Python 2.7 and 3.4**. TrailScraper\n might still run but we no longer actively test for it\n\n.. _section-6:\n\n0.5.1\n-----\n\n.. _added-3:\n\nAdded\n~~~~~\n\n- New command ``guess`` to extend existing policy by guessing matching\n actions #22\n\n.. _fixed-2:\n\nFixed\n~~~~~\n\n- Fixed parsing events that contain resources without an ARN\n (e.g.\u00a0``s3:ListObjects``) #51\n\n.. _section-7:\n\n0.5.0\n-----\n\n**Breaking CLI changes**: split up ``generate-policy`` into ``select``\nand ``generate`` (#38)\n\n.. _added-4:\n\nAdded\n~~~~~\n\n- New command ``select`` to print all CloudTrail records matching a\n filter to stdout\n- New command ``generate`` to take CloudTrail records from stdin and\n generate a policy for it\n\n.. _changed-3:\n\nChanged\n~~~~~~~\n\n- New command ``select`` defaults to not filtering at all whereas\n ``generate-policy`` filtered for recent events by default. Changed to\n make filtering more explicit and predictable instead of surprising\n users who wonder why their events don\u2019t show up\n\n.. _removed-3:\n\nRemoved\n~~~~~~~\n\n- Removed command ``generate-policy``, replaced with ``select`` and\n ``generate``. Use pipes to produce the same behavior:\n\n .. code:: bash\n\n $ trailscraper select | trailscraper generate\n\n.. _section-8:\n\n0.4.4\n-----\n\n.. _fixed-3:\n\nFixed\n~~~~~\n\n- Made trailscraper timezone-aware. Until now, trailscraper implicitly\n treated everything as UTC, meaning relative timestamps (e.g.\u00a0``now``,\n ``two hours ago``) didn\u2019t work properly when filtering logfiles to\n download or records to generate from. (#39)\n\n.. _added-5:\n\nAdded\n~~~~~\n\n- New command ``trailscraper last-event-timestamp`` to get the last\n known event timestamp.\n- New flag ``trailscraper download --wait`` to wait until events for\n the specified timeframe are found. Useful if you are waiting for\n CloudTrail to ship logs for a recent operation.\n\n.. _section-9:\n\n0.4.3\n-----\n\n*skipped because of release-problems*\n\n.. _section-10:\n\n0.4.2\n-----\n\n.. _fixed-4:\n\nFixed\n~~~~~\n\n- Fixed various special cases in mapping CloudTrail to IAM Actions:\n\n - API Gateway\n - App Stream 2\n - DynamoDB Streams\n - Lex\n - Mechanical Turk\n - S3\n - STS\n - Tagging\n\n.. _section-11:\n\n0.4.1\n-----\n\n.. _fixed-5:\n\nFixed\n~~~~~\n\n- Ignore record files that can\u2019t be read (e.g.\u00a0not valid GZIP) in\n Python 2.7 (was only working in Python 3.\\* before)\n- Fixed permissions generated for services that include the API version\n date (e.g.\u00a0Lambda, CloudFront) (#20)\n\n.. _section-12:\n\n0.4.0\n-----\n\n.. _added-6:\n\nAdded\n~~~~~\n\n- Support for CloudTrail ``lookup_events`` API that allows users to\n generate a policy without downloading logs from an S3 bucket. Note\n that this API only returns `\u201ccreate, modify, and delete API\n calls\u201d <https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-supported-services.html>`__\n\n- ``trailscraper download`` now supports ``--from`` and ``--to`` flags\n to specify the timeframe that should be downloaded. Accepts precise\n (e.g.\u00a0\u201c2017-10-12\u201d) and relative (e.g.\u00a0\u201c-2days\u201d) arguments.\n\n- ``trailscraper generate-policy`` now supports ``--from`` and ``--to``\n to filter events to consider for the generated policy. Accepts\n precise (e.g.\u00a0\u201c2017-10-12\u201d) and relative (e.g.\u00a0\u201c-2days\u201d) arguments.\n\n- Performance optimizations: ``generate-policy`` only reads logfiles\n for the timeframe requested\n\n- Added ``--version`` command line argument\n\n.. _changed-4:\n\nChanged\n~~~~~~~\n\n- Set more flexible dependencies\n\n.. _removed-4:\n\nRemoved\n~~~~~~~\n\n- Removed ``--past-days`` parameter in ``trailscraper download``. Was\n replaced by ``--from`` and ``--to`` (see above)\n\n.. _fixed-6:\n\nFixed\n~~~~~\n\n- Ignore record files that can\u2019t be read (e.g.\u00a0not valid GZIP)\n\n.. _section-13:\n\n0.3.0\n-----\n\n.. _added-7:\n\nAdded\n~~~~~\n\n- Support for Python >= 2.7\n\n.. _changed-5:\n\nChanged\n~~~~~~~\n\n- Do not download CloudTrail Logs from S3 if they already exist in the\n target folder (#9)\n- Removed dependency on fork of the awacs-library to simplify\n installation and development\n\n.. _fixed-7:\n\nFixed\n~~~~~\n\n- Bug that led to policy-statements with the same set of actions not\n being combined properly in some cases (#7)\n\n.. _section-14:\n\n0.2.0\n-----\n\n.. _added-8:\n\nAdded\n~~~~~\n\n- Basic filtering for role-arns when generating policy (#3)\n\n.. _section-15:\n\n0.1.0\n-----\n\n*Initial Release*\n\n.. _added-9:\n\nAdded\n~~~~~\n\n- Basic feature to download CloudTrail Logs from S3 for certain\n accounts and timeframe\n- Basic feature to generate IAM Policies from a set of downloaded\n CloudTrail logs\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A command-line tool to get valuable information out of AWS CloudTrail",
"version": "0.8.1",
"split_keywords": [
"aws",
"cloud",
"iam",
"cloudtrail",
"trailscraper"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2fdbcf1305ebbbedf3a32c18af45fb56d9d3d2471ea37498f6f49addd132c992",
"md5": "113ee91a6ec52fb2c987bd2cc9c0b46d",
"sha256": "c7c56dd677f1df2505dbd91ed9f0a0df37f22636df13539c9568b98e05ffc40e"
},
"downloads": -1,
"filename": "trailscraper-0.8.1-py2.py3-none-any.whl",
"has_sig": true,
"md5_digest": "113ee91a6ec52fb2c987bd2cc9c0b46d",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 45760,
"upload_time": "2023-01-29T09:27:45",
"upload_time_iso_8601": "2023-01-29T09:27:45.709891Z",
"url": "https://files.pythonhosted.org/packages/2f/db/cf1305ebbbedf3a32c18af45fb56d9d3d2471ea37498f6f49addd132c992/trailscraper-0.8.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bc9bf425ff02c84a16e434526d3ffe7abfc50589f46a5efe9b02cfd09bec698e",
"md5": "4f72a82e610fcc5ac24aa8100cba621b",
"sha256": "fe0f7970554a7100be6a4dc6ecce0ce0f4a5a3337a689e7035df7ac3c37ec21a"
},
"downloads": -1,
"filename": "trailscraper-0.8.1.tar.gz",
"has_sig": true,
"md5_digest": "4f72a82e610fcc5ac24aa8100cba621b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 49050,
"upload_time": "2023-01-29T09:27:48",
"upload_time_iso_8601": "2023-01-29T09:27:48.207474Z",
"url": "https://files.pythonhosted.org/packages/bc/9b/f425ff02c84a16e434526d3ffe7abfc50589f46a5efe9b02cfd09bec698e/trailscraper-0.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-29 09:27:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "flosell",
"github_project": "trailscraper",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "trailscraper"
}
JSON
