Aegea: Amazon Web Services Operator Interface
=============================================
*Aegea* is a command line interface (CLI) that provides a set of essential commands and terminal dashboards for
operators of Amazon Web Services (AWS) accounts. Aegea lets you build AMIs and Docker images using the
`cloud-init <http://cloudinit.readthedocs.io/>`_ config management package, manage config roles, launch and monitor
instances and services, and manage AWS resources including ELB, RDS, and AWS Batch. Aegea is designed to be used in
conjunction with the existing functionality of the `AWS CLI <https://aws.amazon.com/cli/>`_ and
`boto3 <https://boto3.readthedocs.io/>`_. Aegea can be used in conjunction with tools like
`MiniWDL <https://github.com/chanzuckerberg/miniwdl>`_ to orchestrate large-scale MLops and genomics workflows.
Installation
~~~~~~~~~~~~
::
pip install aegea
Run ``aws configure`` to configure `IAM <https://aws.amazon.com/iam/>`_ access credentials that will be used by the
``aws`` and ``aegea`` commands. You can create a new IAM key at https://console.aws.amazon.com/iam/home#/users. See the
`AWS CLI documentation <https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html>`_ for more details.
Aegea commands
~~~~~~~~~~~~~~
Below are some highlights from Aegea's suite of commands. Run ``aegea --help`` to see the full list of commands.
+----------------------------+-----------------------------------------------------------------------------------------+
| Command | Key functionality |
+============================+=========================================================================================+
| `aegea ls` | List running EC2 instances |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea launch` | Launch EC2 instances and specify options such as spot tenancy, AMI, instance type, etc. |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea ssh`, `aegea scp` | Connect to running instances, transfer files using AWS Systems Manager or other options |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea batch` | Submit, manage and monitor AWS Batch jobs |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea ecs` | Monitor ECS clusters and run Fargate tasks |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea build-ami` | Build EC2 AMIs using cloud-init configuration scripts |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea build-docker-image` | Build AWS ECR docker images using Dockerfiles or cloud-init scripts |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea logs` | Download AWS CloudWatch Logs contents using S3 export |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea grep` | Query AWS CloudWatch Logs contents using CloudWatch Logs Insights |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea cost` | List AWS cost reports generated by AWS Cost Explorer |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea secrets` | List and manage secrets stored in AWS Secrets Manager |
+----------------------------+-----------------------------------------------------------------------------------------+
| `aegea cloudtrail` | Query AWS CloudTrail events to audit the security of your AWS account |
+----------------------------+-----------------------------------------------------------------------------------------+
Aegea Launch
------------
The ``aegea launch`` command launches EC2 instances. It has integrated support for AWS SSM/EC2 Instance Connect, runtime
cloud-init configuration, automatic location of Aegea-built AMIs or up-to-date Ubuntu or Amazon Linux AMIs, automatic
storage configuration, and other options.
Aegea SSH
---------
The ``aegea ssh`` command (and its variant ``aegea scp``) is an SSH configuration wrapper that integrates with the
`AWS Systems Manager <https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html>`_ and
`EC2 Instance Connect <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-using-EC2-Instance-Connect.html>`_
to provide SSH connectivity to your instances without opening any inbound ports (if your instance OS is not configured
with SSM, use ``aegea ssh --no-ssm`` to open a direct connection). ``aegea ssh`` securely resolves instance addresses by
name and pre-fetches their public keys without the Trust-On-First-Use requirement.
Visual Studio Code Remote Development integration
'''''''''''''''''''''''''''''''''''''''''''''''''
`Visual Studio Code <https://code.visualstudio.com/>`_ provides a
`remote development extension pack <https://code.visualstudio.com/docs/remote/remote-overview>`_ which allows your
VSCode IDE to connect remotely to AWS instances. To use VSCode with aegea ssh, open the Remote.SSH plugin settings
(command palette > Remote-SSH: Settings) and enter the path to the ``aegea-ssh`` executable under the "Remote.SSH: Path"
setting (for example, ``/usr/local/bin/aegea-ssh``). After doing this, you can use the "Remote-SSH: Connect to Host"
palette command to connect to any aegea-launched instance by name or ID (for example, ``aegea launch my-test-instance``
followed by entering **my-test-instance** in the **Connect to Host** dialog). Note that VSCode will use the default
AWS CLI profile for authentication.
Aegea Batch
-----------
The `AWS Batch <https://aws.amazon.com/batch>`_ API lets you run non-interactive command line workflows in Docker
containers, managing AWS ECS, Fargate, and EC2/Spot in your account on your behalf. Use the ``aegea batch`` family of
commands to interact with AWS Batch. The key command is ``aegea batch submit`` to submit jobs.
Run ``aegea batch submit --command "echo 'hello world'" --memory 4096 --vcpus 2 --watch``
to run a Batch job that requires 4 GB RAM and 2 cores to be allocated to the Docker container,
and executes the specified command.
You can also use ``aegea batch submit --execute FILE``. This will slurp up FILE (any type of shell script or ELF
executable) and execute it in the job's Docker container.
The concurrency and cost of your Batch jobs is governed by the "Max vCPUs" setting in your compute environment.
To change the capacity or other settings of the default compute environment used by ``aegea batch``, go to
https://console.aws.amazon.com/batch/home?region=us-east-1#/compute-environments, select "aegea_batch", and click
"Edit".
Batch and ECS Fargate
'''''''''''''''''''''
The `ECS Fargate <https://aws.amazon.com/fargate/>`_ API is an interface to the AWS container-based virtualization
platform, `Firecracker <https://github.com/firecracker-microvm/firecracker>`_. ECS Fargate allows you to run workloads
in fully managed containers: no instances run in your account; you are billed by the second of container use, and
containers start up within 10 seconds, compared to minutes for EC2 instances.
AWS Batch can run your jobs on either ECS Container Instances (EC2 instances connected to ECS that Batch manages in your
account) or directly in ECS Fargate containers. While Fargate containers are much faster to start, they have
`lower CPU and memory limits <https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html>`_
of 4 CPUs and 30 GB RAM (compared to 96 CPUs and 768 GB RAM on EC2).
By default, ``aegea batch`` will create and use an AWS Batch compute environment and queue that uses ECS Fargate, but
you can control this by setting the ``--compute-type`` option to ``aegea batch create-compute-environment``.
Aegea also supports direct use of ECS Fargate without Batch via the ``aegea ecs run`` command. Most ``aegea batch``
semantics are applicable to ``aegea ecs``, which interacts with ECS via the "one shot"
`ECS RunTask <https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html>`_ API.
Configuration management
~~~~~~~~~~~~~~~~~~~~~~~~
Aegea supports ingesting configuration from a configurable array of sources. Each source is a JSON or YAML file.
Configuration sources that follow the first source update the configuration using recursive dictionary merging. Sources
are enumerated in the following order of priority:
- Command line options (values take priority over all other sources)
- Any sources listed in the colon-delimited variable ``AEGEA_CONFIG_FILE``
- User configuration source, ``~/.config/aegea/config.yml``
- Site-wide configuration source, ``/etc/aegea/config.yml``
- Configuration defaults from `base_config.yml <https://github.com/kislyuk/aegea/blob/develop/aegea/base_config.yml>`_
**Array merge operators**: When loading a chain of configuration sources, Aegea uses recursive dictionary merging to
combine the sources. Additionally, when the original config value is a list, Aegea supports array manipulation
operators, which let you extend and modify arrays defined in underlying configurations. For example, to enable full S3
access for all instances launched with `aegea launch`, add the following to ``~/.config/aegea/config.yml``::
launch:
iam_policies:
$append: AmazonS3FullAccess
To enable S3 and SQS access, and also list the instance private IP address and availability zone in ``aegea ls``,
add the following::
launch:
iam_policies:
$extend:
- AmazonS3FullAccess
- AmazonSQSFullAccess
ls:
columns:
$extend:
- private_ip_address
- placement.AvailabilityZone
For a listing of available configuration parameters that can be set, run ``aegea configure`` or see
https://github.com/kislyuk/aegea/blob/develop/aegea/base_config.yml. See
https://github.com/kislyuk/tweak#array-merge-operators for a formal description of the array merge operators.
Building AMIs and Docker images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Aegea includes a lightweight configuration management system for building machine images based on
`cloud-init <http://cloudinit.readthedocs.io/>`_ (both Docker images and AMIs are supported).
.. image:: https://github.com/kislyuk/aegea/workflows/Tests/badge.svg
:target: https://github.com/kislyuk/aegea/actions
.. image:: https://img.shields.io/pypi/v/aegea.svg
:target: https://pypi.python.org/pypi/aegea
.. image:: https://img.shields.io/pypi/l/aegea.svg
:target: https://pypi.python.org/pypi/aegea
.. image:: https://codecov.io/gh/kislyuk/aegea/branch/master/graph/badge.svg
:target: https://codecov.io/gh/kislyuk/aegea
Raw data
{
"_id": null,
"home_page": "https://github.com/kislyuk/aegea",
"name": "aegea",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Andrey Kislyuk",
"author_email": "kislyuk@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a3/c9/0dcbbc0e19b77a7b103e80e209257e95c5bfa9dd37d4696155b69359819d/aegea-4.5.0.tar.gz",
"platform": "MacOS X",
"description": "Aegea: Amazon Web Services Operator Interface\n=============================================\n\n*Aegea* is a command line interface (CLI) that provides a set of essential commands and terminal dashboards for\noperators of Amazon Web Services (AWS) accounts. Aegea lets you build AMIs and Docker images using the\n`cloud-init <http://cloudinit.readthedocs.io/>`_ config management package, manage config roles, launch and monitor\ninstances and services, and manage AWS resources including ELB, RDS, and AWS Batch. Aegea is designed to be used in\nconjunction with the existing functionality of the `AWS CLI <https://aws.amazon.com/cli/>`_ and\n`boto3 <https://boto3.readthedocs.io/>`_. Aegea can be used in conjunction with tools like\n`MiniWDL <https://github.com/chanzuckerberg/miniwdl>`_ to orchestrate large-scale MLops and genomics workflows.\n\nInstallation\n~~~~~~~~~~~~\n::\n\n pip install aegea\n\nRun ``aws configure`` to configure `IAM <https://aws.amazon.com/iam/>`_ access credentials that will be used by the\n``aws`` and ``aegea`` commands. You can create a new IAM key at https://console.aws.amazon.com/iam/home#/users. See the\n`AWS CLI documentation <https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html>`_ for more details.\n\nAegea commands\n~~~~~~~~~~~~~~\nBelow are some highlights from Aegea's suite of commands. Run ``aegea --help`` to see the full list of commands.\n\n+----------------------------+-----------------------------------------------------------------------------------------+\n| Command | Key functionality |\n+============================+=========================================================================================+\n| `aegea ls` | List running EC2 instances |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea launch` | Launch EC2 instances and specify options such as spot tenancy, AMI, instance type, etc. |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea ssh`, `aegea scp` | Connect to running instances, transfer files using AWS Systems Manager or other options |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea batch` | Submit, manage and monitor AWS Batch jobs |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea ecs` | Monitor ECS clusters and run Fargate tasks |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea build-ami` | Build EC2 AMIs using cloud-init configuration scripts |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea build-docker-image` | Build AWS ECR docker images using Dockerfiles or cloud-init scripts |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea logs` | Download AWS CloudWatch Logs contents using S3 export |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea grep` | Query AWS CloudWatch Logs contents using CloudWatch Logs Insights |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea cost` | List AWS cost reports generated by AWS Cost Explorer |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea secrets` | List and manage secrets stored in AWS Secrets Manager |\n+----------------------------+-----------------------------------------------------------------------------------------+\n| `aegea cloudtrail` | Query AWS CloudTrail events to audit the security of your AWS account |\n+----------------------------+-----------------------------------------------------------------------------------------+\n\nAegea Launch\n------------\nThe ``aegea launch`` command launches EC2 instances. It has integrated support for AWS SSM/EC2 Instance Connect, runtime\ncloud-init configuration, automatic location of Aegea-built AMIs or up-to-date Ubuntu or Amazon Linux AMIs, automatic\nstorage configuration, and other options.\n\nAegea SSH\n---------\nThe ``aegea ssh`` command (and its variant ``aegea scp``) is an SSH configuration wrapper that integrates with the\n`AWS Systems Manager <https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html>`_ and\n`EC2 Instance Connect <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-using-EC2-Instance-Connect.html>`_\nto provide SSH connectivity to your instances without opening any inbound ports (if your instance OS is not configured\nwith SSM, use ``aegea ssh --no-ssm`` to open a direct connection). ``aegea ssh`` securely resolves instance addresses by\nname and pre-fetches their public keys without the Trust-On-First-Use requirement.\n\nVisual Studio Code Remote Development integration\n'''''''''''''''''''''''''''''''''''''''''''''''''\n`Visual Studio Code <https://code.visualstudio.com/>`_ provides a\n`remote development extension pack <https://code.visualstudio.com/docs/remote/remote-overview>`_ which allows your\nVSCode IDE to connect remotely to AWS instances. To use VSCode with aegea ssh, open the Remote.SSH plugin settings\n(command palette > Remote-SSH: Settings) and enter the path to the ``aegea-ssh`` executable under the \"Remote.SSH: Path\"\nsetting (for example, ``/usr/local/bin/aegea-ssh``). After doing this, you can use the \"Remote-SSH: Connect to Host\"\npalette command to connect to any aegea-launched instance by name or ID (for example, ``aegea launch my-test-instance``\nfollowed by entering **my-test-instance** in the **Connect to Host** dialog). Note that VSCode will use the default\nAWS CLI profile for authentication.\n\nAegea Batch\n-----------\nThe `AWS Batch <https://aws.amazon.com/batch>`_ API lets you run non-interactive command line workflows in Docker\ncontainers, managing AWS ECS, Fargate, and EC2/Spot in your account on your behalf. Use the ``aegea batch`` family of\ncommands to interact with AWS Batch. The key command is ``aegea batch submit`` to submit jobs.\n\nRun ``aegea batch submit --command \"echo 'hello world'\" --memory 4096 --vcpus 2 --watch``\nto run a Batch job that requires 4 GB RAM and 2 cores to be allocated to the Docker container,\nand executes the specified command.\n\nYou can also use ``aegea batch submit --execute FILE``. This will slurp up FILE (any type of shell script or ELF\nexecutable) and execute it in the job's Docker container.\n\nThe concurrency and cost of your Batch jobs is governed by the \"Max vCPUs\" setting in your compute environment.\nTo change the capacity or other settings of the default compute environment used by ``aegea batch``, go to\nhttps://console.aws.amazon.com/batch/home?region=us-east-1#/compute-environments, select \"aegea_batch\", and click\n\"Edit\".\n\nBatch and ECS Fargate\n'''''''''''''''''''''\nThe `ECS Fargate <https://aws.amazon.com/fargate/>`_ API is an interface to the AWS container-based virtualization\nplatform, `Firecracker <https://github.com/firecracker-microvm/firecracker>`_. ECS Fargate allows you to run workloads\nin fully managed containers: no instances run in your account; you are billed by the second of container use, and\ncontainers start up within 10 seconds, compared to minutes for EC2 instances.\n\nAWS Batch can run your jobs on either ECS Container Instances (EC2 instances connected to ECS that Batch manages in your\naccount) or directly in ECS Fargate containers. While Fargate containers are much faster to start, they have\n`lower CPU and memory limits <https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html>`_\nof 4 CPUs and 30 GB RAM (compared to 96 CPUs and 768 GB RAM on EC2).\n\nBy default, ``aegea batch`` will create and use an AWS Batch compute environment and queue that uses ECS Fargate, but\nyou can control this by setting the ``--compute-type`` option to ``aegea batch create-compute-environment``.\n\nAegea also supports direct use of ECS Fargate without Batch via the ``aegea ecs run`` command. Most ``aegea batch``\nsemantics are applicable to ``aegea ecs``, which interacts with ECS via the \"one shot\"\n`ECS RunTask <https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html>`_ API.\n\nConfiguration management\n~~~~~~~~~~~~~~~~~~~~~~~~\nAegea supports ingesting configuration from a configurable array of sources. Each source is a JSON or YAML file.\nConfiguration sources that follow the first source update the configuration using recursive dictionary merging. Sources\nare enumerated in the following order of priority:\n\n- Command line options (values take priority over all other sources)\n- Any sources listed in the colon-delimited variable ``AEGEA_CONFIG_FILE``\n- User configuration source, ``~/.config/aegea/config.yml``\n- Site-wide configuration source, ``/etc/aegea/config.yml``\n- Configuration defaults from `base_config.yml <https://github.com/kislyuk/aegea/blob/develop/aegea/base_config.yml>`_\n\n**Array merge operators**: When loading a chain of configuration sources, Aegea uses recursive dictionary merging to\ncombine the sources. Additionally, when the original config value is a list, Aegea supports array manipulation\noperators, which let you extend and modify arrays defined in underlying configurations. For example, to enable full S3\naccess for all instances launched with `aegea launch`, add the following to ``~/.config/aegea/config.yml``::\n\n launch:\n iam_policies:\n $append: AmazonS3FullAccess\n\nTo enable S3 and SQS access, and also list the instance private IP address and availability zone in ``aegea ls``,\nadd the following::\n\n launch:\n iam_policies:\n $extend:\n - AmazonS3FullAccess\n - AmazonSQSFullAccess\n ls:\n columns:\n $extend:\n - private_ip_address\n - placement.AvailabilityZone\n\nFor a listing of available configuration parameters that can be set, run ``aegea configure`` or see\nhttps://github.com/kislyuk/aegea/blob/develop/aegea/base_config.yml. See\nhttps://github.com/kislyuk/tweak#array-merge-operators for a formal description of the array merge operators.\n\nBuilding AMIs and Docker images\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nAegea includes a lightweight configuration management system for building machine images based on\n`cloud-init <http://cloudinit.readthedocs.io/>`_ (both Docker images and AMIs are supported).\n\n.. image:: https://github.com/kislyuk/aegea/workflows/Tests/badge.svg\n :target: https://github.com/kislyuk/aegea/actions\n.. image:: https://img.shields.io/pypi/v/aegea.svg\n :target: https://pypi.python.org/pypi/aegea\n.. image:: https://img.shields.io/pypi/l/aegea.svg\n :target: https://pypi.python.org/pypi/aegea\n.. image:: https://codecov.io/gh/kislyuk/aegea/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/kislyuk/aegea\n",
"bugtrack_url": null,
"license": "Apache Software License",
"summary": "Amazon Web Services Operator Interface",
"version": "4.5.0",
"project_urls": {
"Homepage": "https://github.com/kislyuk/aegea"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9506a1afc119b36a769f1eeae0cf3bbfd1f9c4de4db6ee2c430fed5100437abd",
"md5": "39e9d60d7db9e8aea266756d6bfc32b3",
"sha256": "0cc9af18f3d687b85fbc8dad33534f048fe3a6050c77e4002b9ba6bc4d342b3b"
},
"downloads": -1,
"filename": "aegea-4.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "39e9d60d7db9e8aea266756d6bfc32b3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 181487,
"upload_time": "2024-07-31T05:58:24",
"upload_time_iso_8601": "2024-07-31T05:58:24.921760Z",
"url": "https://files.pythonhosted.org/packages/95/06/a1afc119b36a769f1eeae0cf3bbfd1f9c4de4db6ee2c430fed5100437abd/aegea-4.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a3c90dcbbc0e19b77a7b103e80e209257e95c5bfa9dd37d4696155b69359819d",
"md5": "d5a04caadd09b9bf33375abb1e85abd2",
"sha256": "d43cdbb167a8ea8933d1a7573457ed1f1b20f54694fef7fb2eb1b5f45fc98f5d"
},
"downloads": -1,
"filename": "aegea-4.5.0.tar.gz",
"has_sig": false,
"md5_digest": "d5a04caadd09b9bf33375abb1e85abd2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 159910,
"upload_time": "2024-07-31T05:58:28",
"upload_time_iso_8601": "2024-07-31T05:58:28.170083Z",
"url": "https://files.pythonhosted.org/packages/a3/c9/0dcbbc0e19b77a7b103e80e209257e95c5bfa9dd37d4696155b69359819d/aegea-4.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-31 05:58:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kislyuk",
"github_project": "aegea",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "aegea"
}
JSON
