# c7n-mailer: Custodian Mailer
% [comment]: # ( !!! IMPORTANT !!! )
% [comment]: # (This file is moved during document generation.)
% [comment]: # (Only edit the original document at ./tools/c7n_mailer/README.md)
A mailer implementation for Custodian. Outbound mail delivery is still somewhat
organization-specific, so this at the moment serves primarily as an example
implementation.
> The Cloud Custodian Mailer can now be easily run in a Docker container. Click [here](https://hub.docker.com/r/cloudcustodian/mailer) for details.
## Message Relay
Custodian Mailer subscribes to an SQS queue, looks up users, and sends emails
via SES, Slack messages, and/or notifications to DataDog or Splunk. Custodian Lambda and instance policies can send messages to the SQS queue. SQS queues
should be cross-account enabled for sending between accounts.
## Tutorial
Our goal in starting out with the Custodian mailer is to install the mailer,
and run a policy that triggers an email to your inbox.
1. [Install](#developer-install-os-x-el-capitan) the mailer on your laptop (if you are not running as a [Docker container](https://hub.docker.com/r/cloudcustodian/mailer)
- or use `pip install c7n-mailer`
2. In your text editor, create a `mailer.yml` file to hold your mailer config.
3. In the AWS console, create a new standard SQS queue (quick create is fine).
Copy the queue URL to `queue_url` in `mailer.yml`.
4. In AWS, locate or create a role that has read access to the queue. Grab the
role ARN and set it as `role` in `mailer.yml`.
There are different notification endpoints options, you can combine both.
### Email:
Make sure your email address is verified in SES, and set it as
`from_address` in `mailer.yml`. By default SES is in sandbox mode where you
must
[verify](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html)
every individual recipient of emails. If need be, make an AWS support ticket to
be taken out of SES sandbox mode.
Your `mailer.yml` should now look something like this:
```yaml
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
from_address: you@example.com
```
You can also set `region` if you are in a region other than `us-east-1` as well as `lambda_tags` to give the mailer tags.
```yaml
region: us-east-2
lambda_tags:
owner: ops
```
Now let's make a Custodian policy to populate your mailer queue. Create a
`test-policy.yml` file with this content (update `to` and `queue` to match your
environment)
```yaml
policies:
- name: c7n-mailer-test
resource: sqs
filters:
- "tag:MailerTest": absent
actions:
- type: notify
template: default
priority_header: '2'
subject: testing the c7n mailer
to:
- you@example.com
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
```
### DataDog:
The standard way to do a DataDog integration is use the
c7n integration with AWS CloudWatch and use the
[DataDog integration with AWS](https://docs.datadoghq.com/integrations/amazon_web_services/)
to collect CloudWatch metrics. The mailer/messenger integration is only
for the case you don't want or you can't use AWS CloudWatch, e.g. in Azure or GCP.
Note this integration requires the additional dependency of Datadog Python bindings:
```
pip install datadog
```
Your `mailer.yml` should now look something like this:
```yaml
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
datadog_api_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
datadog_application_key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
```
(Also set `region` if you are in a region other than `us-east-1`.)
Now, let's make a Custodian policy to populate your mailer queue. Create a
`test-policy.yml`:
```yaml
policies:
- name: c7n-mailer-test
resource: ebs
filters:
- Attachments: []
actions:
- type: notify
to:
- datadog://?metric_name=datadog.metric.name&metric_value_tag=Size
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
```
There is a special `to` format that specifies Datadog delivery and it includes the Datadog configuration via URL parameters:
- `metric_name`: is the name of the metrics send to DataDog.
- `metric_value_tag`: by default the metric value to send to DataDog is `1`, but if you want to use one of the tags returned in the policy, you can set it with the attribute `metric_value_tag`. For example, in the `test-policy.yml` policy, the value used is the size of the EBS volume. The value must be a number and it will be transformed to a float value.
### Slack:
The Custodian mailer supports Slack messaging as a separate notification mechanism for the SQS transport method. To enable Slack integration, you must specify a Slack token in the `slack_token` field in your `mailer.yml` file. For example:
```yaml
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
slack_token: xoxo-token123
```
To enable Slack messaging, several unique fields are evaluated in the policy, as shown in the below example:
```
policies:
- name: c7n-mailer-test
resource: ebs
filters:
- Attachments: []
actions:
- type: notify
slack_template: slack
slack_msg_color: danger
to:
- slack://owners
- slack://foo@bar.com
- slack://#custodian-test
- slack://webhook/#c7n-webhook-test
- slack://tag/resource_tag
- https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
```
Slack messages support the use of a unique template field specified in `slack_template`. This field is unique and usage will not break
existing functionality for messages also specifying an email template in the `template` field. This field is optional, however,
and if not specified, the mailer will use the default value `slack_default`.
The unique template field `slack_msg_color` can be used to specify a color
border for the Slack message. This accepts the Slack presets of `danger` (red),
`warning` (yellow) and `good` (green). It can also accept an HTML hex code. See
the [Slack documentation](https://api.slack.com/reference/messaging/attachments#fields)
for details.
**Note:** if you are using a hex color code, it will need to be wrapped in quotes
like so: `slack_msg_color: '#4287f51'`. Otherwise, the YAML interpreter will consider it a
[comment](https://yaml.org/spec/1.2/spec.html#id2780069).
The Slack integration for c7n-mailer supports several flavors of messaging, listed below. These are not mutually exclusive and any combination of the types can be used, but the preferred method is [incoming webhooks](https://api.slack.com/incoming-webhooks).
| Requires `slack_token` | Key | Type | Notes |
|:---------------------------:|:--------------------------------------------------------------------------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| No | `https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX` | string | **(PREFERRED)** Send to an [incoming webhook](https://api.slack.com/incoming-webhooks) (the channel is defined in the webhook) |
| Yes | `slack://owners` | string | Send to the recipient list generated within email delivery logic |
| Yes | `slack://foo@bar.com` | string | Send to the recipient specified by email address foo@bar.com |
| Yes | `slack://#custodian-test` | string | Send to the Slack channel indicated in string, i.e. #custodian-test |
| No | `slack://webhook/#c7n-webhook-test` | string | **(DEPRECATED)** Send to a Slack webhook; appended with the target channel. **IMPORTANT**: *This requires a `slack_webhook` value defined in the `mailer.yml`.* |
| Yes | `slack://tag/resource-tag` | string | Send to target found in resource tag. Example of value in tag: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX |
Slack delivery can also be set via a resource's tag name. For example, using `slack://tag/slack_channel` will look for a tag name of `slack_channel`, and if matched on a resource will deliver the message to the value of that resource's tag:
`slack_channel:https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`
`slack_channel:custodian-test`
`owner:foo@bar`
Delivery via tag has been tested with webhooks, but it should support all delivery methods.
### Splunk HTTP Event Collector (HEC)
The Custodian mailer supports delivery to the HTTP Event Collector (HEC) endpoint of a Splunk instance as a separate notification mechanism for the SES transport method. To enable Splunk HEC integration, you must specify the URL to the HEC endpoint as well as a valid username and token:
```yaml
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
splunk_hec_url: https://http-inputs-foo.splunkcloud.com/services/collector/event
splunk_hec_token: 268b3cc2-f32e-4a19-a1e8-aee08d86ca7f
```
To send events for a policy to the Splunk HEC endpoint, add a ``to`` address notify action specifying the name of the Splunk index to send events to in the form ``splunkhec://indexName``:
```
policies:
- name: c7n-mailer-test
resource: ebs
filters:
- Attachments: []
actions:
- type: notify
to:
- splunkhec://myIndexName
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
```
The ``splunkhec://indexName`` address type can be combined in the same notify action with other destination types (e.g. email, Slack, DataDog, etc).
### Now run:
```
c7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .
```
Note: You can set the profile via environment variable e.g. `export AWS_DEFAULT_PROFILE=foo`
You should see output similar to the following:
```
(env) $ c7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .
DEBUG:custodian.lambda:Created custodian lambda archive size: 3.01mb
2017-01-12 07:55:16,227: custodian.policy:INFO Running policy c7n-mailer-test resource: sqs region:default c7n:0.8.22.0
2017-01-12 07:55:16,229: custodian.policy:INFO policy: c7n-mailer-test resource:sqs has count:1 time:0.00
2017-01-12 07:55:18,017: custodian.actions:INFO sent message:dead-beef policy:c7n-mailer-test template:default count:1
2017-01-12 07:55:18,017: custodian.policy:INFO policy: c7n-mailer-test action: notify resources: 1 execution_time: 1.79
(env) $
```
Check the AWS console for a new Lambda named `cloud-custodian-mailer`. The
mailer runs every five minutes, so wait a bit and then look for an email in
your inbox. If it doesn't appear, look in the lambda's logs for debugging
information. If it does, congratulations! You are off and running with the
Custodian mailer.
## Usage & Configuration
Once [installed](#developer-install-os-x-el-capitan) you should have a
`c7n-mailer` executable on your path:
```
(env) $ c7n-mailer
usage: c7n-mailer [-h] -c CONFIG
c7n-mailer: error: argument -c/--config is required
(env) $
```
Fundamentally, what `c7n-mailer` does is it deploys a Lambda function (using
[Mu](http://cloudcustodian.io/docs/policy/mu.html)) based on
the configuration you specify in a YAML file. Here is [the
schema](https://github.com/cloud-custodian/cloud-custodian/blob/18d4247e913d54f36a078ed61386695362a3b10d/tools/c7n_mailer/c7n_mailer/cli.py#L43) to which the file must conform
and here is a description of the options:
| Required? | Key | Type | Notes |
|:---------:|:--------------------------|:-----------------|:------------------------------------------------------------------|
| ✅ | `queue_url` | string | the queue to listen to for messages |
| | `from_address` | string | default from address |
| | `endpoint_url` | string | SQS API URL (for use with VPC Endpoints) |
| | `contact_tags` | array of strings | tags that we should look at for address information |
| | `email_base_url` | string | Base URL to construct a valid email address from a resource owner |
| | `additional_email_headers`| object | Additional email headers. |
| | `org_domain` | string | Domain to add to AWS usernames to construct email addresses. |
### Standard Lambda Function Config
| Required? | Key | Type |
|:---------:|:---------------------|:-----------------|
| ✅ | `role` | string |
| | `dead_letter_config` | object |
| | `memory` | integer |
| | `region` | string |
| | `runtime` | string |
| | `security_groups` | array of strings |
| | `subnets` | array of strings |
| | `timeout` | integer |
| | `lambda_name` | string |
| | `lambda_description` | string |
| | `lambda_tags` | object |
| | `lambda_schedule` | string |
### Standard Azure Functions Config
| Required? | Key | Type | Notes |
|:---------:|:----------------------|:-------|:---------------------------------------------------------------------------------------|
| | `function_properties` | object | Contains `appInsights`, `storageAccount` and `servicePlan` objects |
| | `appInsights` | object | Contains `name`, `location` and `resourceGroupName` properties |
| | `storageAccount` | object | Contains `name`, `location` and `resourceGroupName` properties |
| | `servicePlan` | object | Contains `name`, `location`, `resourceGroupName`, `skuTier` and `skuName` properties |
| | `name` | string | |
| | `location` | string | Default: `west us 2` |
| | `resourceGroupName` | string | Default `cloud-custodian` |
| | `skuTier` | string | Default: `Basic` |
| | `skuName` | string | Default: `B1` |
### Mailer Infrastructure Config
| Required? | Key | Type | Notes |
|:---------:|:----------------------------|:--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | `cache_engine` | string | cache engine; either sqlite or redis |
| | `cross_accounts` | object | account to assume back into for sending to SNS topics |
| | `debug` | boolean | debug on/off |
| | `ldap_bind_dn` | string | eg: ou=people,dc=example,dc=com |
| | `ldap_bind_user` | string | eg: FOO\\BAR |
| | `ldap_bind_password` | secured string | ldap bind password |
| | `ldap_bind_password_in_kms` | boolean | defaults to true, most people (except capone) want to set this to false. If set to true, make sure `ldap_bind_password` contains your KMS encrypted ldap bind password as a base64-encoded string. |
| | `ldap_email_attribute` | string | |
| | `ldap_email_key` | string | eg 'mail' |
| | `ldap_manager_attribute` | string | eg 'manager' |
| | `ldap_uid_attribute` | string | |
| | `ldap_uid_regex` | string | |
| | `ldap_uid_tags` | string | |
| | `ldap_uri` | string | eg 'ldaps://example.com:636' |
| | `redis_host` | string | redis host if cache_engine == redis |
| | `redis_port` | integer | redis port, default: 6369 |
| | `ses_region` | string | AWS region that handles SES API calls |
| | `ses_role` | string | ARN of the role to assume to send email with SES |
### SMTP Config
| Required? | Key | Type | Notes |
|:---------:|:----------------|:-----------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | `smtp_server` | string | to configure your lambda role to talk to smtpd in your private vpc, see [here](https://docs.aws.amazon.com/lambda/latest/dg/vpc.html) | |
| | `smtp_port` | integer | smtp port (default is 25) |
| | `smtp_ssl` | boolean | this defaults to True |
| | `smtp_username` | string | |
| | `smtp_password` | secured string | |
If `smtp_server` is unset, `c7n_mailer` will use AWS SES or Azure SendGrid.
#### DataDog Config
| Required? | Key | Type | Notes |
|:---------:|:--------------------------|:-------|:-------------------------|
| | `datadog_api_key` | string | DataDog API key. |
| | `datadog_application_key` | string | Datadog application key. |
These fields are not necessary if c7n_mailer is run in a instance/lambda/etc with the DataDog agent.
### Slack Config
| Required? | Key | Type | Notes |
|:---------:|:--------------|:-------|:----------------|
| | `slack_token` | string | Slack API token |
### SendGrid Config
| Required? | Key | Type | Notes |
|:---------:|:-------------------|:---------------|:-------------------|
| | `sendgrid_api_key` | secured string | SendGrid API token |
### Microsoft Graph Config
| Required? | Key | Type | Notes |
|:---------:|:--------------------------|:---------------|:-------------------------|
| | `graph_token_endpoint` | string | Graph token endpoint |
| | `graph_sendmail_endpoint` | string | Graph sendmail endpoint |
| | `graph_client_id` | string | Client ID |
| | `graph_client_secret` | secured string | Client secret |
### Splunk HEC Config
The following configuration items are *all* optional. The ones marked "Required for Splunk" are only required if you're sending notifications to ``splunkhec://`` destinations.
| Required for Splunk? | Key | Type | Notes |
|:--------------------:|:------------------------|:-----------------|:-----------------------------------------------------------------------------------------------------------------------------------|
| ✅ | `splunk_hec_url` | string | URL to your Splunk HTTP Event Collector endpoint |
| ✅ | `splunk_hec_token` | string | Splunk HEC authentication token for specified username |
| | `splunk_remove_paths` | array of strings | List of [RFC6901](http://tools.ietf.org/html/rfc6901) JSON Pointers to remove from the event, if present, before sending to Splunk |
| | `splunk_actions_list` | boolean | If true, add an `actions` list to the top-level message sent to Splunk, containing the names of all non-notify actions taken |
| | `splunk_max_attempts` | integer | Maximum number of times to try POSTing data to Splunk HEC (default 4) |
| | `splunk_hec_max_length` | integer | Maximum data length that Splunk HEC accepts; an error will be logged for any message sent over this length |
| | `splunk_hec_sourcetype` | string | Configure sourcetype of the payload sent to Splunk HEC. (default is '_json') |
### SDK Config
| Required? | Key | Type | Notes |
|:---------:|:--------------|:-------|:------|
| | `http_proxy` | string | |
| | `https_proxy` | string | |
| | `profile` | string | |
### Secured String
In order to ensure sensitive data is not stored plaintext in a policy, `c7n-mailer` supports secured
strings. You can treat it as a regular `string` or use `secured string` features.
#### AWS
You can use KMS to encrypt your secrets and use encrypted secret in mailer policy.
Custodian tries to decrypt the string using KMS, if it fails c7n treats it as a plaintext secret.
```yaml
plaintext_secret: <raw_secret>
secured_string: <encrypted_secret>
```
#### Azure
You can store your secrets in Azure Key Vault secrets and reference them from the policy.
```yaml
plaintext_secret: <raw_secret>
secured_string:
type: azure.keyvault
secret: https://your-vault.vault.azure.net/secrets/your-secret
```
Note: `secrets.get` permission on the KeyVault for the Service Principal is required.
#### GCP
You can store your secrets as GCP Secret Manager secrets and reference them from the policy.
```yaml
plaintext_secret: <raw_secret>
secured_string:
type: gcp.secretmanager
secret: projects/12345678912/secrets/your-secret
```
An example of an SMTP password set as a secured string:
```yaml
smtp_password:
type: gcp.secretmanager
secret: projects/59808015552/secrets/smtp_pw
```
Note: If you do not specify a version, `/versions/latest` will be appended to your secret location.
## Configuring a policy to send email
Outbound email can be added to any policy by including the `notify` action.
```yaml
policies:
- name: bad-apples
resource: asg
filters:
- "tag:ASV": absent
actions:
- type: notify
template: default
template_format: 'html'
priority_header: '1'
subject: fix your tags
to:
- resource-owner
owner_absent_contact:
- foo@example.com
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/80101010101/cloud-custodian-message-relay
```
So breaking it down, you add an action of type `notify`. You can specify a
template that's used to format the email; customizing templates is described
[below](#writing-an-email-template).
The `to` list specifies the intended recipient for the email. You can specify
either an email address, an SNS topic, a Datadog Metric, or a special value. The special values
are either
- `resource-owner`, in which case the email will be sent to the listed
`OwnerContact` tag on the resource that matched the policy, or
- `event-owner` for push-based/realtime policies that will send to the user
that was responsible for the underlying event.
- `priority_header` to indicate the importance of an email with [headers](https://www.chilkatsoft.com/p/p_471.asp). Different emails clients will display stars, exclamation points or flags depending on the value. Should be an string from 1 to 5.
Both of these special values are best effort, i.e., if no `OwnerContact` tag is
specified then `resource-owner` email will not be delivered, and in the case of
`event-owner` an instance role or system account will not result in an email.
The optional `owner_absent_contact` list specifies email addresses to notify only if
the `resource-owner` special option was unable to find any matching owner contact
tags.
In addition, you may choose to use a custom tag instead of the default `OwnerContact`. In order to configure this, the mailer.yaml must be modified to include the contact_tags and the custom tag. The `resource-owner` will now email the custom tag instead of `OwnerContact`.
```yaml
contact_tags:
- "custom_tag"
```
For reference purposes, the JSON Schema of the `notify` action:
```json
{
"type": "object",
"required": ["type", "transport", "to"],
"properties": {
"type": {"enum": ["notify"]},
"to": {"type": "array", "items": {"type": "string"}},
"owner_absent_contact": {"type": "array", "items": {"type": "string"}},
"subject": {"type": "string"},
"priority_header": {"type": "string"},
"template": {"type": "string"},
"transport": {
"type": "object",
"required": ["type", "queue"],
"properties": {
"queue": {"type": "string"},
"region": {"type": "string"},
"type": {"enum": ["sqs"]}
}
}
}
}
```
## Using on Azure
Requires:
- `c7n_azure` package. See [Installing Azure Plugin](https://cloudcustodian.io/docs/azure/gettingstarted.html#azure-install-cc)
- SendGrid account. See [Using SendGrid with Azure](https://docs.microsoft.com/en-us/azure/sendgrid-dotnet-how-to-send-email)
- [Azure Storage Queue](https://azure.microsoft.com/en-us/services/storage/queues/)
The mailer supports an Azure Storage Queue transport and SendGrid delivery on Azure.
Configuration for this scenario requires only minor changes from AWS deployments.
You will need to grant `Storage Queue Data Contributor` role on the Queue for the identity
mailer is running under.
The notify action in your policy will reflect transport type `asq` with the URL
to an Azure Storage Queue. For example:
```yaml
policies:
- name: azure-notify
resource: azure.resourcegroup
description: send a message to a mailer instance
actions:
- type: notify
template: default
priority_header: '2'
subject: Hello from C7N Mailer
to:
- you@youremail.com
transport:
type: asq
queue: https://storageaccount.queue.core.windows.net/queuename
```
In your mailer configuration, you'll need to provide your SendGrid API key as well as
prefix your queue URL with `asq://` to let mailer know what type of queue it is:
```yaml
queue_url: asq://storageaccount.queue.core.windows.net/queuename
from_address: you@youremail.com
sendgrid_api_key: SENDGRID_API_KEY
```
The mailer will transmit all messages found on the queue on each execution, and will retry
sending 3 times in the event of a failure calling SendGrid. After the retries the queue
message will be discarded.
In addition, SendGrid delivery on Azure supports using resource tags to send emails. For example, in the `to` field:
```yaml
to:
- tag:OwnerEmail
```
This will find the email address associated with the resource's `OwnerEmail` tag, and send an email to the specified address.
If no tag is found, or the associated email address is invalid, no email will be sent.
### Deploying Azure Functions
The `--update-lambda` CLI option will also deploy Azure Functions if you have an Azure
mailer configuration.
`c7n-mailer --config mailer.yml --update-lambda`
where a simple `mailer.yml` using Consumption functions may look like:
```yaml
queue_url: asq://storage.queue.core.windows.net/custodian
from_address: foo@mail.com
sendgrid_api_key: <key>
function_properties:
servicePlan:
name: 'testmailer1'
```
### Configuring Function Identity
You can configure the service principal used for api calls made by the
mailer azure function by specifying an identity configuration under
function properties. Mailer supports User Assigned Identities, System
Managed Identities, defaulting to an embedding of the cli user's
service principals credentials.
When specifying a user assigned identity, unlike in a custodian
function policy where simply providing an name is sufficient, the
uuid/id and client id of the identity must be provided. You can
retrieve this information on the cli using the `az identity list`.
```yaml
function_properties:
identity:
type: UserAssigned
id: "/subscriptions/333fd504-7f11-2270-88c8-7325a27f7222/resourcegroups/c7n/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mailer"
client_id: "b9cb06fa-dfb8-4342-add3-aab5acb2abbc"
```
A system managed identity can also be used, and the Azure platform will
create an identity when the function is provisoned, however the function's identity
then needs to be retrieved and mapped to rbac permissions post provisioning, this
user management activity must be performed manually.
```yaml
function_properties:
identity:
type: SystemAssigned
```
## Using on GCP
Requires:
- `c7n_gcp` package. See [GCP Getting Started](https://cloudcustodian.io/docs/gcp/gettingstarted.html)
- `google-cloud-secret-manager` package, for pulling in secured string values.
- A working SMTP Account.
- [GCP Pubsub Subscription](https://cloud.google.com/pubsub/docs/)
The mailer supports GCP Pubsub transports and SMTP/Email delivery, as well as Datadog and Splunk.
Configuration for this scenario requires only minor changes from AWS deployments.
The notify action in your policy will reflect transport type `projects` with the URL
to a GCP Pub/Sub Topic. For example:
```yaml
policies:
- name: gcp-notify
resource: gcp.compute
description: example policy
actions:
- type: notify
template: default
priority_header: '2'
subject: Hello from C7N Mailer
to:
- you@youremail.com
transport:
type: pubsub
topic: projects/myproject/topics/mytopic
```
In your mailer configuration, you'll need to provide your SMTP account information
as well as your topic subscription path in the queue_url variable. Please note that the
subscription you specify should be subscribed to the topic you assign in your policies'
notify action for GCP resources.
```yaml
queue_url: projects/myproject/subscriptions/mysubscription
from_address: you@youremail.com
# c7n-mailer currently requires a role be present, even if it's empty
role: ""
smtp_server: my.smtp.add.ress
smtp_port: 25
smtp_ssl: true
smtp_username: smtpuser
smtp_password:
type: gcp.secretmanager
secret: projects/12345678912/secrets/smtppassword
```
The mailer will transmit all messages found on the queue on each execution using SMTP/Email delivery.
### Deploying GCP Functions
GCP Cloud Functions for c7n-mailer are currently not supported.
## Writing an email template
Templates are authored in [jinja2](http://jinja.pocoo.org/docs/dev/templates/).
Drop a file with the `.j2` extension into the a templates directory, and send a pull request to this
repo. You can then reference it in the `notify` action as the `template`
variable by file name minus extension. Templates ending with `.html.j2` are
sent as HTML-formatted emails, all others are sent as plain text.
You can use `-t` or `--templates` cli argument to pass custom folder with your templates.
The following variables are available when rendering templates:
| variable | value |
|:------------------|:-------------------------------------------------------------|
| `recipient` | email address |
| `resources` | list of resources that matched the policy filters |
| `event` | for CWE-push-based lambda policies, the event that triggered |
| `action` | `notify` action that generated this SQS message |
| `policy` | policy that triggered this notify action |
| `account` | short name of the aws account |
| `region` | region the policy was executing in |
| `execution_start` | The time policy started executing |
The following extra global functions are available:
| signature | behavior |
|:-----------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------|
| `format_struct(struct)` | pretty print a json structure |
| `resource_tag(resource, key)` | retrieve a tag value from a resource or return an empty string, aliased as get_resource_tag_value |
| `format_resource(resource, resource_type)` | renders a one line summary of a resource |
| `date_time_format(utc_str, tz_str='US/Eastern', format='%Y %b %d %H:%M %Z')` | customize rendering of an utc datetime string |
| `search(expression, value)` | jmespath search value using expression |
| `yaml_safe(value)` | yaml dumper |
The following extra jinja filters are available:
| filter | behavior |
|:-----------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>utc_string|date_time_format(tz_str='US/Pacific', format='%Y %b %d %H:%M %Z')</code> | pretty [format](https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior) the date / time |
| <code>30|get_date_time_delta</code> | Convert a time [delta](https://docs.python.org/2/library/datetime.html#datetime.timedelta) like '30' days in the future, to a datetime string. You can also use negative values for the past. |
## Developer Install (OS X El Capitan)
Clone the repository:
```
$ git clone https://github.com/cloud-custodian/cloud-custodian
```
Install dependencies (with virtualenv):
```
$ virtualenv c7n_mailer
$ source c7n_mailer/bin/activate
$ cd tools/c7n_mailer
$ pip install -r requirements.txt
```
Install the extensions:
```
python setup.py develop
```
## Testing Templates and Recipients
A ``c7n-mailer-replay`` entrypoint is provided to assist in testing email notifications
and templates. This script operates on an actual SQS message from cloud-custodian itself,
which you can either retrieve from the SQS queue or replicate locally. By default it expects
the message file to be base64-encoded, gzipped JSON, just like c7n sends to SQS. With the
``-p`` | ``--plain`` argument, it will expect the message file to contain plain JSON.
``c7n-mailer-replay`` has three main modes of operation:
* With no additional arguments, it will render the template specified by the policy the
message is for, and actually send mail from the local machine as ``c7n-mailer`` would.
This only works with SES, not SMTP.
* With the ``-T`` | ``--template-print`` argument, it will log the email addresses that would
receive mail, and print the rendered message body template to STDOUT.
* With the ``-d`` | ``--dry-run`` argument, it will print the actual email body (including headers)
that would be sent, for each message that would be sent, to STDOUT.
### Testing Templates for Azure
The ``c7n-mailer-replay`` entrypoint can be used to test templates for Azure with either of the arguments:
* ``-T`` | ``--template-print``
* ``-d`` | ``--dry-run``
Running ``c7n-mailer-replay`` without either of these arguments will throw an error as it will attempt
to authorize with AWS.
The following is an example for retrieving a sample message to test against templates:
* Run a policy with the notify action, providing the name of the template to test, to populate the queue.
* Using the azure cli, save the message locally:
```
$ az storage message get --queue-name <queuename> --account-name <storageaccountname> --query '[].content' > test_message.gz
```
* The example message can be provided to ``c7n-mailer-replay`` by running:
```
$ c7n-mailer-replay test_message.gz -T --config mailer.yml
```
Raw data
{
"_id": null,
"home_page": "https://cloudcustodian.io",
"name": "c7n-mailer",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Cloud Custodian Project",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# c7n-mailer: Custodian Mailer\n\n% [comment]: # ( !!! IMPORTANT !!! )\n% [comment]: # (This file is moved during document generation.)\n% [comment]: # (Only edit the original document at ./tools/c7n_mailer/README.md)\n\nA mailer implementation for Custodian. Outbound mail delivery is still somewhat\norganization-specific, so this at the moment serves primarily as an example\nimplementation.\n\n> The Cloud Custodian Mailer can now be easily run in a Docker container. Click [here](https://hub.docker.com/r/cloudcustodian/mailer) for details.\n\n\n## Message Relay\n\nCustodian Mailer subscribes to an SQS queue, looks up users, and sends emails\nvia SES, Slack messages, and/or notifications to DataDog or Splunk. Custodian Lambda and instance policies can send messages to the SQS queue. SQS queues\nshould be cross-account enabled for sending between accounts.\n\n\n## Tutorial\n\nOur goal in starting out with the Custodian mailer is to install the mailer,\nand run a policy that triggers an email to your inbox.\n\n1. [Install](#developer-install-os-x-el-capitan) the mailer on your laptop (if you are not running as a [Docker container](https://hub.docker.com/r/cloudcustodian/mailer)\n - or use `pip install c7n-mailer`\n2. In your text editor, create a `mailer.yml` file to hold your mailer config.\n3. In the AWS console, create a new standard SQS queue (quick create is fine).\n Copy the queue URL to `queue_url` in `mailer.yml`.\n4. In AWS, locate or create a role that has read access to the queue. Grab the\n role ARN and set it as `role` in `mailer.yml`.\n\nThere are different notification endpoints options, you can combine both.\n\n### Email:\nMake sure your email address is verified in SES, and set it as\n`from_address` in `mailer.yml`. By default SES is in sandbox mode where you\nmust\n[verify](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html)\nevery individual recipient of emails. If need be, make an AWS support ticket to\nbe taken out of SES sandbox mode.\n\nYour `mailer.yml` should now look something like this:\n\n```yaml\nqueue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\nrole: arn:aws:iam::123456790:role/c7n-mailer-test\nfrom_address: you@example.com\n```\n\nYou can also set `region` if you are in a region other than `us-east-1` as well as `lambda_tags` to give the mailer tags.\n\n```yaml\nregion: us-east-2\nlambda_tags:\n owner: ops\n```\n\nNow let's make a Custodian policy to populate your mailer queue. Create a\n`test-policy.yml` file with this content (update `to` and `queue` to match your\nenvironment)\n\n```yaml\n policies:\n - name: c7n-mailer-test\n resource: sqs\n filters:\n - \"tag:MailerTest\": absent\n actions:\n - type: notify\n template: default\n priority_header: '2'\n subject: testing the c7n mailer\n to:\n - you@example.com\n transport:\n type: sqs\n queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\n```\n\n### DataDog:\nThe standard way to do a DataDog integration is use the\nc7n integration with AWS CloudWatch and use the\n[DataDog integration with AWS](https://docs.datadoghq.com/integrations/amazon_web_services/)\nto collect CloudWatch metrics. The mailer/messenger integration is only\nfor the case you don't want or you can't use AWS CloudWatch, e.g. in Azure or GCP.\n\nNote this integration requires the additional dependency of Datadog Python bindings:\n```\npip install datadog\n```\n\nYour `mailer.yml` should now look something like this:\n\n```yaml\nqueue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\nrole: arn:aws:iam::123456790:role/c7n-mailer-test\ndatadog_api_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX\ndatadog_application_key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY\n```\n\n(Also set `region` if you are in a region other than `us-east-1`.)\n\nNow, let's make a Custodian policy to populate your mailer queue. Create a\n`test-policy.yml`:\n\n```yaml\npolicies:\n - name: c7n-mailer-test\n resource: ebs\n filters:\n - Attachments: []\n actions:\n - type: notify\n to:\n - datadog://?metric_name=datadog.metric.name&metric_value_tag=Size\n transport:\n type: sqs\n queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\n```\n\nThere is a special `to` format that specifies Datadog delivery and it includes the Datadog configuration via URL parameters:\n\n- `metric_name`: is the name of the metrics send to DataDog.\n- `metric_value_tag`: by default the metric value to send to DataDog is `1`, but if you want to use one of the tags returned in the policy, you can set it with the attribute `metric_value_tag`. For example, in the `test-policy.yml` policy, the value used is the size of the EBS volume. The value must be a number and it will be transformed to a float value.\n\n### Slack:\n\nThe Custodian mailer supports Slack messaging as a separate notification mechanism for the SQS transport method. To enable Slack integration, you must specify a Slack token in the `slack_token` field in your `mailer.yml` file. For example:\n\n```yaml\nqueue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\nrole: arn:aws:iam::123456790:role/c7n-mailer-test\nslack_token: xoxo-token123\n```\n\nTo enable Slack messaging, several unique fields are evaluated in the policy, as shown in the below example:\n\n```\npolicies:\n - name: c7n-mailer-test\n resource: ebs\n filters:\n - Attachments: []\n actions:\n - type: notify\n slack_template: slack\n slack_msg_color: danger\n to:\n - slack://owners\n - slack://foo@bar.com\n - slack://#custodian-test\n - slack://webhook/#c7n-webhook-test\n - slack://tag/resource_tag\n - https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX\n transport:\n type: sqs\n queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\n```\n\nSlack messages support the use of a unique template field specified in `slack_template`. This field is unique and usage will not break\nexisting functionality for messages also specifying an email template in the `template` field. This field is optional, however,\nand if not specified, the mailer will use the default value `slack_default`.\n\nThe unique template field `slack_msg_color` can be used to specify a color\nborder for the Slack message. This accepts the Slack presets of `danger` (red),\n`warning` (yellow) and `good` (green). It can also accept an HTML hex code. See\nthe [Slack documentation](https://api.slack.com/reference/messaging/attachments#fields)\nfor details.\n\n**Note:** if you are using a hex color code, it will need to be wrapped in quotes\nlike so: `slack_msg_color: '#4287f51'`. Otherwise, the YAML interpreter will consider it a\n[comment](https://yaml.org/spec/1.2/spec.html#id2780069).\n\nThe Slack integration for c7n-mailer supports several flavors of messaging, listed below. These are not mutually exclusive and any combination of the types can be used, but the preferred method is [incoming webhooks](https://api.slack.com/incoming-webhooks).\n\n| Requires `slack_token` | Key | Type | Notes |\n|:---------------------------:|:--------------------------------------------------------------------------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| No | `https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX` | string | **(PREFERRED)** Send to an [incoming webhook](https://api.slack.com/incoming-webhooks) (the channel is defined in the webhook) |\n| Yes | `slack://owners` | string | Send to the recipient list generated within email delivery logic |\n| Yes | `slack://foo@bar.com` | string | Send to the recipient specified by email address foo@bar.com |\n| Yes | `slack://#custodian-test` | string | Send to the Slack channel indicated in string, i.e. #custodian-test |\n| No | `slack://webhook/#c7n-webhook-test` | string | **(DEPRECATED)** Send to a Slack webhook; appended with the target channel. **IMPORTANT**: *This requires a `slack_webhook` value defined in the `mailer.yml`.* |\n| Yes | `slack://tag/resource-tag` | string | Send to target found in resource tag. Example of value in tag: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX |\n\nSlack delivery can also be set via a resource's tag name. For example, using `slack://tag/slack_channel` will look for a tag name of `slack_channel`, and if matched on a resource will deliver the message to the value of that resource's tag:\n\n`slack_channel:https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`\n`slack_channel:custodian-test`\n`owner:foo@bar`\n\nDelivery via tag has been tested with webhooks, but it should support all delivery methods.\n\n### Splunk HTTP Event Collector (HEC)\n\nThe Custodian mailer supports delivery to the HTTP Event Collector (HEC) endpoint of a Splunk instance as a separate notification mechanism for the SES transport method. To enable Splunk HEC integration, you must specify the URL to the HEC endpoint as well as a valid username and token:\n\n```yaml\nqueue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\nrole: arn:aws:iam::123456790:role/c7n-mailer-test\nsplunk_hec_url: https://http-inputs-foo.splunkcloud.com/services/collector/event\nsplunk_hec_token: 268b3cc2-f32e-4a19-a1e8-aee08d86ca7f\n```\n\nTo send events for a policy to the Splunk HEC endpoint, add a ``to`` address notify action specifying the name of the Splunk index to send events to in the form ``splunkhec://indexName``:\n\n```\npolicies:\n - name: c7n-mailer-test\n resource: ebs\n filters:\n - Attachments: []\n actions:\n - type: notify\n to:\n - splunkhec://myIndexName\n transport:\n type: sqs\n queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test\n```\n\nThe ``splunkhec://indexName`` address type can be combined in the same notify action with other destination types (e.g. email, Slack, DataDog, etc).\n\n### Now run:\n\n```\nc7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .\n```\n\nNote: You can set the profile via environment variable e.g. `export AWS_DEFAULT_PROFILE=foo`\n\nYou should see output similar to the following:\n\n```\n(env) $ c7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .\nDEBUG:custodian.lambda:Created custodian lambda archive size: 3.01mb\n2017-01-12 07:55:16,227: custodian.policy:INFO Running policy c7n-mailer-test resource: sqs region:default c7n:0.8.22.0\n2017-01-12 07:55:16,229: custodian.policy:INFO policy: c7n-mailer-test resource:sqs has count:1 time:0.00\n2017-01-12 07:55:18,017: custodian.actions:INFO sent message:dead-beef policy:c7n-mailer-test template:default count:1\n2017-01-12 07:55:18,017: custodian.policy:INFO policy: c7n-mailer-test action: notify resources: 1 execution_time: 1.79\n(env) $\n```\n\nCheck the AWS console for a new Lambda named `cloud-custodian-mailer`. The\nmailer runs every five minutes, so wait a bit and then look for an email in\nyour inbox. If it doesn't appear, look in the lambda's logs for debugging\ninformation. If it does, congratulations! You are off and running with the\nCustodian mailer.\n\n\n## Usage & Configuration\n\nOnce [installed](#developer-install-os-x-el-capitan) you should have a\n`c7n-mailer` executable on your path:\n\n```\n(env) $ c7n-mailer\nusage: c7n-mailer [-h] -c CONFIG\nc7n-mailer: error: argument -c/--config is required\n(env) $\n```\n\nFundamentally, what `c7n-mailer` does is it deploys a Lambda function (using\n[Mu](http://cloudcustodian.io/docs/policy/mu.html)) based on\nthe configuration you specify in a YAML file. Here is [the\nschema](https://github.com/cloud-custodian/cloud-custodian/blob/18d4247e913d54f36a078ed61386695362a3b10d/tools/c7n_mailer/c7n_mailer/cli.py#L43) to which the file must conform\nand here is a description of the options:\n\n| Required? | Key | Type | Notes |\n|:---------:|:--------------------------|:-----------------|:------------------------------------------------------------------|\n| ✅ | `queue_url` | string | the queue to listen to for messages | \n| | `from_address` | string | default from address |\n| | `endpoint_url` | string | SQS API URL (for use with VPC Endpoints) |\n| | `contact_tags` | array of strings | tags that we should look at for address information |\n| | `email_base_url` | string | Base URL to construct a valid email address from a resource owner |\n| | `additional_email_headers`| object | Additional email headers. |\n| | `org_domain` | string | Domain to add to AWS usernames to construct email addresses. |\n\n\n\n### Standard Lambda Function Config\n\n| Required? | Key | Type |\n|:---------:|:---------------------|:-----------------|\n| ✅ | `role` | string |\n| | `dead_letter_config` | object |\n| | `memory` | integer |\n| | `region` | string |\n| | `runtime` | string |\n| | `security_groups` | array of strings |\n| | `subnets` | array of strings |\n| | `timeout` | integer |\n| | `lambda_name` | string |\n| | `lambda_description` | string |\n| | `lambda_tags` | object |\n| | `lambda_schedule` | string |\n\n### Standard Azure Functions Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:----------------------|:-------|:---------------------------------------------------------------------------------------|\n| | `function_properties` | object | Contains `appInsights`, `storageAccount` and `servicePlan` objects |\n| | `appInsights` | object | Contains `name`, `location` and `resourceGroupName` properties |\n| | `storageAccount` | object | Contains `name`, `location` and `resourceGroupName` properties |\n| | `servicePlan` | object | Contains `name`, `location`, `resourceGroupName`, `skuTier` and `skuName` properties |\n| | `name` | string | |\n| | `location` | string | Default: `west us 2` |\n| | `resourceGroupName` | string | Default `cloud-custodian` |\n| | `skuTier` | string | Default: `Basic` |\n| | `skuName` | string | Default: `B1` |\n\n\n\n\n### Mailer Infrastructure Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:----------------------------|:--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| | `cache_engine` | string | cache engine; either sqlite or redis |\n| | `cross_accounts` | object | account to assume back into for sending to SNS topics |\n| | `debug` | boolean | debug on/off |\n| | `ldap_bind_dn` | string | eg: ou=people,dc=example,dc=com |\n| | `ldap_bind_user` | string | eg: FOO\\\\BAR |\n| | `ldap_bind_password` | secured string | ldap bind password |\n| | `ldap_bind_password_in_kms` | boolean | defaults to true, most people (except capone) want to set this to false. If set to true, make sure `ldap_bind_password` contains your KMS encrypted ldap bind password as a base64-encoded string. |\n| | `ldap_email_attribute` | string | |\n| | `ldap_email_key` | string | eg 'mail' |\n| | `ldap_manager_attribute` | string | eg 'manager' |\n| | `ldap_uid_attribute` | string | |\n| | `ldap_uid_regex` | string | |\n| | `ldap_uid_tags` | string | |\n| | `ldap_uri` | string | eg 'ldaps://example.com:636' |\n| | `redis_host` | string | redis host if cache_engine == redis |\n| | `redis_port` | integer | redis port, default: 6369 |\n| | `ses_region` | string | AWS region that handles SES API calls |\n| | `ses_role` | string | ARN of the role to assume to send email with SES |\n\n### SMTP Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:----------------|:-----------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| | `smtp_server` | string | to configure your lambda role to talk to smtpd in your private vpc, see [here](https://docs.aws.amazon.com/lambda/latest/dg/vpc.html) | |\n| | `smtp_port` | integer | smtp port (default is 25) |\n| | `smtp_ssl` | boolean | this defaults to True |\n| | `smtp_username` | string | |\n| | `smtp_password` | secured string | |\n\nIf `smtp_server` is unset, `c7n_mailer` will use AWS SES or Azure SendGrid.\n\n#### DataDog Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:--------------------------|:-------|:-------------------------|\n| | `datadog_api_key` | string | DataDog API key. |\n| | `datadog_application_key` | string | Datadog application key. |\n\nThese fields are not necessary if c7n_mailer is run in a instance/lambda/etc with the DataDog agent.\n\n### Slack Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:--------------|:-------|:----------------|\n| | `slack_token` | string | Slack API token |\n\n### SendGrid Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:-------------------|:---------------|:-------------------|\n| | `sendgrid_api_key` | secured string | SendGrid API token |\n\n\n### Microsoft Graph Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:--------------------------|:---------------|:-------------------------|\n| | `graph_token_endpoint` | string | Graph token endpoint |\n| | `graph_sendmail_endpoint` | string | Graph sendmail endpoint |\n| | `graph_client_id` | string | Client ID |\n| | `graph_client_secret` | secured string | Client secret |\n\n\n### Splunk HEC Config\n\nThe following configuration items are *all* optional. The ones marked \"Required for Splunk\" are only required if you're sending notifications to ``splunkhec://`` destinations.\n\n| Required for Splunk? | Key | Type | Notes |\n|:--------------------:|:------------------------|:-----------------|:-----------------------------------------------------------------------------------------------------------------------------------|\n| ✅ | `splunk_hec_url` | string | URL to your Splunk HTTP Event Collector endpoint |\n| ✅ | `splunk_hec_token` | string | Splunk HEC authentication token for specified username |\n| | `splunk_remove_paths` | array of strings | List of [RFC6901](http://tools.ietf.org/html/rfc6901) JSON Pointers to remove from the event, if present, before sending to Splunk |\n| | `splunk_actions_list` | boolean | If true, add an `actions` list to the top-level message sent to Splunk, containing the names of all non-notify actions taken |\n| | `splunk_max_attempts` | integer | Maximum number of times to try POSTing data to Splunk HEC (default 4) |\n| | `splunk_hec_max_length` | integer | Maximum data length that Splunk HEC accepts; an error will be logged for any message sent over this length |\n| | `splunk_hec_sourcetype` | string | Configure sourcetype of the payload sent to Splunk HEC. (default is '_json') |\n\n### SDK Config\n\n| Required? | Key | Type | Notes |\n|:---------:|:--------------|:-------|:------|\n| | `http_proxy` | string | |\n| | `https_proxy` | string | |\n| | `profile` | string | |\n\n\n### Secured String\n\nIn order to ensure sensitive data is not stored plaintext in a policy, `c7n-mailer` supports secured\nstrings. You can treat it as a regular `string` or use `secured string` features.\n\n#### AWS\n\nYou can use KMS to encrypt your secrets and use encrypted secret in mailer policy.\nCustodian tries to decrypt the string using KMS, if it fails c7n treats it as a plaintext secret.\n\n```yaml\n plaintext_secret: <raw_secret>\n secured_string: <encrypted_secret>\n```\n\n#### Azure\n\nYou can store your secrets in Azure Key Vault secrets and reference them from the policy.\n\n```yaml\n plaintext_secret: <raw_secret>\n secured_string:\n type: azure.keyvault\n secret: https://your-vault.vault.azure.net/secrets/your-secret\n```\n\nNote: `secrets.get` permission on the KeyVault for the Service Principal is required.\n\n#### GCP\n\nYou can store your secrets as GCP Secret Manager secrets and reference them from the policy.\n\n```yaml\n plaintext_secret: <raw_secret>\n secured_string:\n type: gcp.secretmanager\n secret: projects/12345678912/secrets/your-secret\n```\n\nAn example of an SMTP password set as a secured string:\n\n```yaml\n smtp_password:\n type: gcp.secretmanager\n secret: projects/59808015552/secrets/smtp_pw\n```\n\nNote: If you do not specify a version, `/versions/latest` will be appended to your secret location.\n\n## Configuring a policy to send email\n\nOutbound email can be added to any policy by including the `notify` action.\n\n```yaml\n\npolicies:\n - name: bad-apples\n resource: asg\n filters:\n - \"tag:ASV\": absent\n actions:\n - type: notify\n template: default\n template_format: 'html'\n priority_header: '1'\n subject: fix your tags\n to:\n - resource-owner\n owner_absent_contact:\n - foo@example.com\n transport:\n type: sqs\n queue: https://sqs.us-east-1.amazonaws.com/80101010101/cloud-custodian-message-relay\n```\n\nSo breaking it down, you add an action of type `notify`. You can specify a\ntemplate that's used to format the email; customizing templates is described\n[below](#writing-an-email-template).\n\nThe `to` list specifies the intended recipient for the email. You can specify\neither an email address, an SNS topic, a Datadog Metric, or a special value. The special values\nare either\n\n- `resource-owner`, in which case the email will be sent to the listed\n `OwnerContact` tag on the resource that matched the policy, or\n- `event-owner` for push-based/realtime policies that will send to the user\n that was responsible for the underlying event.\n- `priority_header` to indicate the importance of an email with [headers](https://www.chilkatsoft.com/p/p_471.asp). Different emails clients will display stars, exclamation points or flags depending on the value. Should be an string from 1 to 5.\n\nBoth of these special values are best effort, i.e., if no `OwnerContact` tag is\nspecified then `resource-owner` email will not be delivered, and in the case of\n`event-owner` an instance role or system account will not result in an email.\n\nThe optional `owner_absent_contact` list specifies email addresses to notify only if\nthe `resource-owner` special option was unable to find any matching owner contact\ntags.\n\nIn addition, you may choose to use a custom tag instead of the default `OwnerContact`. In order to configure this, the mailer.yaml must be modified to include the contact_tags and the custom tag. The `resource-owner` will now email the custom tag instead of `OwnerContact`.\n\n```yaml\ncontact_tags:\n - \"custom_tag\"\n```\n\n\nFor reference purposes, the JSON Schema of the `notify` action:\n\n```json\n{\n \"type\": \"object\",\n \"required\": [\"type\", \"transport\", \"to\"],\n \"properties\": {\n \"type\": {\"enum\": [\"notify\"]},\n \"to\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n \"owner_absent_contact\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n \"subject\": {\"type\": \"string\"},\n \"priority_header\": {\"type\": \"string\"},\n \"template\": {\"type\": \"string\"},\n \"transport\": {\n \"type\": \"object\",\n \"required\": [\"type\", \"queue\"],\n \"properties\": {\n \"queue\": {\"type\": \"string\"},\n \"region\": {\"type\": \"string\"},\n \"type\": {\"enum\": [\"sqs\"]}\n }\n }\n }\n}\n```\n\n## Using on Azure\n\nRequires:\n\n- `c7n_azure` package. See [Installing Azure Plugin](https://cloudcustodian.io/docs/azure/gettingstarted.html#azure-install-cc)\n- SendGrid account. See [Using SendGrid with Azure](https://docs.microsoft.com/en-us/azure/sendgrid-dotnet-how-to-send-email)\n- [Azure Storage Queue](https://azure.microsoft.com/en-us/services/storage/queues/)\n\nThe mailer supports an Azure Storage Queue transport and SendGrid delivery on Azure.\nConfiguration for this scenario requires only minor changes from AWS deployments.\n\nYou will need to grant `Storage Queue Data Contributor` role on the Queue for the identity\nmailer is running under.\n\nThe notify action in your policy will reflect transport type `asq` with the URL\nto an Azure Storage Queue. For example:\n\n```yaml\npolicies:\n - name: azure-notify\n resource: azure.resourcegroup\n description: send a message to a mailer instance\n actions:\n - type: notify\n template: default\n priority_header: '2'\n subject: Hello from C7N Mailer\n to:\n - you@youremail.com\n transport:\n type: asq\n queue: https://storageaccount.queue.core.windows.net/queuename\n```\n\nIn your mailer configuration, you'll need to provide your SendGrid API key as well as\nprefix your queue URL with `asq://` to let mailer know what type of queue it is:\n\n```yaml\nqueue_url: asq://storageaccount.queue.core.windows.net/queuename\nfrom_address: you@youremail.com\nsendgrid_api_key: SENDGRID_API_KEY\n```\n\nThe mailer will transmit all messages found on the queue on each execution, and will retry\nsending 3 times in the event of a failure calling SendGrid. After the retries the queue\nmessage will be discarded.\n\nIn addition, SendGrid delivery on Azure supports using resource tags to send emails. For example, in the `to` field:\n\n```yaml\nto:\n - tag:OwnerEmail\n```\n\nThis will find the email address associated with the resource's `OwnerEmail` tag, and send an email to the specified address.\nIf no tag is found, or the associated email address is invalid, no email will be sent.\n\n### Deploying Azure Functions\n\nThe `--update-lambda` CLI option will also deploy Azure Functions if you have an Azure\nmailer configuration.\n\n`c7n-mailer --config mailer.yml --update-lambda`\n\nwhere a simple `mailer.yml` using Consumption functions may look like:\n\n```yaml\nqueue_url: asq://storage.queue.core.windows.net/custodian\nfrom_address: foo@mail.com\nsendgrid_api_key: <key>\nfunction_properties:\n servicePlan:\n name: 'testmailer1'\n```\n\n### Configuring Function Identity\n\nYou can configure the service principal used for api calls made by the\nmailer azure function by specifying an identity configuration under\nfunction properties. Mailer supports User Assigned Identities, System\nManaged Identities, defaulting to an embedding of the cli user's\nservice principals credentials.\n\nWhen specifying a user assigned identity, unlike in a custodian\nfunction policy where simply providing an name is sufficient, the\nuuid/id and client id of the identity must be provided. You can\nretrieve this information on the cli using the `az identity list`.\n\n```yaml\n\nfunction_properties:\n identity:\n type: UserAssigned\n id: \"/subscriptions/333fd504-7f11-2270-88c8-7325a27f7222/resourcegroups/c7n/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mailer\"\n client_id: \"b9cb06fa-dfb8-4342-add3-aab5acb2abbc\"\n```\n\nA system managed identity can also be used, and the Azure platform will\ncreate an identity when the function is provisoned, however the function's identity\nthen needs to be retrieved and mapped to rbac permissions post provisioning, this\nuser management activity must be performed manually.\n\n```yaml\n\nfunction_properties:\n identity:\n type: SystemAssigned\n```\n\n## Using on GCP\n\nRequires:\n\n- `c7n_gcp` package. See [GCP Getting Started](https://cloudcustodian.io/docs/gcp/gettingstarted.html)\n- `google-cloud-secret-manager` package, for pulling in secured string values.\n- A working SMTP Account.\n- [GCP Pubsub Subscription](https://cloud.google.com/pubsub/docs/)\n\nThe mailer supports GCP Pubsub transports and SMTP/Email delivery, as well as Datadog and Splunk.\nConfiguration for this scenario requires only minor changes from AWS deployments.\n\nThe notify action in your policy will reflect transport type `projects` with the URL\nto a GCP Pub/Sub Topic. For example:\n\n```yaml\npolicies:\n - name: gcp-notify\n resource: gcp.compute\n description: example policy\n actions:\n - type: notify\n template: default\n priority_header: '2'\n subject: Hello from C7N Mailer\n to:\n - you@youremail.com\n transport:\n type: pubsub\n topic: projects/myproject/topics/mytopic\n```\n\nIn your mailer configuration, you'll need to provide your SMTP account information\nas well as your topic subscription path in the queue_url variable. Please note that the\nsubscription you specify should be subscribed to the topic you assign in your policies'\nnotify action for GCP resources.\n\n```yaml\nqueue_url: projects/myproject/subscriptions/mysubscription\nfrom_address: you@youremail.com\n# c7n-mailer currently requires a role be present, even if it's empty\nrole: \"\"\n\nsmtp_server: my.smtp.add.ress\nsmtp_port: 25\nsmtp_ssl: true\nsmtp_username: smtpuser\nsmtp_password:\n type: gcp.secretmanager\n secret: projects/12345678912/secrets/smtppassword\n```\n\nThe mailer will transmit all messages found on the queue on each execution using SMTP/Email delivery.\n\n### Deploying GCP Functions\n\nGCP Cloud Functions for c7n-mailer are currently not supported.\n\n## Writing an email template\n\nTemplates are authored in [jinja2](http://jinja.pocoo.org/docs/dev/templates/).\nDrop a file with the `.j2` extension into the a templates directory, and send a pull request to this\nrepo. You can then reference it in the `notify` action as the `template`\nvariable by file name minus extension. Templates ending with `.html.j2` are\nsent as HTML-formatted emails, all others are sent as plain text.\n\nYou can use `-t` or `--templates` cli argument to pass custom folder with your templates.\n\nThe following variables are available when rendering templates:\n\n| variable | value |\n|:------------------|:-------------------------------------------------------------|\n| `recipient` | email address |\n| `resources` | list of resources that matched the policy filters |\n| `event` | for CWE-push-based lambda policies, the event that triggered |\n| `action` | `notify` action that generated this SQS message |\n| `policy` | policy that triggered this notify action |\n| `account` | short name of the aws account |\n| `region` | region the policy was executing in |\n| `execution_start` | The time policy started executing |\n\nThe following extra global functions are available:\n\n| signature | behavior |\n|:-----------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------|\n| `format_struct(struct)` | pretty print a json structure |\n| `resource_tag(resource, key)` | retrieve a tag value from a resource or return an empty string, aliased as get_resource_tag_value |\n| `format_resource(resource, resource_type)` | renders a one line summary of a resource |\n| `date_time_format(utc_str, tz_str='US/Eastern', format='%Y %b %d %H:%M %Z')` | customize rendering of an utc datetime string |\n| `search(expression, value)` | jmespath search value using expression |\n| `yaml_safe(value)` | yaml dumper |\n\nThe following extra jinja filters are available:\n\n| filter | behavior |\n|:-----------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| <code>utc_string|date_time_format(tz_str='US/Pacific', format='%Y %b %d %H:%M %Z')</code> | pretty [format](https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior) the date / time |\n| <code>30|get_date_time_delta</code> | Convert a time [delta](https://docs.python.org/2/library/datetime.html#datetime.timedelta) like '30' days in the future, to a datetime string. You can also use negative values for the past. |\n\n\n## Developer Install (OS X El Capitan)\n\nClone the repository:\n```\n$ git clone https://github.com/cloud-custodian/cloud-custodian\n```\nInstall dependencies (with virtualenv):\n```\n$ virtualenv c7n_mailer\n$ source c7n_mailer/bin/activate\n$ cd tools/c7n_mailer\n$ pip install -r requirements.txt\n```\nInstall the extensions:\n```\npython setup.py develop\n```\n\n## Testing Templates and Recipients\n\nA ``c7n-mailer-replay`` entrypoint is provided to assist in testing email notifications\nand templates. This script operates on an actual SQS message from cloud-custodian itself,\nwhich you can either retrieve from the SQS queue or replicate locally. By default it expects\nthe message file to be base64-encoded, gzipped JSON, just like c7n sends to SQS. With the\n``-p`` | ``--plain`` argument, it will expect the message file to contain plain JSON.\n\n``c7n-mailer-replay`` has three main modes of operation:\n\n* With no additional arguments, it will render the template specified by the policy the\n message is for, and actually send mail from the local machine as ``c7n-mailer`` would.\n This only works with SES, not SMTP.\n* With the ``-T`` | ``--template-print`` argument, it will log the email addresses that would\n receive mail, and print the rendered message body template to STDOUT.\n* With the ``-d`` | ``--dry-run`` argument, it will print the actual email body (including headers)\n that would be sent, for each message that would be sent, to STDOUT.\n\n### Testing Templates for Azure\n\nThe ``c7n-mailer-replay`` entrypoint can be used to test templates for Azure with either of the arguments:\n* ``-T`` | ``--template-print``\n* ``-d`` | ``--dry-run``\n\nRunning ``c7n-mailer-replay`` without either of these arguments will throw an error as it will attempt\nto authorize with AWS.\n\nThe following is an example for retrieving a sample message to test against templates:\n\n* Run a policy with the notify action, providing the name of the template to test, to populate the queue.\n\n* Using the azure cli, save the message locally:\n```\n$ az storage message get --queue-name <queuename> --account-name <storageaccountname> --query '[].content' > test_message.gz\n```\n* The example message can be provided to ``c7n-mailer-replay`` by running:\n\n```\n$ c7n-mailer-replay test_message.gz -T --config mailer.yml\n```\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Cloud Custodian - Reference Mailer",
"version": "0.6.34",
"project_urls": {
"Documentation": "https://cloudcustodian.io/docs/",
"Homepage": "https://cloudcustodian.io",
"Repository": "https://github.com/cloud-custodian/cloud-custodian"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1d6cc42af97dbe29d709ec529147ec608c3742733063ffbec22a8cc2f7881ac0",
"md5": "ac789a9c95061770c2ee17fac6b413a0",
"sha256": "8cb78a0b26f60f5d51af5f35df8382c1804fc35e5579fae26e8c3a5b30bdb71c"
},
"downloads": -1,
"filename": "c7n_mailer-0.6.34-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ac789a9c95061770c2ee17fac6b413a0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 62533,
"upload_time": "2024-03-26T21:21:33",
"upload_time_iso_8601": "2024-03-26T21:21:33.340962Z",
"url": "https://files.pythonhosted.org/packages/1d/6c/c42af97dbe29d709ec529147ec608c3742733063ffbec22a8cc2f7881ac0/c7n_mailer-0.6.34-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 21:21:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cloud-custodian",
"github_project": "cloud-custodian",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "c7n-mailer"
}
JSON
