# Amazon Data Firehose Construct Library
<!--BEGIN STABILITY BANNER-->---
> The APIs of higher level constructs in this module are in **developer preview** before they
> become stable. We will only make breaking changes to address unforeseen API issues. Therefore,
> these APIs are not subject to [Semantic Versioning](https://semver.org/), and breaking changes
> will be announced in release notes. This means that while you may use them, you may need to
> update your source code when upgrading to a newer version of this package.
---
<!--END STABILITY BANNER-->
[Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html), [formerly known as Amazon Kinesis Data Firehose](https://aws.amazon.com/about-aws/whats-new/2024/02/amazon-data-firehose-formerly-kinesis-data-firehose/),
is a service for fully-managed delivery of real-time streaming data to storage services
such as Amazon S3, Amazon Redshift, Amazon Elasticsearch, Splunk, or any custom HTTP
endpoint or third-party services such as Datadog, Dynatrace, LogicMonitor, MongoDB, New
Relic, and Sumo Logic.
Amazon Data Firehose delivery streams are distinguished from Kinesis data streams in
their models of consumption. Whereas consumers read from a data stream by actively pulling
data from the stream, a delivery stream pushes data to its destination on a regular
cadence. This means that data streams are intended to have consumers that do on-demand
processing, like AWS Lambda or Amazon EC2. On the other hand, delivery streams are
intended to have destinations that are sources for offline processing and analytics, such
as Amazon S3 and Amazon Redshift.
This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk)
project. It allows you to define Amazon Data Firehose delivery streams.
## Defining a Delivery Stream
In order to define a Delivery Stream, you must specify a destination. An S3 bucket can be
used as a destination. Currently the CDK supports only S3 as a destination which is covered [below](#destinations).
```python
bucket = s3.Bucket(self, "Bucket")
firehose.DeliveryStream(self, "Delivery Stream",
destination=destinations.S3Bucket(bucket)
)
```
The above example defines the following resources:
* An S3 bucket
* An Amazon Data Firehose delivery stream with Direct PUT as the source and CloudWatch
error logging turned on.
* An IAM role which gives the delivery stream permission to write to the S3 bucket.
## Sources
An Amazon Data Firehose delivery stream can accept data from three main sources: Kinesis Data Streams, Managed Streaming for Apache Kafka (MSK), or via a "direct put" (API calls). Currently only Kinesis Data Streams and direct put are supported in the CDK.
See: [Sending Data to a Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html)
in the *Amazon Data Firehose Developer Guide*.
### Kinesis Data Stream
A delivery stream can read directly from a Kinesis data stream as a consumer of the data
stream. Configure this behaviour by passing in a data stream in the `source`
property via the `KinesisStreamSource` class when constructing a delivery stream:
```python
# destination: firehose.IDestination
source_stream = kinesis.Stream(self, "Source Stream")
firehose.DeliveryStream(self, "Delivery Stream",
source=firehose.KinesisStreamSource(source_stream),
destination=destination
)
```
### Direct Put
Data must be provided via "direct put", ie., by using a `PutRecord` or
`PutRecordBatch` API call. There are a number of ways of doing so, such as:
* Kinesis Agent: a standalone Java application that monitors and delivers files while
handling file rotation, checkpointing, and retries. See: [Writing to Amazon Data Firehose Using Kinesis Agent](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html)
in the *Amazon Data Firehose Developer Guide*.
* AWS SDK: a general purpose solution that allows you to deliver data to a delivery stream
from anywhere using Java, .NET, Node.js, Python, or Ruby. See: [Writing to Amazon Data Firehose Using the AWS SDK](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html)
in the *Amazon Data Firehose Developer Guide*.
* CloudWatch Logs: subscribe to a log group and receive filtered log events directly into
a delivery stream. See: [logs-destinations](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-logs-destinations-readme.html).
* Eventbridge: add an event rule target to send events to a delivery stream based on the
rule filtering. See: [events-targets](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-targets-readme.html).
* SNS: add a subscription to send all notifications from the topic to a delivery
stream. See: [sns-subscriptions](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-sns-subscriptions-readme.html).
* IoT: add an action to an IoT rule to send various IoT information to a delivery stream
## Destinations
Amazon Data Firehose supports multiple AWS and third-party services as destinations, including Amazon S3, Amazon Redshift, and more. You can find the full list of supported destination [here](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html).
Currently in the AWS CDK, only S3 is implemented as an L2 construct destination. Other destinations can still be configured using L1 constructs. See [kinesisfirehose-destinations](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-kinesisfirehose-destinations-readme.html)
for the implementations of these destinations.
### S3
Defining a delivery stream with an S3 bucket destination:
```python
# bucket: s3.Bucket
s3_destination = destinations.S3Bucket(bucket)
firehose.DeliveryStream(self, "Delivery Stream",
destination=s3_destination
)
```
The S3 destination also supports custom dynamic prefixes. `dataOutputPrefix`
will be used for files successfully delivered to S3. `errorOutputPrefix` will be added to
failed records before writing them to S3.
```python
# bucket: s3.Bucket
s3_destination = destinations.S3Bucket(bucket,
data_output_prefix="myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}",
error_output_prefix="myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}"
)
```
See: [Custom S3 Prefixes](https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html)
in the *Amazon Data Firehose Developer Guide*.
## Server-side Encryption
Enabling server-side encryption (SSE) requires Amazon Data Firehose to encrypt all data
sent to delivery stream when it is stored at rest. This means that data is encrypted
before being written to the service's internal storage layer and decrypted after it is
received from the internal storage layer. The service manages keys and cryptographic
operations so that sources and destinations do not need to, as the data is encrypted and
decrypted at the boundaries of the service (i.e., before the data is delivered to a
destination). By default, delivery streams do not have SSE enabled.
The Key Management Service keys (KMS keys) used for SSE can either be AWS-owned or
customer-managed. AWS-owned KMS keys are created, owned and managed by AWS for use in
multiple AWS accounts. As a customer, you cannot view, use, track, or manage these keys,
and you are not charged for their use. On the other hand, customer-managed KMS keys are
created and owned within your account and managed entirely by you. As a customer, you are
responsible for managing access, rotation, aliases, and deletion for these keys, and you
are changed for their use.
See: [AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#kms_keys)
in the *KMS Developer Guide*.
```python
# destination: firehose.IDestination
# SSE with an customer-managed key that is explicitly specified
# key: kms.Key
# SSE with an AWS-owned key
firehose.DeliveryStream(self, "Delivery Stream with AWS Owned Key",
encryption=firehose.StreamEncryption.aws_owned_key(),
destination=destination
)
# SSE with an customer-managed key that is created automatically by the CDK
firehose.DeliveryStream(self, "Delivery Stream with Customer Managed Key",
encryption=firehose.StreamEncryption.customer_managed_key(),
destination=destination
)
firehose.DeliveryStream(self, "Delivery Stream with Customer Managed and Provided Key",
encryption=firehose.StreamEncryption.customer_managed_key(key),
destination=destination
)
```
See: [Data Protection](https://docs.aws.amazon.com/firehose/latest/dev/encryption.html)
in the *Amazon Data Firehose Developer Guide*.
## Monitoring
Amazon Data Firehose is integrated with CloudWatch, so you can monitor the performance of
your delivery streams via logs and metrics.
### Logs
Amazon Data Firehose will send logs to CloudWatch when data transformation or data
delivery fails. The CDK will enable logging by default and create a CloudWatch LogGroup
and LogStream with default settings for your Delivery Stream.
When creating a destination, you can provide an `ILoggingConfig`, which can either be an `EnableLogging` or `DisableLogging` instance.
If you use `EnableLogging`, the CDK will create a CloudWatch LogGroup and LogStream with all CloudFormation default settings for you, or you can optionally
specify your own log group to be used for capturing and storing log events. For example:
```python
import aws_cdk.aws_logs as logs
# bucket: s3.Bucket
log_group = logs.LogGroup(self, "Log Group")
destination = destinations.S3Bucket(bucket,
logging_config=destinations.EnableLogging(log_group)
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=destination
)
```
Logging can also be disabled:
```python
# bucket: s3.Bucket
destination = destinations.S3Bucket(bucket,
logging_config=destinations.DisableLogging()
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=destination
)
```
See: [Monitoring using CloudWatch Logs](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html)
in the *Amazon Data Firehose Developer Guide*.
### Metrics
Amazon Data Firehose sends metrics to CloudWatch so that you can collect and analyze the
performance of the delivery stream, including data delivery, data ingestion, data
transformation, format conversion, API usage, encryption, and resource usage. You can then
use CloudWatch alarms to alert you, for example, when data freshness (the age of the
oldest record in the delivery stream) exceeds the buffering limit (indicating that data is
not being delivered to your destination), or when the rate of incoming records exceeds the
limit of records per second (indicating data is flowing into your delivery stream faster
than it is configured to process).
CDK provides methods for accessing delivery stream metrics with default configuration,
such as `metricIncomingBytes`, and `metricIncomingRecords` (see [`IDeliveryStream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.IDeliveryStream.html)
for a full list). CDK also provides a generic `metric` method that can be used to produce
metric configurations for any metric provided by Amazon Data Firehose; the configurations
are pre-populated with the correct dimensions for the delivery stream.
```python
import aws_cdk.aws_cloudwatch as cloudwatch
# delivery_stream: firehose.DeliveryStream
# Alarm that triggers when the per-second average of incoming bytes exceeds 90% of the current service limit
incoming_bytes_percent_of_limit = cloudwatch.MathExpression(
expression="incomingBytes / 300 / bytePerSecLimit",
using_metrics={
"incoming_bytes": delivery_stream.metric_incoming_bytes(statistic=cloudwatch.Statistic.SUM),
"byte_per_sec_limit": delivery_stream.metric("BytesPerSecondLimit")
}
)
cloudwatch.Alarm(self, "Alarm",
metric=incoming_bytes_percent_of_limit,
threshold=0.9,
evaluation_periods=3
)
```
See: [Monitoring Using CloudWatch Metrics](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html)
in the *Amazon Data Firehose Developer Guide*.
## Compression
Your data can automatically be compressed when it is delivered to S3 as either a final or
an intermediary/backup destination. Supported compression formats are: gzip, Snappy,
Hadoop-compatible Snappy, and ZIP, except for Redshift destinations, where Snappy
(regardless of Hadoop-compatibility) and ZIP are not supported. By default, data is
delivered to S3 without compression.
```python
# Compress data delivered to S3 using Snappy
# bucket: s3.Bucket
s3_destination = destinations.S3Bucket(bucket,
compression=destinations.Compression.SNAPPY
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=s3_destination
)
```
## Buffering
Incoming data is buffered before it is delivered to the specified destination. The
delivery stream will wait until the amount of incoming data has exceeded some threshold
(the "buffer size") or until the time since the last data delivery occurred exceeds some
threshold (the "buffer interval"), whichever happens first. You can configure these
thresholds based on the capabilities of the destination and your use-case. By default, the
buffer size is 5 MiB and the buffer interval is 5 minutes.
```python
# Increase the buffer interval and size to 10 minutes and 8 MiB, respectively
# bucket: s3.Bucket
destination = destinations.S3Bucket(bucket,
buffering_interval=Duration.minutes(10),
buffering_size=Size.mebibytes(8)
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=destination
)
```
See: [Data Delivery Frequency](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#frequency)
in the *Amazon Data Firehose Developer Guide*.
Zero buffering, where Amazon Data Firehose stream can be configured to not buffer data before delivery, is supported by
setting the "buffer interval" to 0.
```python
# Setup zero buffering
# bucket: s3.Bucket
destination = destinations.S3Bucket(bucket,
buffering_interval=Duration.seconds(0)
)
firehose.DeliveryStream(self, "ZeroBufferDeliveryStream",
destination=destination
)
```
See: [Buffering Hints](https://docs.aws.amazon.com/firehose/latest/dev/buffering-hints.html).
## Destination Encryption
Your data can be automatically encrypted when it is delivered to S3 as a final or an
intermediary/backup destination. Amazon Data Firehose supports Amazon S3 server-side
encryption with AWS Key Management Service (AWS KMS) for encrypting delivered data in
Amazon S3. You can choose to not encrypt the data or to encrypt with a key from the list
of AWS KMS keys that you own. For more information,
see [Protecting Data Using Server-Side Encryption with AWS KMS–Managed Keys (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html).
By default, encryption isn’t directly enabled on the delivery stream; instead, it uses the default encryption settings of the destination S3 bucket.
```python
# bucket: s3.Bucket
# key: kms.Key
destination = destinations.S3Bucket(bucket,
encryption_key=key
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=destination
)
```
## Backup
A delivery stream can be configured to back up data to S3 that it attempted to deliver to
the configured destination. Backed up data can be all the data that the delivery stream
attempted to deliver or just data that it failed to deliver (Redshift and S3 destinations
can only back up all data). CDK can create a new S3 bucket where it will back up data, or
you can provide a bucket where data will be backed up. You can also provide a prefix under
which your backed-up data will be placed within the bucket. By default, source data is not
backed up to S3.
```python
# Enable backup of all source records (to an S3 bucket created by CDK).
# bucket: s3.Bucket
# Explicitly provide an S3 bucket to which all source records will be backed up.
# backup_bucket: s3.Bucket
firehose.DeliveryStream(self, "Delivery Stream Backup All",
destination=
destinations.S3Bucket(bucket,
s3_backup=destinations.DestinationS3BackupProps(
mode=destinations.BackupMode.ALL
)
)
)
firehose.DeliveryStream(self, "Delivery Stream Backup All Explicit Bucket",
destination=
destinations.S3Bucket(bucket,
s3_backup=destinations.DestinationS3BackupProps(
bucket=backup_bucket
)
)
)
# Explicitly provide an S3 prefix under which all source records will be backed up.
firehose.DeliveryStream(self, "Delivery Stream Backup All Explicit Prefix",
destination=
destinations.S3Bucket(bucket,
s3_backup=destinations.DestinationS3BackupProps(
mode=destinations.BackupMode.ALL,
data_output_prefix="mybackup"
)
)
)
```
If any Data Processing or Transformation is configured on your Delivery Stream, the source
records will be backed up in their original format.
## Data Processing/Transformation
Data can be transformed before being delivered to destinations. There are two types of
data processing for delivery streams: record transformation with AWS Lambda, and record
format conversion using a schema stored in an AWS Glue table. If both types of data
processing are configured, then the Lambda transformation is performed first. By default,
no data processing occurs. This construct library currently only supports data
transformation with AWS Lambda. See [#15501](https://github.com/aws/aws-cdk/issues/15501)
to track the status of adding support for record format conversion.
### Data transformation with AWS Lambda
To transform the data, Amazon Data Firehose will call a Lambda function that you provide
and deliver the data returned in place of the source record. The function must return a
result that contains records in a specific format, including the following fields:
* `recordId` -- the ID of the input record that corresponds the results.
* `result` -- the status of the transformation of the record: "Ok" (success), "Dropped"
(not processed intentionally), or "ProcessingFailed" (not processed due to an error).
* `data` -- the transformed data, Base64-encoded.
The data is buffered up to 1 minute and up to 3 MiB by default before being sent to the
function, but can be configured using `bufferInterval` and `bufferSize`
in the processor configuration (see: [Buffering](#buffering)). If the function invocation
fails due to a network timeout or because of hitting an invocation limit, the invocation
is retried 3 times by default, but can be configured using `retries` in the processor
configuration.
```python
# bucket: s3.Bucket
# Provide a Lambda function that will transform records before delivery, with custom
# buffering and retry configuration
lambda_function = lambda_.Function(self, "Processor",
runtime=lambda_.Runtime.NODEJS_LATEST,
handler="index.handler",
code=lambda_.Code.from_asset(path.join(__dirname, "process-records"))
)
lambda_processor = firehose.LambdaFunctionProcessor(lambda_function,
buffer_interval=Duration.minutes(5),
buffer_size=Size.mebibytes(5),
retries=5
)
s3_destination = destinations.S3Bucket(bucket,
processor=lambda_processor
)
firehose.DeliveryStream(self, "Delivery Stream",
destination=s3_destination
)
```
```python
import path as path
import aws_cdk.aws_kinesisfirehose_alpha as firehose
import aws_cdk.aws_kms as kms
import aws_cdk.aws_lambda_nodejs as lambdanodejs
import aws_cdk.aws_logs as logs
import aws_cdk.aws_s3 as s3
import aws_cdk as cdk
import aws_cdk.aws_kinesisfirehose_destinations_alpha as destinations
app = cdk.App()
stack = cdk.Stack(app, "aws-cdk-firehose-delivery-stream-s3-all-properties")
bucket = s3.Bucket(stack, "Bucket",
removal_policy=cdk.RemovalPolicy.DESTROY,
auto_delete_objects=True
)
backup_bucket = s3.Bucket(stack, "BackupBucket",
removal_policy=cdk.RemovalPolicy.DESTROY,
auto_delete_objects=True
)
log_group = logs.LogGroup(stack, "LogGroup",
removal_policy=cdk.RemovalPolicy.DESTROY
)
data_processor_function = lambdanodejs.NodejsFunction(stack, "DataProcessorFunction",
entry=path.join(__dirname, "lambda-data-processor.js"),
timeout=cdk.Duration.minutes(1)
)
processor = firehose.LambdaFunctionProcessor(data_processor_function,
buffer_interval=cdk.Duration.seconds(60),
buffer_size=cdk.Size.mebibytes(1),
retries=1
)
key = kms.Key(stack, "Key",
removal_policy=cdk.RemovalPolicy.DESTROY
)
backup_key = kms.Key(stack, "BackupKey",
removal_policy=cdk.RemovalPolicy.DESTROY
)
firehose.DeliveryStream(stack, "Delivery Stream",
destination=destinations.S3Bucket(bucket,
logging_config=destinations.EnableLogging(log_group),
processor=processor,
compression=destinations.Compression.GZIP,
data_output_prefix="regularPrefix",
error_output_prefix="errorPrefix",
buffering_interval=cdk.Duration.seconds(60),
buffering_size=cdk.Size.mebibytes(1),
encryption_key=key,
s3_backup=destinations.DestinationS3BackupProps(
mode=destinations.BackupMode.ALL,
bucket=backup_bucket,
compression=destinations.Compression.ZIP,
data_output_prefix="backupPrefix",
error_output_prefix="backupErrorPrefix",
buffering_interval=cdk.Duration.seconds(60),
buffering_size=cdk.Size.mebibytes(1),
encryption_key=backup_key
)
)
)
firehose.DeliveryStream(stack, "ZeroBufferingDeliveryStream",
destination=destinations.S3Bucket(bucket,
compression=destinations.Compression.GZIP,
data_output_prefix="regularPrefix",
error_output_prefix="errorPrefix",
buffering_interval=cdk.Duration.seconds(0)
)
)
app.synth()
```
See: [Data Transformation](https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html)
in the *Amazon Data Firehose Developer Guide*.
## Specifying an IAM role
The DeliveryStream class automatically creates IAM service roles with all the minimum
necessary permissions for Amazon Data Firehose to access the resources referenced by your
delivery stream. One service role is created for the delivery stream that allows Amazon
Data Firehose to read from a Kinesis data stream (if one is configured as the delivery
stream source) and for server-side encryption. Note that if the DeliveryStream is created
without specifying a `source` or `encryptionKey`, this role is not created as it is not needed.
Another service role is created for each destination, which gives Amazon Data Firehose write
access to the destination resource, as well as the ability to invoke data transformers and
read schemas for record format conversion. If you wish, you may specify your own IAM role for
either the delivery stream or the destination service role, or both. It must have the correct
trust policy (it must allow Amazon Data Firehose to assume it) or delivery stream creation or
data delivery will fail. Other required permissions to destination resources, encryption keys, etc.,
will be provided automatically.
```python
# Specify the roles created above when defining the destination and delivery stream.
# bucket: s3.Bucket
# Create service roles for the delivery stream and destination.
# These can be used for other purposes and granted access to different resources.
# They must include the Amazon Data Firehose service principal in their trust policies.
# Two separate roles are shown below, but the same role can be used for both purposes.
delivery_stream_role = iam.Role(self, "Delivery Stream Role",
assumed_by=iam.ServicePrincipal("firehose.amazonaws.com")
)
destination_role = iam.Role(self, "Destination Role",
assumed_by=iam.ServicePrincipal("firehose.amazonaws.com")
)
destination = destinations.S3Bucket(bucket, role=destination_role)
firehose.DeliveryStream(self, "Delivery Stream",
destination=destination,
role=delivery_stream_role
)
```
See [Controlling Access](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html)
in the *Amazon Data Firehose Developer Guide*.
## Granting application access to a delivery stream
IAM roles, users or groups which need to be able to work with delivery streams should be
granted IAM permissions.
Any object that implements the `IGrantable` interface (i.e., has an associated principal)
can be granted permissions to a delivery stream by calling:
* `grantPutRecords(principal)` - grants the principal the ability to put records onto the
delivery stream
* `grant(principal, ...actions)` - grants the principal permission to a custom set of
actions
```python
# Give the role permissions to write data to the delivery stream
# delivery_stream: firehose.DeliveryStream
lambda_role = iam.Role(self, "Role",
assumed_by=iam.ServicePrincipal("lambda.amazonaws.com")
)
delivery_stream.grant_put_records(lambda_role)
```
The following write permissions are provided to a service principal by the
`grantPutRecords()` method:
* `firehose:PutRecord`
* `firehose:PutRecordBatch`
## Granting a delivery stream access to a resource
Conversely to the above, Amazon Data Firehose requires permissions in order for delivery
streams to interact with resources that you own. For example, if an S3 bucket is specified
as a destination of a delivery stream, the delivery stream must be granted permissions to
put and get objects from the bucket. When using the built-in AWS service destinations
found in the `@aws-cdk/aws-kinesisfirehose-destinations-alpha` module, the CDK grants the
permissions automatically. However, custom or third-party destinations may require custom
permissions. In this case, use the delivery stream as an `IGrantable`, as follows:
```python
# delivery_stream: firehose.DeliveryStream
fn = lambda_.Function(self, "Function",
code=lambda_.Code.from_inline("exports.handler = (event) => {}"),
runtime=lambda_.Runtime.NODEJS_LATEST,
handler="index.handler"
)
fn.grant_invoke(delivery_stream)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/aws/aws-cdk",
"name": "aws-cdk.aws-kinesisfirehose-alpha",
"maintainer": null,
"docs_url": null,
"requires_python": "~=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Amazon Web Services",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/8c/20/45098d412c8e64d0b1ea13513cc321b0167f66f774883a6a56301c17fbac/aws_cdk_aws_kinesisfirehose_alpha-2.170.0a0.tar.gz",
"platform": null,
"description": "# Amazon Data Firehose Construct Library\n\n<!--BEGIN STABILITY BANNER-->---\n\n\n\n\n> The APIs of higher level constructs in this module are in **developer preview** before they\n> become stable. We will only make breaking changes to address unforeseen API issues. Therefore,\n> these APIs are not subject to [Semantic Versioning](https://semver.org/), and breaking changes\n> will be announced in release notes. This means that while you may use them, you may need to\n> update your source code when upgrading to a newer version of this package.\n\n---\n<!--END STABILITY BANNER-->\n\n[Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html), [formerly known as Amazon Kinesis Data Firehose](https://aws.amazon.com/about-aws/whats-new/2024/02/amazon-data-firehose-formerly-kinesis-data-firehose/),\nis a service for fully-managed delivery of real-time streaming data to storage services\nsuch as Amazon S3, Amazon Redshift, Amazon Elasticsearch, Splunk, or any custom HTTP\nendpoint or third-party services such as Datadog, Dynatrace, LogicMonitor, MongoDB, New\nRelic, and Sumo Logic.\n\nAmazon Data Firehose delivery streams are distinguished from Kinesis data streams in\ntheir models of consumption. Whereas consumers read from a data stream by actively pulling\ndata from the stream, a delivery stream pushes data to its destination on a regular\ncadence. This means that data streams are intended to have consumers that do on-demand\nprocessing, like AWS Lambda or Amazon EC2. On the other hand, delivery streams are\nintended to have destinations that are sources for offline processing and analytics, such\nas Amazon S3 and Amazon Redshift.\n\nThis module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk)\nproject. It allows you to define Amazon Data Firehose delivery streams.\n\n## Defining a Delivery Stream\n\nIn order to define a Delivery Stream, you must specify a destination. An S3 bucket can be\nused as a destination. Currently the CDK supports only S3 as a destination which is covered [below](#destinations).\n\n```python\nbucket = s3.Bucket(self, \"Bucket\")\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destinations.S3Bucket(bucket)\n)\n```\n\nThe above example defines the following resources:\n\n* An S3 bucket\n* An Amazon Data Firehose delivery stream with Direct PUT as the source and CloudWatch\n error logging turned on.\n* An IAM role which gives the delivery stream permission to write to the S3 bucket.\n\n## Sources\n\nAn Amazon Data Firehose delivery stream can accept data from three main sources: Kinesis Data Streams, Managed Streaming for Apache Kafka (MSK), or via a \"direct put\" (API calls). Currently only Kinesis Data Streams and direct put are supported in the CDK.\n\nSee: [Sending Data to a Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n### Kinesis Data Stream\n\nA delivery stream can read directly from a Kinesis data stream as a consumer of the data\nstream. Configure this behaviour by passing in a data stream in the `source`\nproperty via the `KinesisStreamSource` class when constructing a delivery stream:\n\n```python\n# destination: firehose.IDestination\n\nsource_stream = kinesis.Stream(self, \"Source Stream\")\n\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n source=firehose.KinesisStreamSource(source_stream),\n destination=destination\n)\n```\n\n### Direct Put\n\nData must be provided via \"direct put\", ie., by using a `PutRecord` or\n`PutRecordBatch` API call. There are a number of ways of doing so, such as:\n\n* Kinesis Agent: a standalone Java application that monitors and delivers files while\n handling file rotation, checkpointing, and retries. See: [Writing to Amazon Data Firehose Using Kinesis Agent](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html)\n in the *Amazon Data Firehose Developer Guide*.\n* AWS SDK: a general purpose solution that allows you to deliver data to a delivery stream\n from anywhere using Java, .NET, Node.js, Python, or Ruby. See: [Writing to Amazon Data Firehose Using the AWS SDK](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html)\n in the *Amazon Data Firehose Developer Guide*.\n* CloudWatch Logs: subscribe to a log group and receive filtered log events directly into\n a delivery stream. See: [logs-destinations](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-logs-destinations-readme.html).\n* Eventbridge: add an event rule target to send events to a delivery stream based on the\n rule filtering. See: [events-targets](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-targets-readme.html).\n* SNS: add a subscription to send all notifications from the topic to a delivery\n stream. See: [sns-subscriptions](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-sns-subscriptions-readme.html).\n* IoT: add an action to an IoT rule to send various IoT information to a delivery stream\n\n## Destinations\n\nAmazon Data Firehose supports multiple AWS and third-party services as destinations, including Amazon S3, Amazon Redshift, and more. You can find the full list of supported destination [here](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html).\n\nCurrently in the AWS CDK, only S3 is implemented as an L2 construct destination. Other destinations can still be configured using L1 constructs. See [kinesisfirehose-destinations](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-kinesisfirehose-destinations-readme.html)\nfor the implementations of these destinations.\n\n### S3\n\nDefining a delivery stream with an S3 bucket destination:\n\n```python\n# bucket: s3.Bucket\n\ns3_destination = destinations.S3Bucket(bucket)\n\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=s3_destination\n)\n```\n\nThe S3 destination also supports custom dynamic prefixes. `dataOutputPrefix`\nwill be used for files successfully delivered to S3. `errorOutputPrefix` will be added to\nfailed records before writing them to S3.\n\n```python\n# bucket: s3.Bucket\n\ns3_destination = destinations.S3Bucket(bucket,\n data_output_prefix=\"myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}\",\n error_output_prefix=\"myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}\"\n)\n```\n\nSee: [Custom S3 Prefixes](https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n## Server-side Encryption\n\nEnabling server-side encryption (SSE) requires Amazon Data Firehose to encrypt all data\nsent to delivery stream when it is stored at rest. This means that data is encrypted\nbefore being written to the service's internal storage layer and decrypted after it is\nreceived from the internal storage layer. The service manages keys and cryptographic\noperations so that sources and destinations do not need to, as the data is encrypted and\ndecrypted at the boundaries of the service (i.e., before the data is delivered to a\ndestination). By default, delivery streams do not have SSE enabled.\n\nThe Key Management Service keys (KMS keys) used for SSE can either be AWS-owned or\ncustomer-managed. AWS-owned KMS keys are created, owned and managed by AWS for use in\nmultiple AWS accounts. As a customer, you cannot view, use, track, or manage these keys,\nand you are not charged for their use. On the other hand, customer-managed KMS keys are\ncreated and owned within your account and managed entirely by you. As a customer, you are\nresponsible for managing access, rotation, aliases, and deletion for these keys, and you\nare changed for their use.\n\nSee: [AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#kms_keys)\nin the *KMS Developer Guide*.\n\n```python\n# destination: firehose.IDestination\n# SSE with an customer-managed key that is explicitly specified\n# key: kms.Key\n\n\n# SSE with an AWS-owned key\nfirehose.DeliveryStream(self, \"Delivery Stream with AWS Owned Key\",\n encryption=firehose.StreamEncryption.aws_owned_key(),\n destination=destination\n)\n# SSE with an customer-managed key that is created automatically by the CDK\nfirehose.DeliveryStream(self, \"Delivery Stream with Customer Managed Key\",\n encryption=firehose.StreamEncryption.customer_managed_key(),\n destination=destination\n)\nfirehose.DeliveryStream(self, \"Delivery Stream with Customer Managed and Provided Key\",\n encryption=firehose.StreamEncryption.customer_managed_key(key),\n destination=destination\n)\n```\n\nSee: [Data Protection](https://docs.aws.amazon.com/firehose/latest/dev/encryption.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n## Monitoring\n\nAmazon Data Firehose is integrated with CloudWatch, so you can monitor the performance of\nyour delivery streams via logs and metrics.\n\n### Logs\n\nAmazon Data Firehose will send logs to CloudWatch when data transformation or data\ndelivery fails. The CDK will enable logging by default and create a CloudWatch LogGroup\nand LogStream with default settings for your Delivery Stream.\n\nWhen creating a destination, you can provide an `ILoggingConfig`, which can either be an `EnableLogging` or `DisableLogging` instance.\nIf you use `EnableLogging`, the CDK will create a CloudWatch LogGroup and LogStream with all CloudFormation default settings for you, or you can optionally\nspecify your own log group to be used for capturing and storing log events. For example:\n\n```python\nimport aws_cdk.aws_logs as logs\n# bucket: s3.Bucket\n\n\nlog_group = logs.LogGroup(self, \"Log Group\")\ndestination = destinations.S3Bucket(bucket,\n logging_config=destinations.EnableLogging(log_group)\n)\n\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destination\n)\n```\n\nLogging can also be disabled:\n\n```python\n# bucket: s3.Bucket\n\ndestination = destinations.S3Bucket(bucket,\n logging_config=destinations.DisableLogging()\n)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destination\n)\n```\n\nSee: [Monitoring using CloudWatch Logs](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n### Metrics\n\nAmazon Data Firehose sends metrics to CloudWatch so that you can collect and analyze the\nperformance of the delivery stream, including data delivery, data ingestion, data\ntransformation, format conversion, API usage, encryption, and resource usage. You can then\nuse CloudWatch alarms to alert you, for example, when data freshness (the age of the\noldest record in the delivery stream) exceeds the buffering limit (indicating that data is\nnot being delivered to your destination), or when the rate of incoming records exceeds the\nlimit of records per second (indicating data is flowing into your delivery stream faster\nthan it is configured to process).\n\nCDK provides methods for accessing delivery stream metrics with default configuration,\nsuch as `metricIncomingBytes`, and `metricIncomingRecords` (see [`IDeliveryStream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.IDeliveryStream.html)\nfor a full list). CDK also provides a generic `metric` method that can be used to produce\nmetric configurations for any metric provided by Amazon Data Firehose; the configurations\nare pre-populated with the correct dimensions for the delivery stream.\n\n```python\nimport aws_cdk.aws_cloudwatch as cloudwatch\n\n# delivery_stream: firehose.DeliveryStream\n\n\n# Alarm that triggers when the per-second average of incoming bytes exceeds 90% of the current service limit\nincoming_bytes_percent_of_limit = cloudwatch.MathExpression(\n expression=\"incomingBytes / 300 / bytePerSecLimit\",\n using_metrics={\n \"incoming_bytes\": delivery_stream.metric_incoming_bytes(statistic=cloudwatch.Statistic.SUM),\n \"byte_per_sec_limit\": delivery_stream.metric(\"BytesPerSecondLimit\")\n }\n)\n\ncloudwatch.Alarm(self, \"Alarm\",\n metric=incoming_bytes_percent_of_limit,\n threshold=0.9,\n evaluation_periods=3\n)\n```\n\nSee: [Monitoring Using CloudWatch Metrics](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n## Compression\n\nYour data can automatically be compressed when it is delivered to S3 as either a final or\nan intermediary/backup destination. Supported compression formats are: gzip, Snappy,\nHadoop-compatible Snappy, and ZIP, except for Redshift destinations, where Snappy\n(regardless of Hadoop-compatibility) and ZIP are not supported. By default, data is\ndelivered to S3 without compression.\n\n```python\n# Compress data delivered to S3 using Snappy\n# bucket: s3.Bucket\n\ns3_destination = destinations.S3Bucket(bucket,\n compression=destinations.Compression.SNAPPY\n)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=s3_destination\n)\n```\n\n## Buffering\n\nIncoming data is buffered before it is delivered to the specified destination. The\ndelivery stream will wait until the amount of incoming data has exceeded some threshold\n(the \"buffer size\") or until the time since the last data delivery occurred exceeds some\nthreshold (the \"buffer interval\"), whichever happens first. You can configure these\nthresholds based on the capabilities of the destination and your use-case. By default, the\nbuffer size is 5 MiB and the buffer interval is 5 minutes.\n\n```python\n# Increase the buffer interval and size to 10 minutes and 8 MiB, respectively\n# bucket: s3.Bucket\n\ndestination = destinations.S3Bucket(bucket,\n buffering_interval=Duration.minutes(10),\n buffering_size=Size.mebibytes(8)\n)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destination\n)\n```\n\nSee: [Data Delivery Frequency](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#frequency)\nin the *Amazon Data Firehose Developer Guide*.\n\nZero buffering, where Amazon Data Firehose stream can be configured to not buffer data before delivery, is supported by\nsetting the \"buffer interval\" to 0.\n\n```python\n# Setup zero buffering\n# bucket: s3.Bucket\n\ndestination = destinations.S3Bucket(bucket,\n buffering_interval=Duration.seconds(0)\n)\nfirehose.DeliveryStream(self, \"ZeroBufferDeliveryStream\",\n destination=destination\n)\n```\n\nSee: [Buffering Hints](https://docs.aws.amazon.com/firehose/latest/dev/buffering-hints.html).\n\n## Destination Encryption\n\nYour data can be automatically encrypted when it is delivered to S3 as a final or an\nintermediary/backup destination. Amazon Data Firehose supports Amazon S3 server-side\nencryption with AWS Key Management Service (AWS KMS) for encrypting delivered data in\nAmazon S3. You can choose to not encrypt the data or to encrypt with a key from the list\nof AWS KMS keys that you own. For more information,\nsee [Protecting Data Using Server-Side Encryption with AWS KMS\u2013Managed Keys (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html).\nBy default, encryption isn\u2019t directly enabled on the delivery stream; instead, it uses the default encryption settings of the destination S3 bucket.\n\n```python\n# bucket: s3.Bucket\n# key: kms.Key\n\ndestination = destinations.S3Bucket(bucket,\n encryption_key=key\n)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destination\n)\n```\n\n## Backup\n\nA delivery stream can be configured to back up data to S3 that it attempted to deliver to\nthe configured destination. Backed up data can be all the data that the delivery stream\nattempted to deliver or just data that it failed to deliver (Redshift and S3 destinations\ncan only back up all data). CDK can create a new S3 bucket where it will back up data, or\nyou can provide a bucket where data will be backed up. You can also provide a prefix under\nwhich your backed-up data will be placed within the bucket. By default, source data is not\nbacked up to S3.\n\n```python\n# Enable backup of all source records (to an S3 bucket created by CDK).\n# bucket: s3.Bucket\n# Explicitly provide an S3 bucket to which all source records will be backed up.\n# backup_bucket: s3.Bucket\n\nfirehose.DeliveryStream(self, \"Delivery Stream Backup All\",\n destination=\n destinations.S3Bucket(bucket,\n s3_backup=destinations.DestinationS3BackupProps(\n mode=destinations.BackupMode.ALL\n )\n )\n)\nfirehose.DeliveryStream(self, \"Delivery Stream Backup All Explicit Bucket\",\n destination=\n destinations.S3Bucket(bucket,\n s3_backup=destinations.DestinationS3BackupProps(\n bucket=backup_bucket\n )\n )\n)\n# Explicitly provide an S3 prefix under which all source records will be backed up.\nfirehose.DeliveryStream(self, \"Delivery Stream Backup All Explicit Prefix\",\n destination=\n destinations.S3Bucket(bucket,\n s3_backup=destinations.DestinationS3BackupProps(\n mode=destinations.BackupMode.ALL,\n data_output_prefix=\"mybackup\"\n )\n )\n)\n```\n\nIf any Data Processing or Transformation is configured on your Delivery Stream, the source\nrecords will be backed up in their original format.\n\n## Data Processing/Transformation\n\nData can be transformed before being delivered to destinations. There are two types of\ndata processing for delivery streams: record transformation with AWS Lambda, and record\nformat conversion using a schema stored in an AWS Glue table. If both types of data\nprocessing are configured, then the Lambda transformation is performed first. By default,\nno data processing occurs. This construct library currently only supports data\ntransformation with AWS Lambda. See [#15501](https://github.com/aws/aws-cdk/issues/15501)\nto track the status of adding support for record format conversion.\n\n### Data transformation with AWS Lambda\n\nTo transform the data, Amazon Data Firehose will call a Lambda function that you provide\nand deliver the data returned in place of the source record. The function must return a\nresult that contains records in a specific format, including the following fields:\n\n* `recordId` -- the ID of the input record that corresponds the results.\n* `result` -- the status of the transformation of the record: \"Ok\" (success), \"Dropped\"\n (not processed intentionally), or \"ProcessingFailed\" (not processed due to an error).\n* `data` -- the transformed data, Base64-encoded.\n\nThe data is buffered up to 1 minute and up to 3 MiB by default before being sent to the\nfunction, but can be configured using `bufferInterval` and `bufferSize`\nin the processor configuration (see: [Buffering](#buffering)). If the function invocation\nfails due to a network timeout or because of hitting an invocation limit, the invocation\nis retried 3 times by default, but can be configured using `retries` in the processor\nconfiguration.\n\n```python\n# bucket: s3.Bucket\n# Provide a Lambda function that will transform records before delivery, with custom\n# buffering and retry configuration\nlambda_function = lambda_.Function(self, \"Processor\",\n runtime=lambda_.Runtime.NODEJS_LATEST,\n handler=\"index.handler\",\n code=lambda_.Code.from_asset(path.join(__dirname, \"process-records\"))\n)\nlambda_processor = firehose.LambdaFunctionProcessor(lambda_function,\n buffer_interval=Duration.minutes(5),\n buffer_size=Size.mebibytes(5),\n retries=5\n)\ns3_destination = destinations.S3Bucket(bucket,\n processor=lambda_processor\n)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=s3_destination\n)\n```\n\n```python\nimport path as path\nimport aws_cdk.aws_kinesisfirehose_alpha as firehose\nimport aws_cdk.aws_kms as kms\nimport aws_cdk.aws_lambda_nodejs as lambdanodejs\nimport aws_cdk.aws_logs as logs\nimport aws_cdk.aws_s3 as s3\nimport aws_cdk as cdk\nimport aws_cdk.aws_kinesisfirehose_destinations_alpha as destinations\n\napp = cdk.App()\n\nstack = cdk.Stack(app, \"aws-cdk-firehose-delivery-stream-s3-all-properties\")\n\nbucket = s3.Bucket(stack, \"Bucket\",\n removal_policy=cdk.RemovalPolicy.DESTROY,\n auto_delete_objects=True\n)\n\nbackup_bucket = s3.Bucket(stack, \"BackupBucket\",\n removal_policy=cdk.RemovalPolicy.DESTROY,\n auto_delete_objects=True\n)\nlog_group = logs.LogGroup(stack, \"LogGroup\",\n removal_policy=cdk.RemovalPolicy.DESTROY\n)\n\ndata_processor_function = lambdanodejs.NodejsFunction(stack, \"DataProcessorFunction\",\n entry=path.join(__dirname, \"lambda-data-processor.js\"),\n timeout=cdk.Duration.minutes(1)\n)\n\nprocessor = firehose.LambdaFunctionProcessor(data_processor_function,\n buffer_interval=cdk.Duration.seconds(60),\n buffer_size=cdk.Size.mebibytes(1),\n retries=1\n)\n\nkey = kms.Key(stack, \"Key\",\n removal_policy=cdk.RemovalPolicy.DESTROY\n)\n\nbackup_key = kms.Key(stack, \"BackupKey\",\n removal_policy=cdk.RemovalPolicy.DESTROY\n)\n\nfirehose.DeliveryStream(stack, \"Delivery Stream\",\n destination=destinations.S3Bucket(bucket,\n logging_config=destinations.EnableLogging(log_group),\n processor=processor,\n compression=destinations.Compression.GZIP,\n data_output_prefix=\"regularPrefix\",\n error_output_prefix=\"errorPrefix\",\n buffering_interval=cdk.Duration.seconds(60),\n buffering_size=cdk.Size.mebibytes(1),\n encryption_key=key,\n s3_backup=destinations.DestinationS3BackupProps(\n mode=destinations.BackupMode.ALL,\n bucket=backup_bucket,\n compression=destinations.Compression.ZIP,\n data_output_prefix=\"backupPrefix\",\n error_output_prefix=\"backupErrorPrefix\",\n buffering_interval=cdk.Duration.seconds(60),\n buffering_size=cdk.Size.mebibytes(1),\n encryption_key=backup_key\n )\n )\n)\n\nfirehose.DeliveryStream(stack, \"ZeroBufferingDeliveryStream\",\n destination=destinations.S3Bucket(bucket,\n compression=destinations.Compression.GZIP,\n data_output_prefix=\"regularPrefix\",\n error_output_prefix=\"errorPrefix\",\n buffering_interval=cdk.Duration.seconds(0)\n )\n)\n\napp.synth()\n```\n\nSee: [Data Transformation](https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n## Specifying an IAM role\n\nThe DeliveryStream class automatically creates IAM service roles with all the minimum\nnecessary permissions for Amazon Data Firehose to access the resources referenced by your\ndelivery stream. One service role is created for the delivery stream that allows Amazon\nData Firehose to read from a Kinesis data stream (if one is configured as the delivery\nstream source) and for server-side encryption. Note that if the DeliveryStream is created\nwithout specifying a `source` or `encryptionKey`, this role is not created as it is not needed.\n\nAnother service role is created for each destination, which gives Amazon Data Firehose write\naccess to the destination resource, as well as the ability to invoke data transformers and\nread schemas for record format conversion. If you wish, you may specify your own IAM role for\neither the delivery stream or the destination service role, or both. It must have the correct\ntrust policy (it must allow Amazon Data Firehose to assume it) or delivery stream creation or\ndata delivery will fail. Other required permissions to destination resources, encryption keys, etc.,\nwill be provided automatically.\n\n```python\n# Specify the roles created above when defining the destination and delivery stream.\n# bucket: s3.Bucket\n# Create service roles for the delivery stream and destination.\n# These can be used for other purposes and granted access to different resources.\n# They must include the Amazon Data Firehose service principal in their trust policies.\n# Two separate roles are shown below, but the same role can be used for both purposes.\ndelivery_stream_role = iam.Role(self, \"Delivery Stream Role\",\n assumed_by=iam.ServicePrincipal(\"firehose.amazonaws.com\")\n)\ndestination_role = iam.Role(self, \"Destination Role\",\n assumed_by=iam.ServicePrincipal(\"firehose.amazonaws.com\")\n)\ndestination = destinations.S3Bucket(bucket, role=destination_role)\nfirehose.DeliveryStream(self, \"Delivery Stream\",\n destination=destination,\n role=delivery_stream_role\n)\n```\n\nSee [Controlling Access](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html)\nin the *Amazon Data Firehose Developer Guide*.\n\n## Granting application access to a delivery stream\n\nIAM roles, users or groups which need to be able to work with delivery streams should be\ngranted IAM permissions.\n\nAny object that implements the `IGrantable` interface (i.e., has an associated principal)\ncan be granted permissions to a delivery stream by calling:\n\n* `grantPutRecords(principal)` - grants the principal the ability to put records onto the\n delivery stream\n* `grant(principal, ...actions)` - grants the principal permission to a custom set of\n actions\n\n```python\n# Give the role permissions to write data to the delivery stream\n# delivery_stream: firehose.DeliveryStream\nlambda_role = iam.Role(self, \"Role\",\n assumed_by=iam.ServicePrincipal(\"lambda.amazonaws.com\")\n)\ndelivery_stream.grant_put_records(lambda_role)\n```\n\nThe following write permissions are provided to a service principal by the\n`grantPutRecords()` method:\n\n* `firehose:PutRecord`\n* `firehose:PutRecordBatch`\n\n## Granting a delivery stream access to a resource\n\nConversely to the above, Amazon Data Firehose requires permissions in order for delivery\nstreams to interact with resources that you own. For example, if an S3 bucket is specified\nas a destination of a delivery stream, the delivery stream must be granted permissions to\nput and get objects from the bucket. When using the built-in AWS service destinations\nfound in the `@aws-cdk/aws-kinesisfirehose-destinations-alpha` module, the CDK grants the\npermissions automatically. However, custom or third-party destinations may require custom\npermissions. In this case, use the delivery stream as an `IGrantable`, as follows:\n\n```python\n# delivery_stream: firehose.DeliveryStream\nfn = lambda_.Function(self, \"Function\",\n code=lambda_.Code.from_inline(\"exports.handler = (event) => {}\"),\n runtime=lambda_.Runtime.NODEJS_LATEST,\n handler=\"index.handler\"\n)\nfn.grant_invoke(delivery_stream)\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "The CDK Construct Library for AWS::KinesisFirehose",
"version": "2.170.0a0",
"project_urls": {
"Homepage": "https://github.com/aws/aws-cdk",
"Source": "https://github.com/aws/aws-cdk.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "002372937869d182682485f51afdda4a0beeb9faa170c8768bbef9f0c05a4c8f",
"md5": "5451474bb673480561dcbdc09106ed07",
"sha256": "dbb436ce43ccc8c05c1a2de91ef89a7a40aee2295cfb5c3530d8c399e104ac21"
},
"downloads": -1,
"filename": "aws_cdk.aws_kinesisfirehose_alpha-2.170.0a0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5451474bb673480561dcbdc09106ed07",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "~=3.8",
"size": 116710,
"upload_time": "2024-11-22T04:41:42",
"upload_time_iso_8601": "2024-11-22T04:41:42.570152Z",
"url": "https://files.pythonhosted.org/packages/00/23/72937869d182682485f51afdda4a0beeb9faa170c8768bbef9f0c05a4c8f/aws_cdk.aws_kinesisfirehose_alpha-2.170.0a0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8c2045098d412c8e64d0b1ea13513cc321b0167f66f774883a6a56301c17fbac",
"md5": "97fe0e37c4905596fc83b3d4a4cd70ec",
"sha256": "df1046fbd0d0f20bc88cbe60551d7e3c4ca61b9d6aa1d1b7ed5669e8c0f57f11"
},
"downloads": -1,
"filename": "aws_cdk_aws_kinesisfirehose_alpha-2.170.0a0.tar.gz",
"has_sig": false,
"md5_digest": "97fe0e37c4905596fc83b3d4a4cd70ec",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "~=3.8",
"size": 125374,
"upload_time": "2024-11-22T04:42:28",
"upload_time_iso_8601": "2024-11-22T04:42:28.371774Z",
"url": "https://files.pythonhosted.org/packages/8c/20/45098d412c8e64d0b1ea13513cc321b0167f66f774883a6a56301c17fbac/aws_cdk_aws_kinesisfirehose_alpha-2.170.0a0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-22 04:42:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aws",
"github_project": "aws-cdk",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "aws-cdk.aws-kinesisfirehose-alpha"
}
JSON
