# aws-kinesisfirehose-s3-and-kinesisanalytics module
<!--BEGIN STABILITY BANNER-->---
> All classes are under active development and subject to non-backward compatible changes or removal in any
> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model.
> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.
---
<!--END STABILITY BANNER-->
| **Reference Documentation**:| <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span>|
|:-------------|:-------------|
<div style="height:8px"></div>
| **Language** | **Package** |
|:-------------|-----------------|
| Python|`aws_solutions_constructs.aws_kinesisfirehose_s3_and_kinesisanalytics`|
| Typescript|`@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics`|
| Java|`software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics`|
## Overview
This AWS Solutions Construct implements an Amazon Kinesis Firehose delivery stream connected to an Amazon S3 bucket, and an Amazon Kinesis Analytics application.
Here is a minimal deployable pattern definition:
Typescript
```python
import { Construct } from 'constructs';
import { Stack, StackProps } from 'aws-cdk-lib';
import { KinesisFirehoseToAnalyticsAndS3 } from '@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics';
new KinesisFirehoseToAnalyticsAndS3(this, 'FirehoseToS3AndAnalyticsPattern', {
kinesisAnalyticsProps: {
inputs: [{
inputSchema: {
recordColumns: [{
name: 'ticker_symbol',
sqlType: 'VARCHAR(4)',
mapping: '$.ticker_symbol'
}, {
name: 'sector',
sqlType: 'VARCHAR(16)',
mapping: '$.sector'
}, {
name: 'change',
sqlType: 'REAL',
mapping: '$.change'
}, {
name: 'price',
sqlType: 'REAL',
mapping: '$.price'
}],
recordFormat: {
recordFormatType: 'JSON'
},
recordEncoding: 'UTF-8'
},
namePrefix: 'SOURCE_SQL_STREAM'
}]
}
});
```
Python
```python
from aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics import KinesisFirehoseToAnalyticsAndS3
from aws_cdk import (
aws_kinesisanalytics as kinesisanalytics,
Stack
)
from constructs import Construct
KinesisFirehoseToAnalyticsAndS3(self, 'FirehoseToS3AndAnalyticsPattern',
kinesis_analytics_props=kinesisanalytics.CfnApplicationProps(
inputs=[kinesisanalytics.CfnApplication.InputProperty(
input_schema=kinesisanalytics.CfnApplication.InputSchemaProperty(
record_columns=[kinesisanalytics.CfnApplication.RecordColumnProperty(
name='ticker_symbol',
sql_type='VARCHAR(4)',
mapping='$.ticker_symbol'
), kinesisanalytics.CfnApplication.RecordColumnProperty(
name='sector',
sql_type='VARCHAR(16)',
mapping='$.sector'
), kinesisanalytics.CfnApplication.RecordColumnProperty(
name='change',
sql_type='REAL',
mapping='$.change'
), kinesisanalytics.CfnApplication.RecordColumnProperty(
name='price',
sql_type='REAL',
mapping='$.price'
)],
record_format=kinesisanalytics.CfnApplication.RecordFormatProperty(
record_format_type='JSON'
),
record_encoding='UTF-8'
),
name_prefix='SOURCE_SQL_STREAM'
)]
)
)
```
Java
```java
import software.constructs.Construct;
import java.util.List;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.StackProps;
import software.amazon.awscdk.services.kinesisanalytics.*;
import software.amazon.awscdk.services.kinesisanalytics.CfnApplication.*;
import software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics.*;
new KinesisFirehoseToAnalyticsAndS3(this, "FirehoseToS3AndAnalyticsPattern",
new KinesisFirehoseToAnalyticsAndS3Props.Builder()
.kinesisAnalyticsProps(new CfnApplicationProps.Builder()
.inputs(List.of(new InputProperty.Builder()
.inputSchema(new InputSchemaProperty.Builder()
.recordColumns(List.of(
new RecordColumnProperty.Builder()
.name("ticker_symbol")
.sqlType("VARCHAR(4)")
.mapping("$.ticker_symbol")
.build(),
new RecordColumnProperty.Builder()
.name("sector")
.sqlType("VARCHAR(16)")
.mapping("$.sector")
.build(),
new RecordColumnProperty.Builder()
.name("change")
.sqlType("REAL")
.mapping("$.change")
.build(),
new RecordColumnProperty.Builder()
.name("price")
.sqlType("REAL")
.mapping("$.price")
.build()))
.recordFormat(new RecordFormatProperty.Builder()
.recordFormatType("JSON")
.build())
.recordEncoding("UTF-8")
.build())
.namePrefix("SOURCE_SQL_STREAM")
.build()))
.build())
.build());
```
## Pattern Construct Props
| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|kinesisFirehoseProps?|[`kinesisFirehose.CfnDeliveryStreamProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisfirehose.CfnDeliveryStreamProps.html)|Optional user-provided props to override the default props for the Kinesis Firehose delivery stream.|
|kinesisAnalyticsProps?|[`kinesisAnalytics.CfnApplicationProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisanalytics.CfnApplicationProps.html)|Optional user-provided props to override the default props for the Kinesis Analytics application.|
|existingBucketObj?|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.IBucket.html)|Existing instance of S3 Bucket object. If this is provided, then also providing bucketProps is an error. |
|bucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.BucketProps.html)|User provided props to override the default props for the S3 Bucket.|
|logGroupProps?|[`logs.LogGroupProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.LogGroupProps.html)|User provided props to override the default props for for the CloudWatchLogs LogGroup.|
|loggingBucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.BucketProps.html)|Optional user provided props to override the default props for the S3 Logging Bucket.|
|logS3AccessLogs?| boolean|Whether to turn on Access Logging for the S3 bucket. Creates an S3 bucket with associated storage costs for the logs. Enabling Access Logging is a best practice. default - true|
## Pattern Properties
| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|kinesisAnalytics|[`kinesisAnalytics.CfnApplication`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisanalytics.CfnApplication.html)|Returns an instance of the Kinesis Analytics application created by the pattern.|
|kinesisFirehose|[`kinesisFirehose.CfnDeliveryStream`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisfirehose.CfnDeliveryStream.html)|Returns an instance of the Kinesis Firehose delivery stream created by the pattern.|
|kinesisFirehoseRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_iam.Role.html)|Returns an instance of the iam.Role created by the construct for Kinesis Data Firehose delivery stream.|
|kinesisFirehoseLogGroup|[`logs.LogGroup`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.LogGroup.html)|Returns an instance of the LogGroup created by the construct for Kinesis Data Firehose delivery stream|
|s3Bucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html)|Returns an instance of the S3 bucket created by the pattern.|
|s3LoggingBucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct as the logging bucket for the primary bucket.|
|s3BucketInterface|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.IBucket.html)|Returns an instance of s3.IBucket created by the construct.|
## Default settings
Out of the box implementation of the Construct without any override will set the following defaults:
### Amazon Kinesis Firehose
* Enable CloudWatch logging for Kinesis Firehose
* Configure least privilege access IAM role for Amazon Kinesis Firehose
### Amazon S3 Bucket
* Configure Access logging for S3 Bucket
* Enable server-side encryption for S3 Bucket using AWS managed KMS Key
* Enforce encryption of data in transit
* Turn on the versioning for S3 Bucket
* Don't allow public access for S3 Bucket
* Retain the S3 Bucket when deleting the CloudFormation stack
* Applies Lifecycle rule to move noncurrent object versions to Glacier storage after 90 days
### Amazon Kinesis Data Analytics
* Configure least privilege access IAM role for Amazon Kinesis Analytics
## Architecture
---
© Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
Raw data
{
"_id": null,
"home_page": "https://github.com/awslabs/aws-solutions-constructs.git",
"name": "aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics",
"maintainer": "",
"docs_url": null,
"requires_python": "~=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Amazon Web Services",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/d5/9d/7dc7ac8373dfd18525a527f20d784c78f3ba11f72fc48628c02b34ea0ed2/aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz",
"platform": null,
"description": "# aws-kinesisfirehose-s3-and-kinesisanalytics module\n\n<!--BEGIN STABILITY BANNER-->---\n\n\n\n\n> All classes are under active development and subject to non-backward compatible changes or removal in any\n> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model.\n> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.\n\n---\n<!--END STABILITY BANNER-->\n\n| **Reference Documentation**:| <span style=\"font-weight: normal\">https://docs.aws.amazon.com/solutions/latest/constructs/</span>|\n|:-------------|:-------------|\n\n<div style=\"height:8px\"></div>\n\n| **Language** | **Package** |\n|:-------------|-----------------|\n| Python|`aws_solutions_constructs.aws_kinesisfirehose_s3_and_kinesisanalytics`|\n| Typescript|`@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics`|\n| Java|`software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics`|\n\n## Overview\n\nThis AWS Solutions Construct implements an Amazon Kinesis Firehose delivery stream connected to an Amazon S3 bucket, and an Amazon Kinesis Analytics application.\n\nHere is a minimal deployable pattern definition:\n\nTypescript\n\n```python\nimport { Construct } from 'constructs';\nimport { Stack, StackProps } from 'aws-cdk-lib';\nimport { KinesisFirehoseToAnalyticsAndS3 } from '@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics';\n\nnew KinesisFirehoseToAnalyticsAndS3(this, 'FirehoseToS3AndAnalyticsPattern', {\n kinesisAnalyticsProps: {\n inputs: [{\n inputSchema: {\n recordColumns: [{\n name: 'ticker_symbol',\n sqlType: 'VARCHAR(4)',\n mapping: '$.ticker_symbol'\n }, {\n name: 'sector',\n sqlType: 'VARCHAR(16)',\n mapping: '$.sector'\n }, {\n name: 'change',\n sqlType: 'REAL',\n mapping: '$.change'\n }, {\n name: 'price',\n sqlType: 'REAL',\n mapping: '$.price'\n }],\n recordFormat: {\n recordFormatType: 'JSON'\n },\n recordEncoding: 'UTF-8'\n },\n namePrefix: 'SOURCE_SQL_STREAM'\n }]\n }\n});\n```\n\nPython\n\n```python\nfrom aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics import KinesisFirehoseToAnalyticsAndS3\nfrom aws_cdk import (\n aws_kinesisanalytics as kinesisanalytics,\n Stack\n)\nfrom constructs import Construct\n\nKinesisFirehoseToAnalyticsAndS3(self, 'FirehoseToS3AndAnalyticsPattern',\n kinesis_analytics_props=kinesisanalytics.CfnApplicationProps(\n inputs=[kinesisanalytics.CfnApplication.InputProperty(\n input_schema=kinesisanalytics.CfnApplication.InputSchemaProperty(\n record_columns=[kinesisanalytics.CfnApplication.RecordColumnProperty(\n name='ticker_symbol',\n sql_type='VARCHAR(4)',\n mapping='$.ticker_symbol'\n ), kinesisanalytics.CfnApplication.RecordColumnProperty(\n name='sector',\n sql_type='VARCHAR(16)',\n mapping='$.sector'\n ), kinesisanalytics.CfnApplication.RecordColumnProperty(\n name='change',\n sql_type='REAL',\n mapping='$.change'\n ), kinesisanalytics.CfnApplication.RecordColumnProperty(\n name='price',\n sql_type='REAL',\n mapping='$.price'\n )],\n record_format=kinesisanalytics.CfnApplication.RecordFormatProperty(\n record_format_type='JSON'\n ),\n record_encoding='UTF-8'\n ),\n name_prefix='SOURCE_SQL_STREAM'\n )]\n )\n )\n```\n\nJava\n\n```java\nimport software.constructs.Construct;\nimport java.util.List;\n\nimport software.amazon.awscdk.Stack;\nimport software.amazon.awscdk.StackProps;\nimport software.amazon.awscdk.services.kinesisanalytics.*;\nimport software.amazon.awscdk.services.kinesisanalytics.CfnApplication.*;\nimport software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics.*;\n\nnew KinesisFirehoseToAnalyticsAndS3(this, \"FirehoseToS3AndAnalyticsPattern\",\n new KinesisFirehoseToAnalyticsAndS3Props.Builder()\n .kinesisAnalyticsProps(new CfnApplicationProps.Builder()\n .inputs(List.of(new InputProperty.Builder()\n .inputSchema(new InputSchemaProperty.Builder()\n .recordColumns(List.of(\n new RecordColumnProperty.Builder()\n .name(\"ticker_symbol\")\n .sqlType(\"VARCHAR(4)\")\n .mapping(\"$.ticker_symbol\")\n .build(),\n new RecordColumnProperty.Builder()\n .name(\"sector\")\n .sqlType(\"VARCHAR(16)\")\n .mapping(\"$.sector\")\n .build(),\n new RecordColumnProperty.Builder()\n .name(\"change\")\n .sqlType(\"REAL\")\n .mapping(\"$.change\")\n .build(),\n new RecordColumnProperty.Builder()\n .name(\"price\")\n .sqlType(\"REAL\")\n .mapping(\"$.price\")\n .build()))\n .recordFormat(new RecordFormatProperty.Builder()\n .recordFormatType(\"JSON\")\n .build())\n .recordEncoding(\"UTF-8\")\n .build())\n .namePrefix(\"SOURCE_SQL_STREAM\")\n .build()))\n .build())\n .build());\n```\n\n## Pattern Construct Props\n\n| **Name** | **Type** | **Description** |\n|:-------------|:----------------|-----------------|\n|kinesisFirehoseProps?|[`kinesisFirehose.CfnDeliveryStreamProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisfirehose.CfnDeliveryStreamProps.html)|Optional user-provided props to override the default props for the Kinesis Firehose delivery stream.|\n|kinesisAnalyticsProps?|[`kinesisAnalytics.CfnApplicationProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisanalytics.CfnApplicationProps.html)|Optional user-provided props to override the default props for the Kinesis Analytics application.|\n|existingBucketObj?|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.IBucket.html)|Existing instance of S3 Bucket object. If this is provided, then also providing bucketProps is an error. |\n|bucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.BucketProps.html)|User provided props to override the default props for the S3 Bucket.|\n|logGroupProps?|[`logs.LogGroupProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.LogGroupProps.html)|User provided props to override the default props for for the CloudWatchLogs LogGroup.|\n|loggingBucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.BucketProps.html)|Optional user provided props to override the default props for the S3 Logging Bucket.|\n|logS3AccessLogs?| boolean|Whether to turn on Access Logging for the S3 bucket. Creates an S3 bucket with associated storage costs for the logs. Enabling Access Logging is a best practice. default - true|\n\n## Pattern Properties\n\n| **Name** | **Type** | **Description** |\n|:-------------|:----------------|-----------------|\n|kinesisAnalytics|[`kinesisAnalytics.CfnApplication`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisanalytics.CfnApplication.html)|Returns an instance of the Kinesis Analytics application created by the pattern.|\n|kinesisFirehose|[`kinesisFirehose.CfnDeliveryStream`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisfirehose.CfnDeliveryStream.html)|Returns an instance of the Kinesis Firehose delivery stream created by the pattern.|\n|kinesisFirehoseRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_iam.Role.html)|Returns an instance of the iam.Role created by the construct for Kinesis Data Firehose delivery stream.|\n|kinesisFirehoseLogGroup|[`logs.LogGroup`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.LogGroup.html)|Returns an instance of the LogGroup created by the construct for Kinesis Data Firehose delivery stream|\n|s3Bucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html)|Returns an instance of the S3 bucket created by the pattern.|\n|s3LoggingBucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct as the logging bucket for the primary bucket.|\n|s3BucketInterface|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.IBucket.html)|Returns an instance of s3.IBucket created by the construct.|\n\n## Default settings\n\nOut of the box implementation of the Construct without any override will set the following defaults:\n\n### Amazon Kinesis Firehose\n\n* Enable CloudWatch logging for Kinesis Firehose\n* Configure least privilege access IAM role for Amazon Kinesis Firehose\n\n### Amazon S3 Bucket\n\n* Configure Access logging for S3 Bucket\n* Enable server-side encryption for S3 Bucket using AWS managed KMS Key\n* Enforce encryption of data in transit\n* Turn on the versioning for S3 Bucket\n* Don't allow public access for S3 Bucket\n* Retain the S3 Bucket when deleting the CloudFormation stack\n* Applies Lifecycle rule to move noncurrent object versions to Glacier storage after 90 days\n\n### Amazon Kinesis Data Analytics\n\n* Configure least privilege access IAM role for Amazon Kinesis Analytics\n\n## Architecture\n\n\n\n---\n\n\n\u00a9 Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "CDK constructs for defining an interaction between an Amazon Kinesis Data Firehose delivery stream and (1) an Amazon S3 bucket, and (2) an Amazon Kinesis Data Analytics application.",
"version": "2.45.0",
"project_urls": {
"Homepage": "https://github.com/awslabs/aws-solutions-constructs.git",
"Source": "https://github.com/awslabs/aws-solutions-constructs.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d0700e2ed86d44114ad19013853b199b60ae1eb84d02bc970201ed3ae554c136",
"md5": "57c69f25efcf3878ee85b2b750eaa043",
"sha256": "ae509cb2f986c8956febfc7bdce78e9d7eaa563d5d04f1a02f3440092cad837e"
},
"downloads": -1,
"filename": "aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics-2.45.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "57c69f25efcf3878ee85b2b750eaa043",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "~=3.7",
"size": 155997,
"upload_time": "2023-10-14T14:45:50",
"upload_time_iso_8601": "2023-10-14T14:45:50.649098Z",
"url": "https://files.pythonhosted.org/packages/d0/70/0e2ed86d44114ad19013853b199b60ae1eb84d02bc970201ed3ae554c136/aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics-2.45.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d59d7dc7ac8373dfd18525a527f20d784c78f3ba11f72fc48628c02b34ea0ed2",
"md5": "986d624c08b891df5495e3ae9e62fcef",
"sha256": "d0e74a2c3b4e992c9859f30080564d800418fbf844182445b96b61c34976e3c5"
},
"downloads": -1,
"filename": "aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz",
"has_sig": false,
"md5_digest": "986d624c08b891df5495e3ae9e62fcef",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "~=3.7",
"size": 156514,
"upload_time": "2023-10-14T15:12:24",
"upload_time_iso_8601": "2023-10-14T15:12:24.953122Z",
"url": "https://files.pythonhosted.org/packages/d5/9d/7dc7ac8373dfd18525a527f20d784c78f3ba11f72fc48628c02b34ea0ed2/aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-14 15:12:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "awslabs",
"github_project": "aws-solutions-constructs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics"
}
JSON
