# DynamoDB Lens
A set of tools to help understand DynamoDB table performance capabilities and potential cost optimization opportunities.
## Installation and usage
### Installation
```shell
python3 -m pip install dynamodb-lens
```
### CLI usage
```shell
usage: cli [-h] --table_name TABLE_NAME [--metric_consumed_period_s METRIC_CONSUMED_PERIOD_S] [--save_analysis] [--verbose]
options:
-h, --help show this help message and exit
--table_name TABLE_NAME
--metric_consumed_period_s METRIC_CONSUMED_PERIOD_S
The cloudwatch metric period for consumed WCU/RCU
--save_analysis save the json formatted analysis to a file
--verbose Print the full analysis, otherwise a summary will be printed
python3 -m dynamodb_lens.cli --table_name sentences1 --metric_consumed_period_s 60 --verbose --save_analysis
```
## Table Analyzer
This library analyzes a table's current state, configuration and usage in order to estimate performance capabilities.
A DynamoDB Table's current partition count as well as table settings will determine its performance capabilities.
Partition count is not exposed directly, but we can infer the number of partitions in several ways:
1. Count the number of Open DynamoDB Stream shards, it is a 1:1 mapping of Open shards to Partitions
2. Check cloudwatch max Provisioned settings over the last 3 months
3. Check cloudwatch max consumed WCU/RCU over the last 1 month (more accurate if consumed_period_s set to 60)
4. Check current WCU/RCU settings on the table if Provisioned mode is currently configured
5. Check the current storage utilization of the table
If Streams is not enabled, this program will estimate current number of partitions based on the maximum WCU/RCU values
calculated from the data gathered in #2-5 above.
DynamoDB tables never give back partitions once they've been allocated.
This is what is meant by ["previous peak" in the documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual).
***This program is more accurate if DynamoDB Streams is enabled for the table.***
Enabling streams is outside the scope of this program, it can be done via:
----
`aws dynamodb update-table --table-name <value> --stream-specification StreamEnabled=true,StreamViewType=KEYS_ONLY`
and disabled with:
`aws dynamodb update-table --table-name <value> --stream-specification StreamEnabled=false`
Please review the documentation and understand the implications of running these commands before executing them.
----
### TableAnalyzer class usage
The class can be imported and called directly if the `dynamodb_lens.cli` doesn't fit the use case
The only required parameter is table_name, and optionally `verbose=True|False` plus already instantiated boto3 clients can be passed in.
```python
from dynamodb_lens.analyzer import TableAnalyzer
from dynamodb_lens.utils import return_boto3_client
table = TableAnalyzer(
table_name='foo',
consumed_period_s=60,
lambda_client=return_boto3_client('lambda'),
ddbs_client=return_boto3_client('dynamodbstreams'),
ddb_client=return_boto3_client('dynamodb'),
cw_client=return_boto3_client('cloudwatch'),
verbose=False
)
table.print_analysis()
```
### TableAnalyzer.analysis
The main purpose of TableAnalyzer is to produce a json-formatted analysis variable.
#### Syntax
```json
{
"TableDescription": { dict },
"StreamDescription": { dict },
"EstimationData": {
"Description": "Raw estimation data used for debugging purposes only!",
"WCU": {
"CurrentProvisionedWCU": int,
"MaxConsumedWCU": int,
"MaxProvisionedWCU": int
},
"RCU": {
"CurrentProvisionedRCU": int,
"MaxConsumedRCU": int,
"MaxProvisionedRCU": int
},
"Partitions": {
"MaxProvisionedWCU": int,
"MaxProvisionedRCU": int,
"CurrentTableSize": int
}
},
"Summary": {
"TableName": "string",
"TableArn": "string",
"DeletionProtection": boolean,
"SizeMB": int,
"ItemCount": int,
"BillingMode": "ON-DEMAND|PROVISIONED",
"StreamArn": "string",
"Estimations": {
"EstimationMethod": "StreamOpenShards|OnDemandBaseSpecs|CurrentTableSize|CurrentProvisionedWCU|CurrentProvisionedRCU|MaxConsumedWCU|MaxConsumedRCU|MaxProvisionedWCU|MaxProvisionedRCU",
"EstimationMethodDescription": "string",
"Partitions": int,
"TableMaximums": {
"WCU": int,
"WriteThroughputMBs": int,
"RCU": {
"Base": int,
"EventuallyConsistent": int
},
"ReadThroughputMBs": {
"Base": int,
"EventuallyConsistent": int
}
},
"PartitionMaximums": {
"Comments": "string",
"WCU": int,
"WriteThroughputMBs": int,
"RCU": {
"Base": int,
"EventuallyConsistent": int
},
"ReadThroughputMBs": {
"Base": int,
"EventuallyConsistent": int
}
}
}
}
}
```
#### Structure
- TableDescription (dict)
requires `self.verbose = True`
The boto3 dynamodb client.describe_table response. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/describe_table.html
- StreamDescription (dict)
requires `self.verbose = True`
The boto3 dynamodbstreams client.describe_stream response. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodbstreams/client/describe_stream.html
- EstimationsData (dict)
requires `self.verbose = True`
This is the data used to estimate number of partitions *if streams are not enabled, primarily meant for debugging only*
- WCU (dict)
- RCU (dict)
- Partitions (dict)
- Summary (dict)
- TableName (string)
- The name of the table
- TableArn (string)
- The ARN of the table
- DeletionProtection (bool)
- Is Deletion protection enabled?
- SizeMB (int)
- Size in Megabytes
- ItemCount (int)
- The number of items
- BillingMode (str)
- ON-DEMAND: Pay per request
- PROVISIONED: Pay hourly for predictable throughput
- StreamArn (str)
- The ARN of the stream
- Estimations (dict)
- EstimationMethod (string) - The method that was used to estimate the number of partitions and performance capabilities
- StreamOpenShards
- When DynamoDB Streams are enabled for a table, we simply count the number of Open shards to determine partitions
- OnDemandBaseSpecs
- The table is in On-Demand mode and base table specs for on-demand were used for the calculations
- CurrentTableSize
- Each DynamoDB partition can be up to 10 GB in size, total size (GB)/10
- CurrentProvisionedWCU|CurrentProvisionedRCU
- The current table specifications
- MaxConsumedWCU|MaxConsumedRCU|MaxProvisionedWCU|MaxProvisionedRCU
- Cloudwatch metrics
- EstimationMethodDescription (str)
- Describes the EstimationMethod used and why
- Partitions (int)
- The number of partitions
- If Streams are enabled, this is 100% accurate.
- If we used metric data or table settings then it is an estimate.
- TableMaximums (dict) - Based on number of partitions and current table settings, estimate the table maximums
- WCU (int)
- Total WriteCapacityUnits
- WriteThroughputMbs
- Total Write Throughput
- RCU (dict)
- Base (int)
- ReadCapacityUnits for strong reads
- EventuallyConsistent (int)
- ReadCapacityUnits for eventually consistent reads
- ReadThroughputMBs (int)
- Base (int)
- ReadThroughput in MB/s for strong reads
- EventuallyConsistent (int)
- ReadThroughput in MB/s for eventually consistent reads
- PartitionMaximums (dict) - Based on number of partitions and current table settings, estimate the per-partition maximums
- Comments (string)
- Comments about the partition maximums
- WCU (int)
- Per-partition WriteCapacityUnits
- WriteThroughputMbs (int)
- Per-partition Write Throughput
- RCU (dict)
- Base (int)
- Per-partition ReadCapacityUnits for strong reads
- EventuallyConsistent (int)
- Per-partition ReadCapacityUnits for eventually consistent reads
- ReadThroughputMBs (int)
- Base (int)
- ReadThroughput in MB/s for strong reads
- EventuallyConsistent (int)
- ReadThroughput in MB/s for eventually consistent reads
Raw data
{
"_id": null,
"home_page": "https://github.com/doitintl/dynamodb-lens.git",
"name": "dynamodb-lens",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "dynamodb dynamodb-lens",
"author": "Adam North",
"author_email": "anorth848@gmail.com",
"download_url": "",
"platform": null,
"description": "# DynamoDB Lens\n\nA set of tools to help understand DynamoDB table performance capabilities and potential cost optimization opportunities. \n\n## Installation and usage\n\n### Installation\n```shell\npython3 -m pip install dynamodb-lens\n```\n### CLI usage\n```shell\nusage: cli [-h] --table_name TABLE_NAME [--metric_consumed_period_s METRIC_CONSUMED_PERIOD_S] [--save_analysis] [--verbose]\n\noptions:\n -h, --help show this help message and exit\n --table_name TABLE_NAME\n --metric_consumed_period_s METRIC_CONSUMED_PERIOD_S\n The cloudwatch metric period for consumed WCU/RCU\n --save_analysis save the json formatted analysis to a file\n --verbose Print the full analysis, otherwise a summary will be printed\npython3 -m dynamodb_lens.cli --table_name sentences1 --metric_consumed_period_s 60 --verbose --save_analysis\n```\n\n## Table Analyzer \n\nThis library analyzes a table's current state, configuration and usage in order to estimate performance capabilities. \nA DynamoDB Table's current partition count as well as table settings will determine its performance capabilities. \nPartition count is not exposed directly, but we can infer the number of partitions in several ways:\n1. Count the number of Open DynamoDB Stream shards, it is a 1:1 mapping of Open shards to Partitions\n2. Check cloudwatch max Provisioned settings over the last 3 months\n3. Check cloudwatch max consumed WCU/RCU over the last 1 month (more accurate if consumed_period_s set to 60)\n4. Check current WCU/RCU settings on the table if Provisioned mode is currently configured \n5. Check the current storage utilization of the table\n\nIf Streams is not enabled, this program will estimate current number of partitions based on the maximum WCU/RCU values \ncalculated from the data gathered in #2-5 above. \nDynamoDB tables never give back partitions once they've been allocated. \nThis is what is meant by [\"previous peak\" in the documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual).\n\n***This program is more accurate if DynamoDB Streams is enabled for the table.*** \nEnabling streams is outside the scope of this program, it can be done via: \n\n----\n `aws dynamodb update-table --table-name <value> --stream-specification StreamEnabled=true,StreamViewType=KEYS_ONLY` \nand disabled with: \n `aws dynamodb update-table --table-name <value> --stream-specification StreamEnabled=false` \n\nPlease review the documentation and understand the implications of running these commands before executing them.\n\n----\n\n### TableAnalyzer class usage\nThe class can be imported and called directly if the `dynamodb_lens.cli` doesn't fit the use case \nThe only required parameter is table_name, and optionally `verbose=True|False` plus already instantiated boto3 clients can be passed in. \n\n```python\nfrom dynamodb_lens.analyzer import TableAnalyzer\nfrom dynamodb_lens.utils import return_boto3_client\n\ntable = TableAnalyzer(\n table_name='foo',\n consumed_period_s=60,\n lambda_client=return_boto3_client('lambda'),\n ddbs_client=return_boto3_client('dynamodbstreams'),\n ddb_client=return_boto3_client('dynamodb'),\n cw_client=return_boto3_client('cloudwatch'),\n verbose=False\n)\ntable.print_analysis()\n```\n\n### TableAnalyzer.analysis \nThe main purpose of TableAnalyzer is to produce a json-formatted analysis variable.\n\n#### Syntax \n\n```json\n{\n \"TableDescription\": { dict },\n \"StreamDescription\": { dict },\n \"EstimationData\": {\n \"Description\": \"Raw estimation data used for debugging purposes only!\",\n \"WCU\": {\n \"CurrentProvisionedWCU\": int,\n \"MaxConsumedWCU\": int,\n \"MaxProvisionedWCU\": int\n },\n \"RCU\": {\n \"CurrentProvisionedRCU\": int,\n \"MaxConsumedRCU\": int,\n \"MaxProvisionedRCU\": int\n },\n \"Partitions\": {\n \"MaxProvisionedWCU\": int,\n \"MaxProvisionedRCU\": int,\n \"CurrentTableSize\": int\n }\n },\n \"Summary\": {\n \"TableName\": \"string\",\n \"TableArn\": \"string\",\n \"DeletionProtection\": boolean,\n \"SizeMB\": int,\n \"ItemCount\": int,\n \"BillingMode\": \"ON-DEMAND|PROVISIONED\",\n \"StreamArn\": \"string\",\n \"Estimations\": {\n \"EstimationMethod\": \"StreamOpenShards|OnDemandBaseSpecs|CurrentTableSize|CurrentProvisionedWCU|CurrentProvisionedRCU|MaxConsumedWCU|MaxConsumedRCU|MaxProvisionedWCU|MaxProvisionedRCU\",\n \"EstimationMethodDescription\": \"string\",\n \"Partitions\": int,\n \"TableMaximums\": {\n \"WCU\": int,\n \"WriteThroughputMBs\": int,\n \"RCU\": {\n \"Base\": int,\n \"EventuallyConsistent\": int\n },\n \"ReadThroughputMBs\": {\n \"Base\": int,\n \"EventuallyConsistent\": int\n }\n },\n \"PartitionMaximums\": {\n \"Comments\": \"string\",\n \"WCU\": int,\n \"WriteThroughputMBs\": int,\n \"RCU\": {\n \"Base\": int,\n \"EventuallyConsistent\": int\n },\n \"ReadThroughputMBs\": {\n \"Base\": int,\n \"EventuallyConsistent\": int\n }\n }\n }\n }\n}\n```\n#### Structure \n- TableDescription (dict) \nrequires `self.verbose = True` \nThe boto3 dynamodb client.describe_table response. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/describe_table.html\n- StreamDescription (dict) \nrequires `self.verbose = True` \nThe boto3 dynamodbstreams client.describe_stream response. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodbstreams/client/describe_stream.html\n- EstimationsData (dict) \nrequires `self.verbose = True` \nThis is the data used to estimate number of partitions *if streams are not enabled, primarily meant for debugging only*\n - WCU (dict)\n - RCU (dict)\n - Partitions (dict)\n- Summary (dict)\n - TableName (string)\n - The name of the table\n - TableArn (string)\n - The ARN of the table\n - DeletionProtection (bool)\n - Is Deletion protection enabled?\n - SizeMB (int)\n - Size in Megabytes\n - ItemCount (int)\n - The number of items\n - BillingMode (str)\n - ON-DEMAND: Pay per request \n - PROVISIONED: Pay hourly for predictable throughput\n - StreamArn (str)\n - The ARN of the stream\n - Estimations (dict)\n - EstimationMethod (string) - The method that was used to estimate the number of partitions and performance capabilities\n - StreamOpenShards \n - When DynamoDB Streams are enabled for a table, we simply count the number of Open shards to determine partitions\n - OnDemandBaseSpecs\n - The table is in On-Demand mode and base table specs for on-demand were used for the calculations\n - CurrentTableSize\n - Each DynamoDB partition can be up to 10 GB in size, total size (GB)/10 \n - CurrentProvisionedWCU|CurrentProvisionedRCU\n - The current table specifications\n - MaxConsumedWCU|MaxConsumedRCU|MaxProvisionedWCU|MaxProvisionedRCU\n - Cloudwatch metrics \n - EstimationMethodDescription (str)\n - Describes the EstimationMethod used and why\n - Partitions (int)\n - The number of partitions \n - If Streams are enabled, this is 100% accurate. \n - If we used metric data or table settings then it is an estimate.\n - TableMaximums (dict) - Based on number of partitions and current table settings, estimate the table maximums\n - WCU (int)\n - Total WriteCapacityUnits\n - WriteThroughputMbs\n - Total Write Throughput\n - RCU (dict)\n - Base (int)\n - ReadCapacityUnits for strong reads\n - EventuallyConsistent (int)\n - ReadCapacityUnits for eventually consistent reads\n - ReadThroughputMBs (int)\n - Base (int)\n - ReadThroughput in MB/s for strong reads\n - EventuallyConsistent (int)\n - ReadThroughput in MB/s for eventually consistent reads\n - PartitionMaximums (dict) - Based on number of partitions and current table settings, estimate the per-partition maximums\n - Comments (string)\n - Comments about the partition maximums\n - WCU (int)\n - Per-partition WriteCapacityUnits\n - WriteThroughputMbs (int)\n - Per-partition Write Throughput\n - RCU (dict)\n - Base (int)\n - Per-partition ReadCapacityUnits for strong reads\n - EventuallyConsistent (int)\n - Per-partition ReadCapacityUnits for eventually consistent reads\n - ReadThroughputMBs (int)\n - Base (int)\n - ReadThroughput in MB/s for strong reads\n - EventuallyConsistent (int)\n - ReadThroughput in MB/s for eventually consistent reads\n \n",
"bugtrack_url": null,
"license": "MIT",
"summary": "DynamoDB Lens - Analyze your DynamoDB environment",
"version": "23.10.dev10",
"project_urls": {
"Homepage": "https://github.com/doitintl/dynamodb-lens.git"
},
"split_keywords": [
"dynamodb",
"dynamodb-lens"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "10c223766814044d5fcc566216eb1ea151e913283e57729e835383691db26a1a",
"md5": "c22a2da63f1c7ef9cc6c0ca215079b39",
"sha256": "0ac055235e7676813d1142b6b85e7571e841159176171b3d00602c08ade0fdda"
},
"downloads": -1,
"filename": "dynamodb_lens-23.10.dev10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c22a2da63f1c7ef9cc6c0ca215079b39",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 10947,
"upload_time": "2023-11-16T17:49:31",
"upload_time_iso_8601": "2023-11-16T17:49:31.118960Z",
"url": "https://files.pythonhosted.org/packages/10/c2/23766814044d5fcc566216eb1ea151e913283e57729e835383691db26a1a/dynamodb_lens-23.10.dev10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-16 17:49:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "doitintl",
"github_project": "dynamodb-lens",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "boto3",
"specs": [
[
"~=",
"1.28.67"
]
]
},
{
"name": "setuptools",
"specs": [
[
"~=",
"68.0.0"
]
]
},
{
"name": "botocore",
"specs": [
[
"~=",
"1.31.68"
]
]
},
{
"name": "jmespath",
"specs": [
[
"~=",
"1.0.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"~=",
"2.8.2"
]
]
},
{
"name": "s3transfer",
"specs": [
[
"~=",
"0.7.0"
]
]
},
{
"name": "six",
"specs": [
[
"~=",
"1.16.0"
]
]
},
{
"name": "urllib3",
"specs": [
[
"~=",
"1.26.18"
]
]
}
],
"lcname": "dynamodb-lens"
}
JSON
