A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence
number that is assigned by Kinesis Data Streams.
data record: A data record is the unit of data stored in a Kinesis data stream. Data records are composed of a
sequence number, a partition key, and a data blob, which is an immutable sequence of bytes.
shard: A shard is a uniquely identified sequence of data records in a stream. A stream is composed of one or more
shards, each of which provides a fixed unit of capacity.
sequence number: Each data record has a sequence number that is unique per partition-key within its shard. Kinesis
Data Streams assigns the sequence number after you write to the stream with client.putRecords or client.putRecord.
Sequence numbers for the same partition key generally increase over time. The longer the time period between write
requests, the larger the sequence numbers become.
Sharding / Partitioning in Kinesis Data Stream
Kinesis Data Stream achieves scalability by using shards.
The data capacity of your stream is a function of the number of shards that you specify for the stream. The total
capacity of the stream is the sum of the capacities of its shards.
How to Set Shard Size
You can configure the initial number of shards in two ways:
direct specification: specify number_of_shards directly
indirect specification: specify the average_data_size_in_kb, records_per_second and number_of_consumers
variables and let the module calculate the initial number of shards.
Note: the module calculates the initial number of shards by:
Calculate the incoming write bandwidth in KB (incoming_write_bandwidth_in_KB), which is equal to the
average_data_size_in_KB multiplied by the number_of_records_per_second.
Calculate the outgoing read bandwidth in KB (outgoing_read_bandwidth_in_KB), which is equal to the
incoming_write_bandwidth_in_KB multiplied by the number_of_consumers.
You can then calculate the initial number of shards (number_of_shards) your data stream needs using the following
formula: number_of_shards = max (incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)
A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records
belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to
determine which shard a given data record belongs to. When an application puts data into a stream, it must specify a
partition key.
With a single shard, all data goes into the same shard. There's no other way to use a custom partitioning logic.
How to Re-Shard a Stream
Re-configuring the shard size will result destroying the old Kinesis data stream and re-creating it with a new one. In
order to prevent this, consider
using the UpdateShardCount API.
Updating the shard count is an asynchronous operation. To update the shard count, Kinesis Data Streams performs splits
or merges on individual shards. This can cause short-lived shards to be created, in addition to the final shards. These
short-lived shards count towards your total shard limit for your account in the Region.
You can find more information in the following pages:
Here are some limitation of the Kinesis Data stream you might be interested in :
Data Payload Size: The maximum size of the data payload of a record before base64-encoding is up to 1 MB.
Retention Period: The maximum value of a stream's retention period is 8760 hours (365 days).
Shard Throughput: Each shard can support up to 1 MB/sec or 1,000 records/sec write throughput or up to 2 MB/sec or
2,000 records/sec read throughput.
You can find the latest and full list of limitation/quotas in this page:
Quotas and Limits.
Encryption
Amazon Kinesis Data Streams can automatically encrypt sensitive data as it enters into a stream. Kinesis Data
Streams uses AWS KMS master keys for encryption. With server-side encryption, your Kinesis stream producers and
consumers don't need to manage master keys or cryptographic operations. Your data is automatically encrypted as it
enters and leaves the Kinesis Data Streams service, so your data at rest is encrypted. For more information,
see Data Protection in Amazon Kinesis Data
Streams.
How to Enable Encryption
You can enable encryption in two ways:
default encryption: set encryption_type = "KMS". This will use the default AWS service key for
Kinesis, aws/kinesis.
custom key encryption: If you need to use a Customer Managed Key (CMK), see the
master key module as well
as
documentation on user-generated KMS master keys
for further information on how to create them. You can specify one using kms_key_id = "alias/<my_cmk_alias>"
How to Change KMS Key
You can change the KMS key by reconfiguring the encryption with the kms_key_id and encryption_type variables.
Please note that changing the KMS key for a Kinesis Data Stream does not retroactively re-encrypt previously encrypted
data in the stream with the new KMS key. Any data that was previously encrypted with the old KMS key will remain
encrypted with that key. However, any new data added to the stream after the KMS key change will be encrypted with the
new KMS key.
If you need to re-encrypt the previously encrypted data in the stream with the new KMS key, you will need to manually
copy the data to a new stream that is configured to use the new KMS key for encryption. Alternatively, you can use AWS
Lambda or other AWS services to read the data from the original stream, decrypt it using the old KMS key, and then
re-encrypt it with the new KMS key before writing it to a new stream or another data store.
There is also a sample prototype from AWS that demonstrates continuous data capture (CDC) to replicate data across
regions: https://github.com/aws-samples/aws-kinesis-data-streams-replicator
How do you use this module?
See the root README for instructions on using Terraform modules.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"0f20b1aeee35e40105995318a3673f9de7dce60a"},{"name":"post-upgrade-test-results.sh","path":".circleci/post-upgrade-test-results.sh","sha":"a4867e8fbdc334b7a90259568ee41ea577fbe764"},{"name":"set-upgrade-test-vars.sh","path":".circleci/set-upgrade-test-vars.sh","sha":"a4325dc9a27331b96e6beab4ce9b1f0ce4e31dff"}]},{"name":".github","children":[{"name":"ISSUE_TEMPLATE","children":[{"name":"bug_report.md","path":".github/ISSUE_TEMPLATE/bug_report.md","sha":"d2e87e27c601e423865ed660ec697082470ca60f"},{"name":"feature_request.md","path":".github/ISSUE_TEMPLATE/feature_request.md","sha":"023a33099be2336476930c96e17ff1ba5dc55348"}]},{"name":"pull_request_template.md","path":".github/pull_request_template.md","sha":"6b100e40e323b5b07f40ed30616277c51c9f4b9e"}]},{"name":".gitignore","path":".gitignore","sha":"1b13a3bf1f1425e2e779cbf1a40a88e74a24f57b"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"1d8f329691fa148e29c9f4b1628b118d253d94a3"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"3bc8a8c8ee6e939522f988c5407a5d62b2d4ac26"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"947b78a929481a192a688dbcd8aef5e959102397"},{"name":"README.md","path":"README.md","sha":"aee4ffa5875881a7308befe61a08dcdd3e62ecd2"},{"name":"examples","children":[{"name":"kinesis-firehose","children":[{"name":"README.md","path":"examples/kinesis-firehose/README.md","sha":"63a635a255416c679c6923066e710cb0ebc4c48b"},{"name":"main.tf","path":"examples/kinesis-firehose/main.tf","sha":"a3066e551d63f28b29c7e993ac7521c2e3f0230e"},{"name":"outputs.tf","path":"examples/kinesis-firehose/outputs.tf","sha":"2d2f21ce3b128f4128501eb8baca321edd265b40"},{"name":"vars.tf","path":"examples/kinesis-firehose/vars.tf","sha":"c2db039403ad41d8dc4a9f76f01977ac8afab322"}]},{"name":"kinesis","children":[{"name":"README.md","path":"examples/kinesis/README.md","sha":"a10663a758592a4001140edd7eff20d1dcf64b36"},{"name":"main.tf","path":"examples/kinesis/main.tf","sha":"b1538aa291b260b64aea6b429200033822e6c4d9"},{"name":"outputs.tf","path":"examples/kinesis/outputs.tf","sha":"952a5cf88e478992ff28c5774e8f37217f89d296"},{"name":"variables.tf","path":"examples/kinesis/variables.tf","sha":"f80fdeed4e088f648533e1f0b0aee590710b5e95"}]},{"name":"msk-serverless","children":[{"name":"README.md","path":"examples/msk-serverless/README.md","sha":"6af235e4cfa4afd19d919742748ffa53b053f572"},{"name":"main.tf","path":"examples/msk-serverless/main.tf","sha":"59a618443c467071b87f036292c29f503054aed2"},{"name":"outputs.tf","path":"examples/msk-serverless/outputs.tf","sha":"12fee8ee670f50d82a4fa6e93d74cedad8cce3e0"},{"name":"variables.tf","path":"examples/msk-serverless/variables.tf","sha":"96901f971338ff0a67749068057acd0bddc42646"}]},{"name":"msk-tiered-storage","children":[{"name":"README.md","path":"examples/msk-tiered-storage/README.md","sha":"daaced8743acbabc4e7b2231d74cb782a67669fb"},{"name":"main.tf","path":"examples/msk-tiered-storage/main.tf","sha":"4b78009056e330699b467aed8839561e0149a1cb"},{"name":"outputs.tf","path":"examples/msk-tiered-storage/outputs.tf","sha":"af52b0aa83770a269a0948b58b358ecca3fc01a9"},{"name":"variables.tf","path":"examples/msk-tiered-storage/variables.tf","sha":"25397bcdfb81ce9077348ae17981dbc7eacf1539"}]},{"name":"msk-with-iam-auth","children":[{"name":"README.md","path":"examples/msk-with-iam-auth/README.md","sha":"6af235e4cfa4afd19d919742748ffa53b053f572"},{"name":"main.tf","path":"examples/msk-with-iam-auth/main.tf","sha":"12a64f83e90f2ff23eb4aff455562d9fe4cade3d"},{"name":"outputs.tf","path":"examples/msk-with-iam-auth/outputs.tf","sha":"ef6b634e22db3e19f2e7848a0a0410c2f7f72035"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/msk-with-iam-auth/user-data/user-data.sh","sha":"37084170a864f40703074f41d8237b02940ae003"}]},{"name":"variables.tf","path":"examples/msk-with-iam-auth/variables.tf","sha":"34206410a50c7ba3c6e3934607a32de923b392f9"}]},{"name":"sns-sqs-connection","children":[{"name":"README.md","path":"examples/sns-sqs-connection/README.md","sha":"9ab1842aef0a29fd9df9d9895052b8b008e92628"},{"name":"main.tf","path":"examples/sns-sqs-connection/main.tf","sha":"7670d37614681fe4e59aa353e8cc69572bfb443a"},{"name":"outputs.tf","path":"examples/sns-sqs-connection/outputs.tf","sha":"cbe0132f92c314bf95992232cd4e733102e7b08c"},{"name":"variables.tf","path":"examples/sns-sqs-connection/variables.tf","sha":"00a1de12b64b2f8f54ab6f8029a6269ebf1c9b17"}]},{"name":"sns","children":[{"name":"README.md","path":"examples/sns/README.md","sha":"d20e499ef9e61789460c42b65d84698cba354ffe"},{"name":"main.tf","path":"examples/sns/main.tf","sha":"dba59be72cc16ed8ef18522db692abf8e5570b27"},{"name":"outputs.tf","path":"examples/sns/outputs.tf","sha":"4f3fdd7d70805e7c7520730b598dca452785ecc5"},{"name":"variables.tf","path":"examples/sns/variables.tf","sha":"2570453ca049efe8c975f15d972abd8f921e9a77"}]},{"name":"sqs-lambda-connection","children":[{"name":"README.md","path":"examples/sqs-lambda-connection/README.md","sha":"a93b874b1b91e8a08d5f3728537038a0ec8bfaa2"},{"name":"main.tf","path":"examples/sqs-lambda-connection/main.tf","sha":"c4d23d3c65d16061d7cfd7ddf4317ea4ccd62997"},{"name":"outputs.tf","path":"examples/sqs-lambda-connection/outputs.tf","sha":"ba567af0da54eb835d5ce7c7e63aff8ed554d5c1"},{"name":"sqs-process","children":[{"name":"index.py","path":"examples/sqs-lambda-connection/sqs-process/index.py","sha":"dc030ea073e067cf2dc43e18d090f583bde29293"}]},{"name":"variables.tf","path":"examples/sqs-lambda-connection/variables.tf","sha":"1bb3e543e10679ee00fa52f125f34eb40e8cccf4"}]},{"name":"sqs","children":[{"name":"dead-letter-queue","children":[{"name":"README.md","path":"examples/sqs/dead-letter-queue/README.md","sha":"afb19a5c1926f3cc1192151c7d8ee1851da5bd7f"},{"name":"main.tf","path":"examples/sqs/dead-letter-queue/main.tf","sha":"b5207d5f9e1c44054b9e5042901c8a11e3d1ff96"},{"name":"outputs.tf","path":"examples/sqs/dead-letter-queue/outputs.tf","sha":"b3d13e984797fab06aeae8abfd599b2a0f12d9f8"},{"name":"variables.tf","path":"examples/sqs/dead-letter-queue/variables.tf","sha":"9b7fdb5dd19661c154ee515d97da55f839770ae7"}]},{"name":"fifo-queue-with-encryption","children":[{"name":"README.md","path":"examples/sqs/fifo-queue-with-encryption/README.md","sha":"56fcb9edfe2c4befd84fb31109598845ab48ba74"},{"name":"main.tf","path":"examples/sqs/fifo-queue-with-encryption/main.tf","sha":"c42c2524f61ec8730b0dc192f04beefc50d99efe"},{"name":"outputs.tf","path":"examples/sqs/fifo-queue-with-encryption/outputs.tf","sha":"b3d13e984797fab06aeae8abfd599b2a0f12d9f8"},{"name":"variables.tf","path":"examples/sqs/fifo-queue-with-encryption/variables.tf","sha":"7155c480e6a99e1d998ec9a222e2d16378d90ce3"}]},{"name":"no-dead-letter-queue","children":[{"name":"README.md","path":"examples/sqs/no-dead-letter-queue/README.md","sha":"23a010877a28dbbed0ff2c1de54b7886f6db66e8"},{"name":"main.tf","path":"examples/sqs/no-dead-letter-queue/main.tf","sha":"f452e1e8706c7d6a7041bd13061a03ff64f05074"},{"name":"outputs.tf","path":"examples/sqs/no-dead-letter-queue/outputs.tf","sha":"e2961511ac23997e4f3e2cd8991559ffdbe10ac0"},{"name":"variables.tf","path":"examples/sqs/no-dead-letter-queue/variables.tf","sha":"a59eb9bc6c66d8a4bf9d9e4d418e78c21035d67b"}]}]}]},{"name":"modules","children":[{"name":"kinesis-firehose","children":[{"name":"README.md","path":"modules/kinesis-firehose/README.md","sha":"646b8bd78f910e71be3ce87f901c0bd51d27a3b8"},{"name":"main.tf","path":"modules/kinesis-firehose/main.tf","sha":"5df7b08a42aa449fd1075bf5e253ae566ab2e001"},{"name":"outputs.tf","path":"modules/kinesis-firehose/outputs.tf","sha":"291ffbe53d2134c703fe6393d57804bb95cdd4a9"},{"name":"vars.tf","path":"modules/kinesis-firehose/vars.tf","sha":"beff58e1741f6348957c4dfd68249381f47d0c9c"}]},{"name":"kinesis","children":[{"name":"README.md","path":"modules/kinesis/README.md","sha":"4a600af2539f758ab9caa3f404c967afea036c8d","toggled":true},{"name":"main.tf","path":"modules/kinesis/main.tf","sha":"f5da37dc8cc7bc4a3b6b4cbd823713b5c08d87dd"},{"name":"outputs.tf","path":"modules/kinesis/outputs.tf","sha":"6a03e228332416c46f03acf869a1e364ca901c32"},{"name":"variables.tf","path":"modules/kinesis/variables.tf","sha":"66a28b150ea67757c31c52624724446edf5289e2"}],"toggled":true},{"name":"msk","children":[{"name":"README.md","path":"modules/msk/README.md","sha":"0ea023f7c0676eb66b7a1fd6144a0c2f6947e577"},{"name":"main.tf","path":"modules/msk/main.tf","sha":"5962019c0a892bac6af5bce755064a0c09154c44"},{"name":"outputs.tf","path":"modules/msk/outputs.tf","sha":"ac5465c3811228b04cfd03a18b1fb7e2142958e0"},{"name":"variables.tf","path":"modules/msk/variables.tf","sha":"728d8b4e296ea603353e5a4997f39d1956af5748"}]},{"name":"sns-sqs-connection","children":[{"name":"README.md","path":"modules/sns-sqs-connection/README.md","sha":"b4f86c6463caf238a8ab01fbf87bad3ba2f10b2f"},{"name":"main.tf","path":"modules/sns-sqs-connection/main.tf","sha":"b301639d9802bace005e41bc596d9914264f4fd9"},{"name":"outputs.tf","path":"modules/sns-sqs-connection/outputs.tf","sha":"f771fdc188025b64ef0219d4a1d8389d67a7c780"},{"name":"variables.tf","path":"modules/sns-sqs-connection/variables.tf","sha":"384c82e9679a0d5844aec4ad44aa671acac72503"}]},{"name":"sns","children":[{"name":"README.md","path":"modules/sns/README.md","sha":"7c154e8af0034b7dbc5825c57c07656df5f4a997"},{"name":"main.tf","path":"modules/sns/main.tf","sha":"c18eb2de9a0d594de21beb861ef5965bd70cc134"},{"name":"outputs.tf","path":"modules/sns/outputs.tf","sha":"b187ff24e06f69c9d92e412cafbd05c11734f033"},{"name":"variables.tf","path":"modules/sns/variables.tf","sha":"8a8ac91e642cd367eee8d84eccad3c166664d3fb"}]},{"name":"sqs-lambda-connection","children":[{"name":"README.md","path":"modules/sqs-lambda-connection/README.md","sha":"4a5c2d28641c9d5600c13d13f23d37f0401f41e8"},{"name":"main.tf","path":"modules/sqs-lambda-connection/main.tf","sha":"20932dea0a4dbfae31928295246893986f72b831"},{"name":"outputs.tf","path":"modules/sqs-lambda-connection/outputs.tf","sha":"1776f39a27d6502062aab149ab8c831251875fad"},{"name":"variables.tf","path":"modules/sqs-lambda-connection/variables.tf","sha":"43ac7496926f6fb3ce22348f9f8bac40339bfb9a"}]},{"name":"sqs","children":[{"name":"README.md","path":"modules/sqs/README.md","sha":"c5d450a5bcf2e7c2d131ae0e4ba607b8745372e0"},{"name":"main.tf","path":"modules/sqs/main.tf","sha":"17ca9bfc6de1b5478c3bb68a03db69054001ff4b"},{"name":"outputs.tf","path":"modules/sqs/outputs.tf","sha":"e1d941e64d321ac3f6f36ff7fa8f8c4d07ffe6c5"},{"name":"variables.tf","path":"modules/sqs/variables.tf","sha":"9f50555e07318eac2a8aab9f3ff82b04bd7d4ffa"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"031cdc06d155eff31f9c5648a1c92ee5fb816ea0"},{"name":"common.go","path":"test/common.go","sha":"f8574c59bb96494db5d644cdeb7d01735195dd3e"},{"name":"go.mod","path":"test/go.mod","sha":"9179500054e96a98377cd6bf49caccb02290df80"},{"name":"go.sum","path":"test/go.sum","sha":"7ef564d3e21815e6fabf3796378b8f14fa19a25e"},{"name":"kinesis_firehose_test.go","path":"test/kinesis_firehose_test.go","sha":"8d5041f4fecfead619bdaf6d458867b60977e5c7"},{"name":"kinesis_test.go","path":"test/kinesis_test.go","sha":"8d6e1f2efe46e6ba374b7f84a4c2427cf9496ad2"},{"name":"msk_serverless_test.go","path":"test/msk_serverless_test.go","sha":"66b1d0048f3e8d8941c326de100fe5e87cf5d73a"},{"name":"msk_test.go","path":"test/msk_test.go","sha":"44bdef31e1cda2eb209937bc605da28d333d5332"},{"name":"msk_tiered_storage_test.go","path":"test/msk_tiered_storage_test.go","sha":"77f7ff0e6e9c8f310dc6bc5efebf94591747a1dd"},{"name":"sns_sqs_connection_test.go","path":"test/sns_sqs_connection_test.go","sha":"4d3886c2f311837a515814053e9781abcb8567a1"},{"name":"sns_test.go","path":"test/sns_test.go","sha":"0f817301ce6102993752dea1fe5dde9b29c5fdeb"},{"name":"sqs_dead_letter_queue_test.go","path":"test/sqs_dead_letter_queue_test.go","sha":"fb338072c1da0e4b9f4af835428bf640da040252"},{"name":"sqs_encrypted_fifo_queue_test.go","path":"test/sqs_encrypted_fifo_queue_test.go","sha":"d54f013c4ecb9f685a4aea19aeab8455025d52b5"},{"name":"sqs_lambda_connection_test.go","path":"test/sqs_lambda_connection_test.go","sha":"db4af174857be14359692c1d001939fff76e6632"},{"name":"sqs_standard_test.go","path":"test/sqs_standard_test.go","sha":"7d54c5085ba793d6ec55bf5083382b5bb66bede9"},{"name":"terratest_options.go","path":"test/terratest_options.go","sha":"3e47c5021909080a0567172a85f9167620d6e993"},{"name":"upgrades","children":[{"name":"upgrade_test.go","path":"test/upgrades/upgrade_test.go","sha":"6bd8072cfb37593a0c75fe649307ac3a8dfb72ef"}]},{"name":"validation","children":[{"name":"validate_all_modules_and_examples_test.go","path":"test/validation/validate_all_modules_and_examples_test.go","sha":"33d73c385b64c4fc870033e99427e683c31dc45a"}]}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"kinesis-data-stream-module\">Kinesis Data Stream Module</h1><div class=\"preview__body--border\"></div><p>This module creates a <a href=\"https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html\" class=\"preview__body--description--blue\" target=\"_blank\">Kinesis Data Stream</a>.</p>\n<h2 class=\"preview__body--subtitle\" id=\"about-kinesis-data-stream\">About Kinesis Data Stream</h2>\n<p>A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence\nnumber that is assigned by Kinesis Data Streams.</p>\n<ul>\n<li><strong>data record</strong>: A data record is the unit of data stored in a Kinesis data stream. Data records are composed of a\nsequence number, a partition key, and a data blob, which is an immutable sequence of bytes.</li>\n<li><strong>shard</strong>: A shard is a uniquely identified sequence of data records in a stream. A stream is composed of one or more\nshards, each of which provides a fixed unit of capacity.</li>\n<li><strong>sequence number</strong>: Each data record has a sequence number that is unique per partition-key within its shard. Kinesis\nData Streams assigns the sequence number after you write to the stream with client.putRecords or client.putRecord.\nSequence numbers for the same partition key generally increase over time. The longer the time period between write\nrequests, the larger the sequence numbers become.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"sharding-partitioning-in-kinesis-data-stream\">Sharding / Partitioning in Kinesis Data Stream</h3>\n<p>Kinesis Data Stream achieves scalability by using <a href=\"https://en.wikipedia.org/wiki/Shard_(database_architecture)\" class=\"preview__body--description--blue\" target=\"_blank\">shards</a>.\nThe data capacity of your stream is a function of the number of shards that you specify for the stream. The total\ncapacity of the stream is the sum of the capacities of its shards.</p>\n<h4 id=\"how-to-set-shard-size\">How to Set Shard Size</h4>\n<p>You can configure the initial number of shards in two ways:</p>\n<ul>\n<li><strong>direct specification</strong>: specify <code>number_of_shards</code> directly</li>\n<li><strong>indirect specification</strong>: specify the <code>average_data_size_in_kb</code>, <code>records_per_second</code> and <code>number_of_consumers</code>\nvariables and let the module calculate the initial number of shards.</li>\n</ul>\n<p><strong>Note</strong>: the module calculates the initial number of shards by:</p>\n<ol>\n<li>Calculate the incoming write bandwidth in KB (incoming_write_bandwidth_in_KB), which is equal to the\naverage_data_size_in_KB multiplied by the number_of_records_per_second.</li>\n<li>Calculate the outgoing read bandwidth in KB (outgoing_read_bandwidth_in_KB), which is equal to the\nincoming_write_bandwidth_in_KB multiplied by the number_of_consumers.</li>\n<li>You can then calculate the initial number of shards (number_of_shards) your data stream needs using the following\nformula: number_of_shards = max (incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)</li>\n</ol>\n<p>Refer\nto <a href=\"https://aws.amazon.com/kinesis/data-streams/faqs/?nc=sn&loc=6\" class=\"preview__body--description--blue\" target=\"_blank\">the suggestion of calculating the initial number of shards FAQ</a>\nfor more information.</p>\n<h3 class=\"preview__body--subtitle\" id=\"how-does-data-partition-work\">How does Data Partition Work</h3>\n<p>A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records\nbelonging to a stream into multiple shards. It uses the partition key that is associated with each data record to\ndetermine which shard a given data record belongs to. When an application puts data into a stream, it must specify a\npartition key.</p>\n<p>With a single shard, all data goes into the same shard. There's no other way to use a custom partitioning logic.</p>\n<h3 class=\"preview__body--subtitle\" id=\"how-to-re-shard-a-stream\">How to Re-Shard a Stream</h3>\n<p>Re-configuring the shard size will result destroying the old Kinesis data stream and re-creating it with a new one. In\norder to prevent this, consider\nusing <a href=\"https://docs.aws.amazon.com/kinesis/latest/APIReference/API_UpdateShardCount.html\" class=\"preview__body--description--blue\" target=\"_blank\">the UpdateShardCount API</a>.\nUpdating the shard count is an asynchronous operation. To update the shard count, Kinesis Data Streams performs splits\nor merges on individual shards. This can cause short-lived shards to be created, in addition to the final shards. These\nshort-lived shards count towards your total shard limit for your account in the Region.\nYou can find more information in the following pages:</p>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-resharding.html\" class=\"preview__body--description--blue\" target=\"_blank\">Resharding a Stream</a></li>\n<li>https://github.com/hashicorp/terraform/issues/11816</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"limitation\">Limitation</h3>\n<p>Here are some limitation of the Kinesis Data stream you might be interested in :</p>\n<ul>\n<li><strong>Data Payload Size</strong>: The maximum size of the data payload of a record before base64-encoding is up to 1 MB.</li>\n<li><strong>Retention Period</strong>: The maximum value of a stream's retention period is 8760 hours (365 days).</li>\n<li><strong>Shard Throughput</strong>: Each shard can support up to 1 MB/sec or 1,000 records/sec write throughput or up to 2 MB/sec or\n2,000 records/sec read throughput.</li>\n</ul>\n<p>You can find the latest and full list of limitation/quotas in this page:\n<a href=\"https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html\" class=\"preview__body--description--blue\" target=\"_blank\">Quotas and Limits</a>.</p>\n<h3 class=\"preview__body--subtitle\" id=\"encryption\">Encryption</h3>\n<p>Amazon Kinesis Data Streams can automatically encrypt sensitive data as it enters into a stream. Kinesis Data\nStreams uses AWS KMS master keys for encryption. With server-side encryption, your Kinesis stream producers and\nconsumers don't need to manage master keys or cryptographic operations. Your data is automatically encrypted as it\nenters and leaves the Kinesis Data Streams service, so your data at rest is encrypted. For more information,\nsee <a href=\"https://docs.aws.amazon.com/streams/latest/dev/server-side-encryption.html\" class=\"preview__body--description--blue\" target=\"_blank\">Data Protection in Amazon Kinesis Data\nStreams</a>.</p>\n<h4 id=\"how-to-enable-encryption\">How to Enable Encryption</h4>\n<p>You can enable encryption in two ways:</p>\n<ul>\n<li><strong>default encryption</strong>: set <code>encryption_type = "KMS"</code>. This will use the default AWS service key for\nKinesis, <code>aws/kinesis</code>.</li>\n<li><strong>custom key encryption</strong>: If you need to use a Customer Managed Key (CMK), see the\n<a href=\"/repos/module-security-public/modules/kms-master-key\" class=\"preview__body--description--blue\">master key module</a> as well\nas\n<a href=\"https://docs.aws.amazon.com/streams/latest/dev/creating-using-sse-master-keys.html\" class=\"preview__body--description--blue\" target=\"_blank\">documentation on user-generated KMS master keys</a>\nfor further information on how to create them. You can specify one using <code>kms_key_id = "alias/<my_cmk_alias>"</code></li>\n</ul>\n<h4 id=\"how-to-change-kms-key\">How to Change KMS Key</h4>\n<p>You can change the KMS key by reconfiguring the encryption with the <code>kms_key_id</code> and <code>encryption_type</code> variables.</p>\n<p>Please note that changing the KMS key for a Kinesis Data Stream does not retroactively re-encrypt previously encrypted\ndata in the stream with the new KMS key. Any data that was previously encrypted with the old KMS key will remain\nencrypted with that key. However, any new data added to the stream after the KMS key change will be encrypted with the\nnew KMS key.</p>\n<p>If you need to re-encrypt the previously encrypted data in the stream with the new KMS key, you will need to manually\ncopy the data to a new stream that is configured to use the new KMS key for encryption. Alternatively, you can use AWS\nLambda or other AWS services to read the data from the original stream, decrypt it using the old KMS key, and then\nre-encrypt it with the new KMS key before writing it to a new stream or another data store.</p>\n<h3 class=\"preview__body--subtitle\" id=\"replication\">Replication</h3>\n<p>Amazon Kinesis Data Stream does not support replication out of the box. One way to implement replication is to use\nLambda. You can find more information from this AWS\narticle: <a href=\"https://aws.amazon.com/blogs/big-data/build-highly-available-streams-with-amazon-kinesis-data-streams/\" class=\"preview__body--description--blue\" target=\"_blank\">Build highly available streams with Amazon Kinesis Data Streams\n</a></p>\n<p>There is also a sample prototype from AWS that demonstrates continuous data capture (CDC) to replicate data across\nregions: https://github.com/aws-samples/aws-kinesis-data-streams-replicator</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<ul>\n<li>See the <a href=\"/repos/v0.12.1/package-messaging/README.md\" class=\"preview__body--description--blue\">root README</a> for instructions on using Terraform modules.</li>\n<li>See the <a href=\"/repos/v0.12.1/package-messaging/examples/kinesis\" class=\"preview__body--description--blue\">kinesis examples</a> folder for an example usage.</li>\n<li>See the <a href=\"/repos/v0.12.1/package-messaging/modules/kinesis/variables.tf\" class=\"preview__body--description--blue\">variables.tf</a> for all the variables you can set on this module.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"examples\">Examples</h3>\n<p>Aside from the example module linked above, here are some examples of how you might deploy a Kinesis stream with this\nmodule:</p>\n<pre>module <span class=\"hljs-string\">\"kinesis\"</span> {\n <span class=\"hljs-attr\">source</span> = <span class=\"hljs-string\">\"git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/kinesis?ref=v0.0.1\"</span>\n\n <span class=\"hljs-attr\">name</span> = <span class=\"hljs-string\">\"my-stream\"</span>\n <span class=\"hljs-attr\">retention_period</span> = <span class=\"hljs-number\">48</span>\n\n <span class=\"hljs-attr\">number_of_shards</span> = <span class=\"hljs-number\">1</span>\n <span class=\"hljs-attr\">shard_level_metrics</span> = [\n <span class=\"hljs-string\">\"IncomingBytes\"</span>,\n <span class=\"hljs-string\">\"IncomingRecords\"</span>,\n <span class=\"hljs-string\">\"IteratorAgeMilliseconds\"</span>,\n <span class=\"hljs-string\">\"OutgoingBytes\"</span>,\n <span class=\"hljs-string\">\"OutgoingRecords\"</span>,\n <span class=\"hljs-string\">\"ReadProvisionedThroughputExceeded\"</span>,\n <span class=\"hljs-string\">\"WriteProvisionedThroughputExceeded\"</span>\n ]\n\n}\n</pre>\n<pre>module <span class=\"hljs-string\">\"kinesis\"</span> {\n <span class=\"hljs-attr\">source</span> = <span class=\"hljs-string\">\"git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/kinesis?ref=v0.0.1\"</span>\n <span class=\"hljs-attr\">name</span> = <span class=\"hljs-string\">\"my-stream\"</span>\n <span class=\"hljs-attr\">retention_period</span> = <span class=\"hljs-number\">48</span>\n\n <span class=\"hljs-attr\">average_data_size_in_kb</span> = <span class=\"hljs-number\">20</span>\n <span class=\"hljs-attr\">records_per_second</span> = <span class=\"hljs-number\">10</span>\n <span class=\"hljs-attr\">number_of_consumers</span> = <span class=\"hljs-number\">10</span>\n\n <span class=\"hljs-attr\">shard_level_metrics</span> = [\n <span class=\"hljs-string\">\"ReadProvisionedThroughputExceeded\"</span>,\n <span class=\"hljs-string\">\"WriteProvisionedThroughputExceeded\"</span>\n ]\n}\n</pre>\n","repoName":"package-messaging","repoRef":"v0.12.5","serviceDescriptor":{"serviceName":"Kinesis","serviceRepoName":"package-messaging","serviceRepoOrg":"gruntwork-io","serviceMainReadmePath":"/modules/kinesis","cloudProviders":["aws"],"description":"Create Kinesis streams with configurable or auto-calculated shard and retention settings.","imageUrl":"kinesis.png","licenseType":"subscriber","technologies":["Terraform"],"compliance":[],"tags":[""]},"serviceCategoryName":"Messaging & streaming","fileName":"README.md","filePath":"/modules/kinesis","title":"Repo Browser: Kinesis","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
