Browse the Repo

file-type-icon.circleci
file-type-iconexamples
file-type-iconmodules
file-type-iconkinesis
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvars.tf
file-type-iconsns
file-type-iconsqs
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconLICENSE.txt
file-type-iconREADME.md

Browse the Repo

file-type-icon.circleci
file-type-iconexamples
file-type-iconmodules
file-type-iconkinesis
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvars.tf
file-type-iconsns
file-type-iconsqs
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconLICENSE.txt
file-type-iconREADME.md
Kinesis

Kinesis

Create Kinesis streams with configurable or auto-calculated shard and retention settings.

Preview the Code

mobile file icon

README.md

down

Kinesis Stream Module

This module makes it easy to deploy a Kinesis stream

How do you use this module?

  • See the root README for instructions on using Terraform modules.
  • See the examples folder for example usage.
  • See vars.tf for all the variables you can set on this module.

Shard Sizing

Kinesis streams acheive scalability by using shards. This module allows you to either specify number_of_shards directly or to specify the average_data_size_in_kb, records_per_second and number_of_consumers variables and the module will calculate the proper number of shards that should be used based on AWS best practices.

incoming_write_bandwidth_in_kb = average_data_size_in_kb * records_per_second

outgoing_read_bandwidth_in_kb = incoming_write_bandwidth_in_kb * number_of_consumers

number_of_shards = max(incoming_write_bandwidth_in_kb/1000, outgoing_read_bandwidth_in_kb/2000)

Encryption

Kinesis streams support server-side encryption as described in the Kinesis SSE documentation. It can be switched on retrospectively for existing streams with no interruptions (although only new data will be encrypted).

To enable encryption, set the following parameter

encryption_type = "SSE"

This will use the default AWS service key for Kinesis, aws/kinesis.

If you need to use a custom key, see the master key module as well as documentation on user-generated KMS master keys for further information on how to create them. You can specify one using

kms_key_id = "alias/<my_cmk_alias>"

Examples

Here are some examples of how you might deploy a Kinesis stream with this module:

module "kinesis" {
  source = "git::git@github.com:gruntwork-io/package-messaging.git//modules/kinesis?ref=v0.0.1"

  name = "my-stream"
  retention_period = 48

  number_of_shards = 1
  shard_level_metrics = [
    "IncomingBytes",
    "IncomingRecords",
    "IteratorAgeMilliseconds",
    "OutgoingBytes",
    "OutgoingRecords",
    "ReadProvisionedThroughputExceeded",
    "WriteProvisionedThroughputExceeded"
  ]
 
}
module "kinesis" {
  source = "git::git@github.com:gruntwork-io/package-messaging.git//modules/kinesis?ref=v0.0.1"
  name = "my-stream"
  retention_period = 48
  
  average_data_size_in_kb = 20
  records_per_second = 10
  number_of_consumers = 10
  
  shard_level_metrics = [
      "ReadProvisionedThroughputExceeded",
      "WriteProvisionedThroughputExceeded"
    ]
}

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?