Amazon Managed Streaming for Apache Kafka (Amazon MSK) Module
This Terraform module configures and launches an Amazon MSK cluster.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache
Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating,
and deleting clusters. Managing all the data-plane operations, such running producers and consumers, is up to you.
It runs open-source versions of Apache Kafka, meaning existing applications, tooling,
and plugins from partners and the Apache Kafka community are supported without requiring changes to application code.
You can read more about supported Apache Kafka versions in the official documentation.
Note that this module does not support Amazon MSK Serverless,
which is still in preview.
How do you use this module?
See the root README for instructions on using Terraform modules.
See variables.tf for all the variables you can set on this module.
Cluster Configuration
Amazon MSK provides a default configuration
for brokers, topics, and Apache ZooKeeper nodes. You can also create custom configurations
with var.server_properties and use them to create new MSK clusters or to update existing clusters.
Capacity Planning
When planning the capacity for your cluster, there are multiple factors that need to be taken into consideration, including:
Performance and throughput
Fault tolerance
Storage capacity
To ensure high availability for production workloads, it is recommended to have a topic replication factor > 1. This means that
your topics are partitioned and replicated across multiple brokers in the cluster, leading to better fault tolerance and
parallelism for your consumers. As a rule of thumb, the optimal number of partitions for a topic should be equal to, or a
multiple of, the number of brokers in your cluster. Note that the number of partitions can only be increased, not decreased.
See https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html for further details on planning the capacity
and configuration of your cluster.
Storage Auto Scaling
Amount of required EBS storage depends on multiple factors, for example number of topics, amount
and size of your data, data retention and replication factor. As such it is not possible to give an exact recommendation, instead
the storage requirements should be calculated based on your use case. It is important to monitor disk usage and increase disk
size when needed.
The module will set the initial EBS volume size with input variable initial_ebs_volume_size and automatically scale the broker
volumes up until broker_storage_autoscaling_max_capacity is reached. You can optionally disable scale in with input
variable disable_broker_storage_scale_in. You can use broker_storage_autoscaling_target_percentage to control the scaling threshold.
Monitoring
Monitoring With CloudWatch
Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze metrics for your MSK serverless cluster.
You can set the monitoring level for an MSK cluster to one of the following: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, or PER_TOPIC_PER_PARTITION.
You can read more about metrics and monitoring here: https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html
Open Monitoring with Prometheus
You can also monitor your MSK cluster with Prometheus, an open-source monitoring system for
time-series metric data. You can also use tools that are compatible with Prometheus-formatted metrics or tools that integrate
with Amazon MSK Open Monitoring, like Datadog, Lenses, New Relic, and Sumo logic. You can read more about Open Monitoring
with Prometheus here: https://docs.aws.amazon.com/msk/latest/developerguide/open-monitoring.html
All metrics emitted by Apache Kafka to JMX are accessible using open monitoring with Prometheus. For information about Apache Kafka
metrics, see Monitoring in the Apache Kafka documentation.
Encryption
Amazon MSK allows you to enable encryption at rest and in transit.
The certificates that Amazon MSK uses for encryption must be renewed every 13 months. Amazon MSK automatically renews these
certificates for all clusters.
Encryption at Rest
Amazon MSK integrates with AWS Key Management Service (KMS) to offer transparent server-side encryption. Amazon MSK always
encrypts your data at rest. When you create an MSK cluster, you can specify the AWS KMS customer master key (CMK) with
var.encryption_at_rest_kms_key_arn that you want Amazon MSK to use to encrypt your data at rest. If no key is specified,
an AWS managed KMS (aws/msk managed service) key will be used for encrypting the data at rest.
Encryption in Transit
Amazon MSK uses TLS 1.2. By default, it encrypts data in transit between the brokers of your MSK cluster. You can override
this default using var.encryption_in_transit_in_cluster input variable at the time you create the cluster. You can also control client-to-broker encryption using var.encryption_in_transit_client_broker input variable.
Logging
Broker logs enable you to troubleshoot your Apache Kafka applications and to analyze their communications with your MSK cluster.
You can deliver Apache Kafka broker logs to one or more of the following destination types:
Amazon CloudWatch Logs
Amazon S3
Amazon Kinesis Data Firehose.
You can read more about MSK logging here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-logging.html
Authentication and Authorization
You can use IAM to authenticate clients and to
allow or deny Apache Kafka actions. Alternatively, you can use TLS
or SASL/SCRAM to authenticate clients, and
Apache Kafka ACLs to allow or deny actions.
You can read more about available authentication and authorization options here: https://docs.aws.amazon.com/msk/latest/developerguide/kafka_apis_iam.html
Connecting to Kafka brokers
Once you've used this module to deploy the Kafka brokers, you'll want to connect to them from Kafka clients (e.g.,
Kafka consumers and producers in your apps) to read and write data. To do this, you typically need to configure the
bootstrap.servers property for your Kafka client with the IP addresses of a few of your Kafka brokers (you don't
need all the IPs, as the rest will be discovered automatically via ZooKeeper):
Depending on which client authentication method you configured, there are a number of output variables (bootstrap_brokers_*) that
provide you with a list of bootstrap servers. You can also get the list of bootstrap servers using the AWS Cli:
MSK Connect is a feature of Amazon MSK that makes
it easy for developers to stream data to and from their Apache Kafka clusters. With MSK Connect, you can deploy fully managed
connectors built for Kafka Connect that move data into or pull data from popular data stores like Amazon S3 and Amazon
OpenSearch Service. You can deploy connectors developed by 3rd parties like Debezium for streaming change logs from databases
into an Apache Kafka cluster, or deploy an existing connector with no code changes. Connectors automatically scale to adjust
for changes in load and you pay only for the resources that you use.
Kafka Cluster Migration
You can mirror or migrate your cluster using MirrorMaker, which is part of Apache Kafka. Kafka MirrorMaker is a utility
that helps to replicate the data between two Apache Kafka clusters within or across regions.
For further information about migrating Kafka clusters, see: https://docs.aws.amazon.com/msk/latest/developerguide/migration.html
ZooKeeper
Kafka depends on ZooKeeper to work. Amazon MSK manages the Apache ZooKeeper nodes for you.
Each Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.
Controlling Access to Apache ZooKeeper
For security reasons you may want to limit access to the Apache ZooKeeper nodes that are part of your Amazon MSK cluster.
To limit access to the nodes, you can assign a separate security group to them. You can then decide who gets access to that security group.
As ZooKeeper security group configuration requires manual actions, this module does not include support for that. To change
the security group for ZooKeeper, follow these instructions: https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-security.html
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"bcfd56d5b2e88d8f95c5ed6697919363ef76653b"}]},{"name":".github","children":[{"name":"ISSUE_TEMPLATE","children":[{"name":"bug_report.md","path":".github/ISSUE_TEMPLATE/bug_report.md","sha":"d2e87e27c601e423865ed660ec697082470ca60f"},{"name":"feature_request.md","path":".github/ISSUE_TEMPLATE/feature_request.md","sha":"023a33099be2336476930c96e17ff1ba5dc55348"}]},{"name":"pull_request_template.md","path":".github/pull_request_template.md","sha":"6b100e40e323b5b07f40ed30616277c51c9f4b9e"}]},{"name":".gitignore","path":".gitignore","sha":"1b13a3bf1f1425e2e779cbf1a40a88e74a24f57b"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"1d8f329691fa148e29c9f4b1628b118d253d94a3"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"0616d223168080313c63eab2e7b9b66b90213052"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"947b78a929481a192a688dbcd8aef5e959102397"},{"name":"README.md","path":"README.md","sha":"aee4ffa5875881a7308befe61a08dcdd3e62ecd2"},{"name":"examples","children":[{"name":"kinesis","children":[{"name":"README.md","path":"examples/kinesis/README.md","sha":"a10663a758592a4001140edd7eff20d1dcf64b36"},{"name":"main.tf","path":"examples/kinesis/main.tf","sha":"bf2f6c5191408d19c18096f36d7eee5ee1d92dcc"},{"name":"outputs.tf","path":"examples/kinesis/outputs.tf","sha":"952a5cf88e478992ff28c5774e8f37217f89d296"},{"name":"variables.tf","path":"examples/kinesis/variables.tf","sha":"1065357fa72427e839f19969607c92a217fecd62"}]},{"name":"msk-with-iam-auth","children":[{"name":"README.md","path":"examples/msk-with-iam-auth/README.md","sha":"6af235e4cfa4afd19d919742748ffa53b053f572"},{"name":"main.tf","path":"examples/msk-with-iam-auth/main.tf","sha":"3583527e33af8d61839f41dd92949187920f1f51"},{"name":"outputs.tf","path":"examples/msk-with-iam-auth/outputs.tf","sha":"ef6b634e22db3e19f2e7848a0a0410c2f7f72035"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/msk-with-iam-auth/user-data/user-data.sh","sha":"c003cf5f3adec330d1181e8648684106cbaf9384"}]},{"name":"variables.tf","path":"examples/msk-with-iam-auth/variables.tf","sha":"4a122056ffbf362f23f0996d48112895eb46de43"}]},{"name":"sns-sqs-connection","children":[{"name":"README.md","path":"examples/sns-sqs-connection/README.md","sha":"9ab1842aef0a29fd9df9d9895052b8b008e92628"},{"name":"main.tf","path":"examples/sns-sqs-connection/main.tf","sha":"428a2115feaaa399787627f4454fb417f3261695"},{"name":"outputs.tf","path":"examples/sns-sqs-connection/outputs.tf","sha":"cbe0132f92c314bf95992232cd4e733102e7b08c"},{"name":"variables.tf","path":"examples/sns-sqs-connection/variables.tf","sha":"62ad583ebbafbfdf66f2149f453a88ae45ec14ea"}]},{"name":"sns","children":[{"name":"README.md","path":"examples/sns/README.md","sha":"d20e499ef9e61789460c42b65d84698cba354ffe"},{"name":"main.tf","path":"examples/sns/main.tf","sha":"2d5066419a7734031a36b987f7a272211469f867"},{"name":"outputs.tf","path":"examples/sns/outputs.tf","sha":"4f3fdd7d70805e7c7520730b598dca452785ecc5"},{"name":"variables.tf","path":"examples/sns/variables.tf","sha":"45c2d2eb3fe524bd334629f82cd35805404132f6"}]},{"name":"sqs-lambda-connection","children":[{"name":"README.md","path":"examples/sqs-lambda-connection/README.md","sha":"a93b874b1b91e8a08d5f3728537038a0ec8bfaa2"},{"name":"main.tf","path":"examples/sqs-lambda-connection/main.tf","sha":"bce89d1e78e8299cc106603615dd6c62dac01c92"},{"name":"outputs.tf","path":"examples/sqs-lambda-connection/outputs.tf","sha":"3dd33dd0e765e67e1c717330a95254a6c4f4e054"},{"name":"variables.tf","path":"examples/sqs-lambda-connection/variables.tf","sha":"64b2b533875ba3563998fe014d09a77076fe00e4"}]},{"name":"sqs","children":[{"name":"dead-letter-queue","children":[{"name":"README.md","path":"examples/sqs/dead-letter-queue/README.md","sha":"afb19a5c1926f3cc1192151c7d8ee1851da5bd7f"},{"name":"main.tf","path":"examples/sqs/dead-letter-queue/main.tf","sha":"b855b459b0725d234059eedee01960a85f747819"},{"name":"outputs.tf","path":"examples/sqs/dead-letter-queue/outputs.tf","sha":"b3d13e984797fab06aeae8abfd599b2a0f12d9f8"},{"name":"variables.tf","path":"examples/sqs/dead-letter-queue/variables.tf","sha":"5b3ac6fcefb6b6a9b2ed2bf31a08fb319277b643"}]},{"name":"fifo-queue-with-encryption","children":[{"name":"README.md","path":"examples/sqs/fifo-queue-with-encryption/README.md","sha":"56fcb9edfe2c4befd84fb31109598845ab48ba74"},{"name":"main.tf","path":"examples/sqs/fifo-queue-with-encryption/main.tf","sha":"3b367f99520bdccbd2b66f51fef175e0bd1ee059"},{"name":"outputs.tf","path":"examples/sqs/fifo-queue-with-encryption/outputs.tf","sha":"b3d13e984797fab06aeae8abfd599b2a0f12d9f8"},{"name":"variables.tf","path":"examples/sqs/fifo-queue-with-encryption/variables.tf","sha":"7155c480e6a99e1d998ec9a222e2d16378d90ce3"}]},{"name":"no-dead-letter-queue","children":[{"name":"README.md","path":"examples/sqs/no-dead-letter-queue/README.md","sha":"23a010877a28dbbed0ff2c1de54b7886f6db66e8"},{"name":"main.tf","path":"examples/sqs/no-dead-letter-queue/main.tf","sha":"f3a03460753bac31b4ada61216579f710762b44f"},{"name":"outputs.tf","path":"examples/sqs/no-dead-letter-queue/outputs.tf","sha":"e2961511ac23997e4f3e2cd8991559ffdbe10ac0"},{"name":"variables.tf","path":"examples/sqs/no-dead-letter-queue/variables.tf","sha":"a59eb9bc6c66d8a4bf9d9e4d418e78c21035d67b"}]}]}]},{"name":"modules","children":[{"name":"kinesis","children":[{"name":"README.md","path":"modules/kinesis/README.md","sha":"2755785a2c97e818989121565c1e6c4b4e2060bd"},{"name":"main.tf","path":"modules/kinesis/main.tf","sha":"1a71cd2a83f8d056c8f403c95d61c66b10ba0d32"},{"name":"outputs.tf","path":"modules/kinesis/outputs.tf","sha":"6a03e228332416c46f03acf869a1e364ca901c32"},{"name":"variables.tf","path":"modules/kinesis/variables.tf","sha":"71e59e5243f93f6a9a2237fb6779e241009cb5a3"}]},{"name":"msk","children":[{"name":"README.md","path":"modules/msk/README.md","sha":"8ed5181fd43ec36405581e63b27f2351ec50ab51","toggled":true},{"name":"main.tf","path":"modules/msk/main.tf","sha":"1522641efcfe23c2e76e1f354bc0454f8d650a79"},{"name":"outputs.tf","path":"modules/msk/outputs.tf","sha":"3819e63f34ef96848ab3c3036f0d4b744d986558"},{"name":"variables.tf","path":"modules/msk/variables.tf","sha":"c068d00afb67dbd314e3099f78028c8df530c9c2"}],"toggled":true},{"name":"sns-sqs-connection","children":[{"name":"README.md","path":"modules/sns-sqs-connection/README.md","sha":"b4f86c6463caf238a8ab01fbf87bad3ba2f10b2f"},{"name":"main.tf","path":"modules/sns-sqs-connection/main.tf","sha":"b8c831f697f21f1453701288ba69d685d6c2662d"},{"name":"outputs.tf","path":"modules/sns-sqs-connection/outputs.tf","sha":"f771fdc188025b64ef0219d4a1d8389d67a7c780"},{"name":"variables.tf","path":"modules/sns-sqs-connection/variables.tf","sha":"036c7ef148acec6ae018a0f24e16a08808630093"}]},{"name":"sns","children":[{"name":"README.md","path":"modules/sns/README.md","sha":"d364612270684e6508fd2956845277f5dd729568"},{"name":"main.tf","path":"modules/sns/main.tf","sha":"c7f51fb274b4e7733f020d927660f673107cde1e"},{"name":"outputs.tf","path":"modules/sns/outputs.tf","sha":"b187ff24e06f69c9d92e412cafbd05c11734f033"},{"name":"variables.tf","path":"modules/sns/variables.tf","sha":"309b61562b1cf9fd6c75508d58d6ebaa6799afaf"}]},{"name":"sqs-lambda-connection","children":[{"name":"README.md","path":"modules/sqs-lambda-connection/README.md","sha":"4a5c2d28641c9d5600c13d13f23d37f0401f41e8"},{"name":"main.tf","path":"modules/sqs-lambda-connection/main.tf","sha":"97957d3776a3249c839f13cdbbe8eb0098cf3503"},{"name":"outputs.tf","path":"modules/sqs-lambda-connection/outputs.tf","sha":"1776f39a27d6502062aab149ab8c831251875fad"},{"name":"variables.tf","path":"modules/sqs-lambda-connection/variables.tf","sha":"43ac7496926f6fb3ce22348f9f8bac40339bfb9a"}]},{"name":"sqs","children":[{"name":"README.md","path":"modules/sqs/README.md","sha":"c5d450a5bcf2e7c2d131ae0e4ba607b8745372e0"},{"name":"main.tf","path":"modules/sqs/main.tf","sha":"3e86ac0d978a87c1fb7cf811ffcf808478144ae3"},{"name":"outputs.tf","path":"modules/sqs/outputs.tf","sha":"e1d941e64d321ac3f6f36ff7fa8f8c4d07ffe6c5"},{"name":"variables.tf","path":"modules/sqs/variables.tf","sha":"677dff17ecb1669d5ef878d1068beb372c420b3e"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"031cdc06d155eff31f9c5648a1c92ee5fb816ea0"},{"name":"common.go","path":"test/common.go","sha":"d2f4c530eb49c07f511451ba2f8b3d80137bee10"},{"name":"go.mod","path":"test/go.mod","sha":"b8f23c37200b27f3b16e0edebf7a4303b2b9045f"},{"name":"go.sum","path":"test/go.sum","sha":"617efa8a713e3484215323f043e5a5e4930bf107"},{"name":"kinesis_test.go","path":"test/kinesis_test.go","sha":"8e1e4d0f08f773fea03391ff53f2ec1634246f94"},{"name":"msk_test.go","path":"test/msk_test.go","sha":"bb9ac4194aa3d5491d8c0e8592bc28f7addad7cc"},{"name":"sns_test.go","path":"test/sns_test.go","sha":"54d1374118a64af95f8bbb4477a4b4cf408b406f"},{"name":"sqs_dead_letter_queue_test.go","path":"test/sqs_dead_letter_queue_test.go","sha":"172d7a2f536fce51fddcf4f99010535975c656e3"},{"name":"sqs_encrypted_fifo_queue_test.go","path":"test/sqs_encrypted_fifo_queue_test.go","sha":"17102c0ac2d35b1768e673425055787a2505b448"},{"name":"sqs_standard_test.go","path":"test/sqs_standard_test.go","sha":"99238520e87a85c925dad4547787e860d7cb26d1"},{"name":"terratest_options.go","path":"test/terratest_options.go","sha":"eb2e3f26d1a18a604140344267affe968870116d"},{"name":"validation","children":[{"name":"validate_all_modules_and_examples_test.go","path":"test/validation/validate_all_modules_and_examples_test.go","sha":"33d73c385b64c4fc870033e99427e683c31dc45a"}]}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"amazon-managed-streaming-for-apache-kafka-amazon-msk-module\">Amazon Managed Streaming for Apache Kafka (Amazon MSK) Module</h1><div class=\"preview__body--border\"></div><p>This Terraform module configures and launches an <a href=\"https://aws.amazon.com/msk/\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon MSK</a> cluster.</p>\n<p>Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache\nKafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating,\nand deleting clusters. Managing all the data-plane operations, such running producers and consumers, is up to you.</p>\n<p>It runs open-source versions of <a href=\"https://github.com/apache/kafka\" class=\"preview__body--description--blue\" target=\"_blank\">Apache Kafka</a>, meaning existing applications, tooling,\nand plugins from partners and the Apache Kafka community are supported without requiring changes to application code.\nYou can read more about supported Apache Kafka versions in <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html\" class=\"preview__body--description--blue\" target=\"_blank\">the official documentation</a>.</p>\n<p>Note that this module does not support <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/serverless.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon MSK Serverless</a>,\nwhich is still in preview.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<ul>\n<li>See the <a href=\"/repos/v0.9.1/package-messaging/README.md\" class=\"preview__body--description--blue\">root README</a> for instructions on using Terraform modules.</li>\n<li>See the <a href=\"/repos/v0.9.1/package-messaging/examples\" class=\"preview__body--description--blue\">examples</a> folder for example usage.</li>\n<li>See <a href=\"/repos/v0.9.1/package-messaging/modules/msk/variables.tf\" class=\"preview__body--description--blue\">variables.tf</a> for all the variables you can set on this module.</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"cluster-configuration\">Cluster Configuration</h2>\n<p>Amazon MSK provides a <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-default-configuration.html\" class=\"preview__body--description--blue\" target=\"_blank\">default configuration</a>\nfor brokers, topics, and Apache ZooKeeper nodes. You can also create <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html\" class=\"preview__body--description--blue\" target=\"_blank\">custom configurations</a>\nwith <code>var.server_properties</code> and use them to create new MSK clusters or to update existing clusters.</p>\n<h3 class=\"preview__body--subtitle\" id=\"capacity-planning\">Capacity Planning</h3>\n<p>When planning the capacity for your cluster, there are multiple factors that need to be taken into consideration, including:</p>\n<ul>\n<li>Performance and throughput</li>\n<li>Fault tolerance</li>\n<li>Storage capacity</li>\n</ul>\n<p>To ensure high availability for production workloads, it is recommended to have a topic replication factor > 1. This means that\nyour topics are partitioned and replicated across multiple brokers in the cluster, leading to better fault tolerance and\nparallelism for your consumers. As a rule of thumb, the optimal number of partitions for a topic should be equal to, or a\nmultiple of, the number of brokers in your cluster. Note that the number of partitions can only be increased, not decreased.</p>\n<p>See https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html for further details on planning the capacity\nand configuration of your cluster.</p>\n<h3 class=\"preview__body--subtitle\" id=\"storage-auto-scaling\">Storage Auto Scaling</h3>\n<p>Amount of required EBS storage depends on multiple factors, for example number of topics, amount\nand size of your data, data retention and replication factor. As such it is not possible to give an exact recommendation, instead\nthe storage requirements should be calculated based on your use case. It is important to monitor disk usage and increase disk\nsize when needed.</p>\n<p>The module will set the initial EBS volume size with input variable <code>initial_ebs_volume_size</code> and automatically scale the broker\nvolumes up until <code>broker_storage_autoscaling_max_capacity</code> is reached. You can optionally disable scale in with input\nvariable <code>disable_broker_storage_scale_in</code>. You can use <code>broker_storage_autoscaling_target_percentage</code> to control the scaling threshold.</p>\n<h2 class=\"preview__body--subtitle\" id=\"monitoring\">Monitoring</h2>\n<h3 class=\"preview__body--subtitle\" id=\"monitoring-with-cloud-watch\">Monitoring With CloudWatch</h3>\n<p>Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze metrics for your MSK serverless cluster.\nYou can set the monitoring level for an MSK cluster to one of the following: <code>DEFAULT</code>, <code>PER_BROKER</code>, <code>PER_TOPIC_PER_BROKER</code>, or <code>PER_TOPIC_PER_PARTITION</code>.\nYou can read more about metrics and monitoring here: https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html</p>\n<h3 class=\"preview__body--subtitle\" id=\"open-monitoring-with-prometheus\">Open Monitoring with Prometheus</h3>\n<p>You can also monitor your MSK cluster with <a href=\"https://prometheus.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Prometheus</a>, an open-source monitoring system for\ntime-series metric data. You can also use tools that are compatible with Prometheus-formatted metrics or tools that integrate\nwith Amazon MSK Open Monitoring, like Datadog, Lenses, New Relic, and Sumo logic. You can read more about Open Monitoring\nwith Prometheus here: https://docs.aws.amazon.com/msk/latest/developerguide/open-monitoring.html</p>\n<p>All metrics emitted by Apache Kafka to JMX are accessible using open monitoring with Prometheus. For information about Apache Kafka\nmetrics, see <a href=\"https://kafka.apache.org/documentation/#monitoring\" class=\"preview__body--description--blue\" target=\"_blank\">Monitoring</a> in the Apache Kafka documentation.</p>\n<h2 class=\"preview__body--subtitle\" id=\"encryption\">Encryption</h2>\n<p>Amazon MSK allows you to enable encryption at rest and in transit.\nThe certificates that Amazon MSK uses for encryption must be renewed every 13 months. Amazon MSK automatically renews these\ncertificates for all clusters.</p>\n<h3 class=\"preview__body--subtitle\" id=\"encryption-at-rest\">Encryption at Rest</h3>\n<p>Amazon MSK integrates with AWS Key Management Service (KMS) to offer transparent server-side encryption. Amazon MSK always\nencrypts your data at rest. When you create an MSK cluster, you can specify the AWS KMS customer master key (CMK) with\n<code>var.encryption_at_rest_kms_key_arn</code> that you want Amazon MSK to use to encrypt your data at rest. If no key is specified,\nan AWS managed KMS (<code>aws/msk</code> managed service) key will be used for encrypting the data at rest.</p>\n<h3 class=\"preview__body--subtitle\" id=\"encryption-in-transit\">Encryption in Transit</h3>\n<p>Amazon MSK uses TLS 1.2. By default, it encrypts data in transit between the brokers of your MSK cluster. You can override\nthis default using <code>var.encryption_in_transit_in_cluster</code> input variable at the time you create the cluster. You can also control client-to-broker encryption using <code>var.encryption_in_transit_client_broker</code> input variable.</p>\n<h2 class=\"preview__body--subtitle\" id=\"logging\">Logging</h2>\n<p>Broker logs enable you to troubleshoot your Apache Kafka applications and to analyze their communications with your MSK cluster.\nYou can deliver Apache Kafka broker logs to one or more of the following destination types:</p>\n<ul>\n<li>Amazon CloudWatch Logs</li>\n<li>Amazon S3</li>\n<li>Amazon Kinesis Data Firehose.</li>\n</ul>\n<p>You can read more about MSK logging here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-logging.html</p>\n<h2 class=\"preview__body--subtitle\" id=\"authentication-and-authorization\">Authentication and Authorization</h2>\n<p>You can use <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html\" class=\"preview__body--description--blue\" target=\"_blank\">IAM</a> to authenticate clients and to\nallow or deny Apache Kafka actions. Alternatively, you can use <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html\" class=\"preview__body--description--blue\" target=\"_blank\">TLS</a>\nor <a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html\" class=\"preview__body--description--blue\" target=\"_blank\">SASL/SCRAM</a> to authenticate clients, and\n<a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html\" class=\"preview__body--description--blue\" target=\"_blank\">Apache Kafka ACLs</a> to allow or deny actions.\nYou can read more about available authentication and authorization options here: https://docs.aws.amazon.com/msk/latest/developerguide/kafka_apis_iam.html</p>\n<h2 class=\"preview__body--subtitle\" id=\"connecting-to-kafka-brokers\">Connecting to Kafka brokers</h2>\n<p>Once you've used this module to deploy the Kafka brokers, you'll want to connect to them from Kafka clients (e.g.,\nKafka consumers and producers in your apps) to read and write data. To do this, you typically need to configure the\n<code>bootstrap.servers</code> property for your Kafka client with the IP addresses of a few of your Kafka brokers (you don't\nneed all the IPs, as the rest will be discovered automatically via ZooKeeper):</p>\n<pre>--bootstrap.servers=<span class=\"hljs-number\">10.0</span><span class=\"hljs-number\">.0</span><span class=\"hljs-number\">.4</span>:<span class=\"hljs-number\">9092</span>,<span class=\"hljs-number\">10.0</span><span class=\"hljs-number\">.0</span><span class=\"hljs-number\">.5</span>:<span class=\"hljs-number\">9092</span>,<span class=\"hljs-number\">10.0</span><span class=\"hljs-number\">.0</span><span class=\"hljs-number\">.6</span>:<span class=\"hljs-number\">9092</span>\n</pre>\n<p>Depending on which client authentication method you configured, there are a number of output variables (<code>bootstrap_brokers_*</code>) that\nprovide you with a list of bootstrap servers. You can also get the list of bootstrap servers using the AWS Cli:</p>\n<pre>$ <span class=\"hljs-string\">aws </span><span class=\"hljs-string\">kafka </span><span class=\"hljs-built_in\">get-bootstrap-brokers</span> <span class=\"hljs-built_in\">--cluster-arn</span> <span class=\"hljs-string\">ClusterArn\n</span>\n{\n <span class=\"hljs-string\">\"BootstrapBrokerStringSaslIam\"</span>: <span class=\"hljs-string\">\"b-1.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9098,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9098\"</span>\n}\n</pre>\n<h3 class=\"preview__body--subtitle\" id=\"msk-connect\">MSK Connect</h3>\n<p><a href=\"https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html\" class=\"preview__body--description--blue\" target=\"_blank\">MSK Connect</a> is a feature of Amazon MSK that makes\nit easy for developers to stream data to and from their Apache Kafka clusters. With MSK Connect, you can deploy fully managed\nconnectors built for Kafka Connect that move data into or pull data from popular data stores like Amazon S3 and Amazon\nOpenSearch Service. You can deploy connectors developed by 3rd parties like Debezium for streaming change logs from databases\ninto an Apache Kafka cluster, or deploy an existing connector with no code changes. Connectors automatically scale to adjust\nfor changes in load and you pay only for the resources that you use.</p>\n<h2 class=\"preview__body--subtitle\" id=\"kafka-cluster-migration\">Kafka Cluster Migration</h2>\n<p>You can mirror or migrate your cluster using MirrorMaker, which is part of Apache Kafka. Kafka MirrorMaker is a utility\nthat helps to replicate the data between two Apache Kafka clusters within or across regions.</p>\n<p>For further information about migrating Kafka clusters, see: https://docs.aws.amazon.com/msk/latest/developerguide/migration.html</p>\n<h2 class=\"preview__body--subtitle\" id=\"zoo-keeper\">ZooKeeper</h2>\n<p>Kafka depends on <a href=\"https://zookeeper.apache.org/\" class=\"preview__body--description--blue\" target=\"_blank\">ZooKeeper</a> to work. Amazon MSK manages the Apache ZooKeeper nodes for you.\nEach Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.</p>\n<h3 class=\"preview__body--subtitle\" id=\"controlling-access-to-apache-zoo-keeper\">Controlling Access to Apache ZooKeeper</h3>\n<p>For security reasons you may want to limit access to the Apache ZooKeeper nodes that are part of your Amazon MSK cluster.\nTo limit access to the nodes, you can assign a separate security group to them. You can then decide who gets access to that security group.</p>\n<p>As ZooKeeper security group configuration requires manual actions, this module does not include support for that. To change\nthe security group for ZooKeeper, follow these instructions: https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-security.html</p>\n","repoName":"package-messaging","repoRef":"v0.8.2","serviceDescriptor":{"serviceName":"Kinesis","serviceRepoName":"package-messaging","serviceRepoOrg":"gruntwork-io","serviceMainReadmePath":"/modules/kinesis","cloudProviders":["aws"],"description":"Create Kinesis streams with configurable or auto-calculated shard and retention settings.","imageUrl":"kinesis.png","licenseType":"subscriber","technologies":["Terraform"],"compliance":[],"tags":[""]},"serviceCategoryName":"Messaging & streaming","fileName":"README.md","filePath":"/modules/msk","title":"Repo Browser: Kinesis","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
