This folder contains an example Packer template that can be used to create an Amazon
Machine Image (AMI) with Kafka and all of its
dependencies installed. A few notes about this Packer template:
It builds two AMIs, one on top of Amazon Linux and one on top of Ubuntu. Pick which ever distro you prefer.
This AMI includes an example of how to configure Kafka with SSL. It expects you to use the generate-key-stores
module to generate a Key Store and Trust Store before building the AMI.
This AMI only includes Kafka. It does not include ZooKeeper, which is expected to be running in a separate cluster,
with a separate AMI (see the zookeeper-ami example). For an example of Kafka and ZooKeeper
co-located in the same cluster, see kafka-zookeeper-ami.
Run the generate-key-stores module to create a Key Store, Trust Store, and
self-signed SSL certificate. You can run the check-for-key-store.sh script to print out example usage for the
generate-key-stores.sh script.
Set the GITHUB_OAUTH_TOKEN environment variable to a valid GitHub auth token with "repo" access. You can generate
one here: https://github.com/settings/tokens
Run packer build -var "package_kafka_branch=<branch>" -only <target os> kafka.json to create a new AMI in your AWS account. Note down the ID of this new AMI.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"cffdc89fc933135f3e9fece69b656f34186ebe2e"}]},{"name":".gitignore","path":".gitignore","sha":"e68eece82c5bbdddf63121f38b66cdea255b5567"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"78c6562dad87745c957f47039fdf85bb647c52c2"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"4be01a6334d39aa5bf6abe6baae701f5e2a8c5ac"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"689cf10ec98e3297a75bdd9b9fb5da10b7a675f8"},{"name":"README.md","path":"README.md","sha":"3b28f35855c132ef18878116e94b623ef2831d0d"},{"name":"examples","children":[{"name":"confluent-oss-ami","children":[{"name":"README.md","path":"examples/confluent-oss-ami/README.md","sha":"675ce60c000b84facb441e96fa16e89614fe2b6b"},{"name":"check-for-kafka-trust-store.sh","path":"examples/confluent-oss-ami/check-for-kafka-trust-store.sh","sha":"e2e7d323e5153471809af13b4434c9edfad1b85b"},{"name":"check-for-key-store.sh","path":"examples/confluent-oss-ami/check-for-key-store.sh","sha":"120539a6d63dfd2ed208e92c0a25a32b6b86a2bd"},{"name":"config","children":[{"name":"README.md","path":"examples/confluent-oss-ami/config/README.md","sha":"50e3000eda90f23880d9f34da435f1b170e213f5"},{"name":"kafka-connect","children":[{"name":"config","children":[{"name":"dev.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/dev.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"},{"name":"prod.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/prod.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"},{"name":"stage.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/stage.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/dev.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/prod.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/stage.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"kafka-rest","children":[{"name":"config","children":[{"name":"dev.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/dev.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"},{"name":"prod.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/prod.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"},{"name":"stage.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/stage.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/dev.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/prod.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/stage.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"schema-registry","children":[{"name":"config","children":[{"name":"dev.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/dev.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"},{"name":"prod.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/prod.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"},{"name":"stage.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/stage.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/dev.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/prod.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/stage.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]}]}]},{"name":"configure-common-dependencies.sh","path":"examples/confluent-oss-ami/configure-common-dependencies.sh","sha":"0b2923cd75314f12d8b2bf338b0e2db654d827a9"},{"name":"configure-kafka-connect.sh","path":"examples/confluent-oss-ami/configure-kafka-connect.sh","sha":"ec4542292ad6d60f67c122fc33698e482d05ec43"},{"name":"configure-kafka-rest.sh","path":"examples/confluent-oss-ami/configure-kafka-rest.sh","sha":"5184cfc4b5396930d92ee694e68c9bc9e448d71a"},{"name":"configure-schema-registry.sh","path":"examples/confluent-oss-ami/configure-schema-registry.sh","sha":"ac4ed5c06630d352cb077147560c2fe02b9217f1"},{"name":"confluent-oss.json","path":"examples/confluent-oss-ami/confluent-oss.json","sha":"00194938b3a1ee8fbcb6ea1a640c869a495a3d6c"},{"name":"ssl","children":[{"name":"README.md","path":"examples/confluent-oss-ami/ssl/README.md","sha":"2b7b50749a90c78e026f597f677e44ece9f2458d"},{"name":"ca-cert","path":"examples/confluent-oss-ami/ssl/ca-cert","sha":"fb02e172efcdc4ad4c660e137059be86926108f4"},{"name":"cert","path":"examples/confluent-oss-ami/ssl/cert","sha":"0f486b16f80eebe97d9135542d229404a6b48ddc"},{"name":"kafka","children":[{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]}]}]},{"name":"kafka-ami","children":[{"name":"README.md","path":"examples/kafka-ami/README.md","sha":"692e94969d0352ab15fb3f87a67b44085a1285a5","toggled":true},{"name":"check-for-kafka-key-store.sh","path":"examples/kafka-ami/check-for-kafka-key-store.sh","sha":"27141a3a4a2ffe07fae4e83503d833ef3e3ec36b"},{"name":"config","children":[{"name":"README.md","path":"examples/kafka-ami/config/README.md","sha":"ec0ff5b551e31f4783712d3ed169dbbb757ca9f9"},{"name":"kafka","children":[{"name":"config","children":[{"name":"dev.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/dev.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"},{"name":"prod.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/prod.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"},{"name":"stage.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/stage.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/dev.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"},{"name":"prod.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/prod.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"},{"name":"stage.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/stage.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]}]},{"name":"configure-kafka-server.sh","path":"examples/kafka-ami/configure-kafka-server.sh","sha":"530dadf54e3fdb78a2b4cab532b0c669515c7a4b"},{"name":"kafka.json","path":"examples/kafka-ami/kafka.json","sha":"92552c3a8b1dcdf96bfbbf8944d9ef7f7dbfdca3"},{"name":"ssl","children":[{"name":"README.md","path":"examples/kafka-ami/ssl/README.md","sha":"51859e48ac5ba48f1278f479d38112e69e761fa3"}]}],"toggled":true},{"name":"kafka-zookeeper-confluent-oss-ami","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/README.md","sha":"1626f60062f2b2ba3d8352a459067b5fae812dbf"},{"name":"config","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/config/README.md","sha":"c14625eae0097f9e3428ce11d5a0a39580bde664"},{"name":"kafka-connect","children":[{"name":"config","children":[{"name":"worker-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-connect/config/worker-4.0.x.properties","sha":"7cde3283393b1b511a79b87936a0430c4607641f"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-connect/log4j/log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"kafka-rest","children":[{"name":"config","children":[{"name":"kafka-rest-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-rest/config/kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-rest/log4j/log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"kafka","children":[{"name":"config","children":[{"name":"server-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka/config/server-4.0.x.properties","sha":"f9ae2462af98e6e98f4b64df21be5bbdd4df56a9"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka/log4j/log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]},{"name":"schema-registry","children":[{"name":"config","children":[{"name":"schema-registry-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/schema-registry/config/schema-registry-4.0.x.properties","sha":"6d499d60b09982b45424f4060e04c586dd39287c"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/schema-registry/log4j/log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]}]}]},{"name":"configure-kafka-zk-confluent-server.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/configure-kafka-zk-confluent-server.sh","sha":"9951ea6b1d7b8fb5fba6b5c1e0c071c94f59d6f1"},{"name":"docker-compose.yml","path":"examples/kafka-zookeeper-confluent-oss-ami/docker-compose.yml","sha":"8beedfce4e773f452dee733f4f26cf5e8b0cb763"},{"name":"kafka-zookeeper-confluent-oss.json","path":"examples/kafka-zookeeper-confluent-oss-ami/kafka-zookeeper-confluent-oss.json","sha":"e4e62a5d33f1343788c5706746b6f7ea5e40a8bd"},{"name":"mock","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/README.md","sha":"d373af41223ad92e574a958e4e17b78da61fe725"},{"name":"bash-commons","children":[{"name":"aws.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/bash-commons/aws.sh","sha":"96813a5b3c4e6c09193e2096754e2b5c22894146"},{"name":"docker.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/bash-commons/docker.sh","sha":"92d0daf43d60c6ae506eaf51a373e59ffef1b7f6"}]},{"name":"modules","children":[{"name":"attach-eni","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/modules/attach-eni","sha":"da052caea4586b27c2dc13e521092e9403fcc327"},{"name":"mount-ebs-volume","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/modules/mount-ebs-volume","sha":"b29c3826e9d6c688043bcb47466652a26a543a4b"}]},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/user-data/user-data.sh","sha":"31964f5f291ce525a04c0aedd3a9c8c84557944a"}]}]},{"name":"wait_for_zk.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/wait_for_zk.sh","sha":"0ac5e9e1bb712d727f14c8777b35c0ccfbfbf59e"}]},{"name":"kafka-zookeeper-confluent-oss-colocated-cluster","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/README.md","sha":"1ab7d9295b492aee332af4c8e507320eef2351ee"},{"name":"main.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/main.tf","sha":"5a8c4d0366f493d800e10a8ab130efcb17ee1f53"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/outputs.tf","sha":"331d0632825a6f349d01751717ecf84bb08581c7"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/user-data/user-data.sh","sha":"a63ab90111ce199b179f3db24f04e00e6cfeaf69"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/vars.tf","sha":"b671f2f0d2f1ac9888ec630f42275ccd7e3d6a4e"}]},{"name":"kafka-zookeeper-confluent-oss-standalone-clusters","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/README.md","sha":"c951818155d03b44f3335a54da07f2ba43360bbf"},{"name":"main.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/main.tf","sha":"6d8e6c6c568e0c9f75370429299583d710bb669d"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/outputs.tf","sha":"3c12deec6a6f70912b60bd14f20ac5f40152046e"},{"name":"user-data","children":[{"name":"confluent-tools-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/confluent-tools-cluster-user-data.sh","sha":"01b77ddab8cebc5367c05af25f361113d1d34a5f"},{"name":"kafka-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/kafka-cluster-user-data.sh","sha":"4216a3d67c3fab1db94c78d290eb685952268b46"},{"name":"zookeeper-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/zookeeper-cluster-user-data.sh","sha":"e5edb3e727f377436ae9c991cd2c7b01fc52baba"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/vars.tf","sha":"80ed87cfc775f8e81415d9c6d0d636fa4eec4fed"}]},{"name":"kafka-zookeeper-standalone-clusters","children":[{"name":"README.md","path":"examples/kafka-zookeeper-standalone-clusters/README.md","sha":"9780235b75ceb497d3fd299c4794eb0f2601ef15"},{"name":"main.tf","path":"examples/kafka-zookeeper-standalone-clusters/main.tf","sha":"5b09f6956f534e8c6002b19a4911eeb5c36a0f2c"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-standalone-clusters/outputs.tf","sha":"051d6e5ddda7cef04fd9f5031a57694124762eca"},{"name":"user-data","children":[{"name":"kafka-user-data.sh","path":"examples/kafka-zookeeper-standalone-clusters/user-data/kafka-user-data.sh","sha":"ccb4d9324cabce27d4617c7e32ea10a10014385a"},{"name":"zookeeper-user-data.sh","path":"examples/kafka-zookeeper-standalone-clusters/user-data/zookeeper-user-data.sh","sha":"2e144c5dc55c4e721ecf595ae60df4553024191f"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-standalone-clusters/vars.tf","sha":"fc70b12e7d2c784e3a491ad08d10005f44ed71a7"}]},{"name":"zookeeper-ami","children":[{"name":"README.md","path":"examples/zookeeper-ami/README.md","sha":"6ebd1619152754561ae480c0ec00945ff7d2df43"},{"name":"configure-zookeeper-server.sh","path":"examples/zookeeper-ami/configure-zookeeper-server.sh","sha":"7ff6dbe9cf785b6d832fdd37c7d0f1baa24e97d2"},{"name":"zookeeper.json","path":"examples/zookeeper-ami/zookeeper.json","sha":"9393f3e97c6f9f6df9161bd9dc5da6f36b629794"}]}],"toggled":true},{"name":"modules","children":[{"name":"bash-commons","children":[{"name":"README.md","path":"modules/bash-commons/README.md","sha":"0b7b7bf23db870999e14ee833d05488ba44a136c"},{"name":"install.sh","path":"modules/bash-commons/install.sh","sha":"bedb09f6eaa00a323ae1dd814afec954ca3efeeb"},{"name":"lib","children":[{"name":"array.sh","path":"modules/bash-commons/lib/array.sh","sha":"2d4e0ef22dc608392e99522e8ff0eb68ed1f708c"},{"name":"assert.sh","path":"modules/bash-commons/lib/assert.sh","sha":"bfaf1740050694ed05d03bcff7dbc99724c4fc43"},{"name":"aws.sh","path":"modules/bash-commons/lib/aws.sh","sha":"e6986c813e1fef28dfd5881b0193c4925e8dc66b"},{"name":"file.sh","path":"modules/bash-commons/lib/file.sh","sha":"196b04006ff622844d6d198f78130b2d2fd7c0c6"},{"name":"java.sh","path":"modules/bash-commons/lib/java.sh","sha":"3cc8614fd91c2d9e0816555e558ee12ba0a8c95b"},{"name":"log.sh","path":"modules/bash-commons/lib/log.sh","sha":"1b5887a63f9e7de613707866753e2cfe910da4d1"},{"name":"os.sh","path":"modules/bash-commons/lib/os.sh","sha":"3371306dc7959874cf6b8935d9454e5c1c942c4d"},{"name":"strings.sh","path":"modules/bash-commons/lib/strings.sh","sha":"67a96995df1886ff0d4ce528b5fb26cfbe7b044d"}]}]},{"name":"confluent-tools-cluster","children":[{"name":"README.md","path":"modules/confluent-tools-cluster/README.md","sha":"658308333cc07a35f46a8ec04ff882c74db52bfe"},{"name":"main.tf","path":"modules/confluent-tools-cluster/main.tf","sha":"3e8dc317addb6b6597f1b1d8bf21c1de423d28bf"},{"name":"outputs.tf","path":"modules/confluent-tools-cluster/outputs.tf","sha":"12f380d0aff454ad6d556c116d84c344a62bb999"},{"name":"vars.tf","path":"modules/confluent-tools-cluster/vars.tf","sha":"8234819f003e0566e96cb3a5fa14f71e6020139d"}]},{"name":"confluent-tools-iam-permissions","children":[{"name":"README.md","path":"modules/confluent-tools-iam-permissions/README.md","sha":"06057008bc666088f319da89baa9d16b68fdce30"},{"name":"main.tf","path":"modules/confluent-tools-iam-permissions/main.tf","sha":"acca97b3a587bc88a5d5d72ad42b642f862ea3cb"},{"name":"vars.tf","path":"modules/confluent-tools-iam-permissions/vars.tf","sha":"9803fc14dc414f11fc9f236685dbb0db8f5273e6"}]},{"name":"confluent-tools-security-group-rules","children":[{"name":"README.md","path":"modules/confluent-tools-security-group-rules/README.md","sha":"5ee75e53e2ba7c2893e147fe86a2ba48fc8feab0"},{"name":"main.tf","path":"modules/confluent-tools-security-group-rules/main.tf","sha":"1112e576a3e0bca04bdb3cd9d23e45e2442d4fb1"},{"name":"vars.tf","path":"modules/confluent-tools-security-group-rules/vars.tf","sha":"3faf1ad90e4c918aefbec1545235ecf97ecc08cf"}]},{"name":"generate-key-stores","children":[{"name":"README.md","path":"modules/generate-key-stores/README.md","sha":"e4cacd66c8857a552810dc5a8d328a5395659392"},{"name":"generate-key-stores.sh","path":"modules/generate-key-stores/generate-key-stores.sh","sha":"0a88e8990323488496bbd2dd7d22d99ce3a5e2ed"},{"name":"install.sh","path":"modules/generate-key-stores/install.sh","sha":"33c6e02e94425b4d58050ca7719e70c0292e1ba6"}]},{"name":"install-confluent-tools","children":[{"name":"README.md","path":"modules/install-confluent-tools/README.md","sha":"ca24e1b76b9a0299b4c36e07bff2941e6960e0c1"},{"name":"install.sh","path":"modules/install-confluent-tools/install.sh","sha":"6d49d8f343a5ed55acc8367d60b11b4af7a916a8"},{"name":"security","children":[{"name":"confluent.key","path":"modules/install-confluent-tools/security/confluent.key","sha":"1025a2c6dfa66f224c0c45ac172fd8d3efce1744"}]}]},{"name":"install-kafka","children":[{"name":"README.md","path":"modules/install-kafka/README.md","sha":"c94e75bda5a0c753d35a67d785af65d46b551e5b"},{"name":"install.sh","path":"modules/install-kafka/install.sh","sha":"4c3f21c4a4a868841a6a813a82a2c59c9977a1db"}]},{"name":"kafka-cluster","children":[{"name":"README.md","path":"modules/kafka-cluster/README.md","sha":"a054f23f1a0aebcc5be1f1ec0368bb67f17e20f6"},{"name":"main.tf","path":"modules/kafka-cluster/main.tf","sha":"6ce0d49d1d9a28c34e210a5ef97ae18adcd43fef"},{"name":"outputs.tf","path":"modules/kafka-cluster/outputs.tf","sha":"d53790ae73bc413f3a21acb52064bddf30d0d517"},{"name":"vars.tf","path":"modules/kafka-cluster/vars.tf","sha":"7bee28a5d8850ee7ed6647941776f07b7a488560"}]},{"name":"kafka-iam-permissions","children":[{"name":"README.md","path":"modules/kafka-iam-permissions/README.md","sha":"3763dd58a4cd52e71b5ffe0d3e5e9fe0cf48053f"},{"name":"main.tf","path":"modules/kafka-iam-permissions/main.tf","sha":"78309ea5cc1228dd8a3289ae56a6594da1e17f5a"},{"name":"vars.tf","path":"modules/kafka-iam-permissions/vars.tf","sha":"d29e8e5a07834701c3050e6369607747f85e43d8"}]},{"name":"kafka-security-group-rules","children":[{"name":"README.md","path":"modules/kafka-security-group-rules/README.md","sha":"63627f3af8842e4af90d0c31337004ef2503fc9c"},{"name":"main.tf","path":"modules/kafka-security-group-rules/main.tf","sha":"7f1bae14bffd2b3a9ce53c910144c52970d56e10"},{"name":"vars.tf","path":"modules/kafka-security-group-rules/vars.tf","sha":"9c27d84eb7bfd6291acafc3e844f0ab7e1ec970f"}]},{"name":"run-health-checker","children":[{"name":"README.md","path":"modules/run-health-checker/README.md","sha":"8bc3cab46b5eecb25e642364dbd57ea35a0be71c"},{"name":"bin","children":[{"name":"run-health-checker","path":"modules/run-health-checker/bin/run-health-checker","sha":"6fd6be586f4bb541aea624748b3476403c748c73"}]},{"name":"install.sh","path":"modules/run-health-checker/install.sh","sha":"46d9a77efa02421828f16751d37dbd1f95c030cd"}]},{"name":"run-kafka-connect","children":[{"name":"README.md","path":"modules/run-kafka-connect/README.md","sha":"0f180c7e494588f218cfe2ace99c671c8dfcadcf"},{"name":"bin","children":[{"name":"run-kafka-connect","path":"modules/run-kafka-connect/bin/run-kafka-connect","sha":"aa5b4ab8059dce08ad261e3f8b30197129899ec8"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka-connect/config/README.md","sha":"5c6f2b6e63f1eba41957dd083c0787ac139c9622"},{"name":"kafka-connect","children":[{"name":"worker-3.3.x.properties","path":"modules/run-kafka-connect/config/kafka-connect/worker-3.3.x.properties","sha":"6bd29bb369f3aaea951e0a00d157ba167e943813"},{"name":"worker-4.0.x.properties","path":"modules/run-kafka-connect/config/kafka-connect/worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka-connect/config/log4j/log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"install.sh","path":"modules/run-kafka-connect/install.sh","sha":"feeab73124981a2d2fb61b0f20c69a1c0d919c0e"},{"name":"security","children":[{"name":"README.md","path":"modules/run-kafka-connect/security/README.md","sha":"8f7dd7449eba0f9f49085af72a7cfa03d4b6d85d"},{"name":"confluent-3.3.1-2.11.tar.gz.checksum","path":"modules/run-kafka-connect/security/confluent-3.3.1-2.11.tar.gz.checksum","sha":"c7aed490972e7b1565795221488d18449bd0bae1"},{"name":"confluent-4.0.0-2.11.tar.gz.checksum","path":"modules/run-kafka-connect/security/confluent-4.0.0-2.11.tar.gz.checksum","sha":"27b7a13f188475b4157386aa2761915633b72aa3"}]}]},{"name":"run-kafka-rest","children":[{"name":"README.md","path":"modules/run-kafka-rest/README.md","sha":"93c501f797818e3fee1e078cfbb93cc35577cc91"},{"name":"bin","children":[{"name":"run-kafka-rest","path":"modules/run-kafka-rest/bin/run-kafka-rest","sha":"2bf9688efcf50cd5f49a1b9d6e8fa282973d8674"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka-rest/config/README.md","sha":"772b1a95a54aadd3c0f0dce34eac10f3d1967634"},{"name":"kafka-rest","children":[{"name":"kafka-rest-3.3.x.properties","path":"modules/run-kafka-rest/config/kafka-rest/kafka-rest-3.3.x.properties","sha":"21263ea344efbfa26e1a792b8bb29c33606c0d7a"},{"name":"kafka-rest-4.0.x.properties","path":"modules/run-kafka-rest/config/kafka-rest/kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka-rest/config/log4j/log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"install.sh","path":"modules/run-kafka-rest/install.sh","sha":"7f9af0bcc67e0ac3380c7f34ea7df8f72555aeda"}]},{"name":"run-kafka","children":[{"name":"README.md","path":"modules/run-kafka/README.md","sha":"b1fcb424860e462141f9a17423b0c3ecc3f2de08"},{"name":"bin","children":[{"name":"run-kafka","path":"modules/run-kafka/bin/run-kafka","sha":"ee4aebad8587284a006b27fe5160d9574a862b1d"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka/config/README.md","sha":"e08702423b137b254ad0a0a070f7300f094ba046"},{"name":"kafka","children":[{"name":"server-3.3.x.properties","path":"modules/run-kafka/config/kafka/server-3.3.x.properties","sha":"b5aa3757d41f0e6fef81799b58c222774a0da63d"},{"name":"server-4.0.x.properties","path":"modules/run-kafka/config/kafka/server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka/config/log4j/log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]},{"name":"install.sh","path":"modules/run-kafka/install.sh","sha":"155b136dc2514a3531cc6a1419814ecd9ca17449"}]},{"name":"run-schema-registry","children":[{"name":"README.md","path":"modules/run-schema-registry/README.md","sha":"343ddd7b1e9b24054c05722b3cb6ea4e63d88419"},{"name":"bin","children":[{"name":"run-schema-registry","path":"modules/run-schema-registry/bin/run-schema-registry","sha":"71611caec137fa810443f44f14ae97ce7b1d3592"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-schema-registry/config/README.md","sha":"93a30f8adf3f778682463bbc8715a38f70908866"},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-schema-registry/config/log4j/log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]},{"name":"schema-registry","children":[{"name":"schema-registry.properties","path":"modules/run-schema-registry/config/schema-registry/schema-registry.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"}]}]},{"name":"install.sh","path":"modules/run-schema-registry/install.sh","sha":"11a3e89d9d4984939428e6e48ce9134e00d97879"}]}]},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"cf72a9f58a3b36aee8053b37dfc6e1a617f93f7b"},{"name":"generate_key_stores_test.go","path":"test/generate_key_stores_test.go","sha":"37d3d1bab86512871d9a6921bbd6b32fecd6f227"},{"name":"go.mod","path":"test/go.mod","sha":"f1828c6afad9169dc7440c34a8d9e84ad05bdf2f"},{"name":"go.sum","path":"test/go.sum","sha":"80d4beefaba600ff4b616d7d0c3095bb2e422e7a"},{"name":"kafka_zookeeper_confluent_colocated_cluster_test.go","path":"test/kafka_zookeeper_confluent_colocated_cluster_test.go","sha":"d699e24b76753322060acf50c46b4adfa1712061"},{"name":"kafka_zookeeper_confluent_standalone_clusters_test.go","path":"test/kafka_zookeeper_confluent_standalone_clusters_test.go","sha":"063205c0c5e351f7ee12bbe90906bc3b72e31ec8"},{"name":"kafka_zookeeper_standalone_clusters_test.go","path":"test/kafka_zookeeper_standalone_clusters_test.go","sha":"6d69aee5213518cf624ca9cd73270479b4b9e0d3"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"532ce67ff4c3130ccdcc51eebce9f6d39adb0082"},{"name":"test_helpers_kafka.go","path":"test/test_helpers_kafka.go","sha":"492d13274f46c259e789b93fff1ae4211fa5aad5"},{"name":"test_helpers_kafka_connect.go","path":"test/test_helpers_kafka_connect.go","sha":"1ecef92e9a45501fb4bf834579d02ddcc05e7103"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"d25c865cdc1c4d0940a3f75cd6de791a2bafae67"},{"name":"test_helpers_rest_proxy.go","path":"test/test_helpers_rest_proxy.go","sha":"b0e1a53cc2ec9e8b4c369f07387301e7e42f8c7d"},{"name":"test_helpers_schema_registry.go","path":"test/test_helpers_schema_registry.go","sha":"b15a39916cd41441705265ccabeb23e0460b260f"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"kafka-example-ami\">Kafka Example AMI</h1><div class=\"preview__body--border\"></div><p>This folder contains an example <a href=\"https://www.packer.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Packer</a> template that can be used to create an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon\nMachine Image (AMI)</a> with Kafka and all of its\ndependencies installed. A few notes about this Packer template:</p>\n<ul>\n<li>\n<p>It builds two AMIs, one on top of Amazon Linux and one on top of Ubuntu. Pick which ever distro you prefer.</p>\n</li>\n<li>\n<p>This AMI includes an example of how to configure Kafka with SSL. It expects you to use the <a href=\"/repos/v0.10.1/package-kafka/modules/generate-key-stores\" class=\"preview__body--description--blue\">generate-key-stores\nmodule</a> to generate a Key Store and Trust Store before building the AMI.</p>\n</li>\n<li>\n<p>This AMI <em>only</em> includes Kafka. It does not include ZooKeeper, which is expected to be running in a separate cluster,\nwith a separate AMI (see the <a href=\"/repos/v0.10.1/package-kafka/examples/zookeeper-ami\" class=\"preview__body--description--blue\">zookeeper-ami example</a>). For an example of Kafka and ZooKeeper\nco-located in the same cluster, see <a href=\"/repos/v0.10.1/package-kafka/examples/kafka-zookeeper-confluent-oss-ami\" class=\"preview__body--description--blue\">kafka-zookeeper-ami</a>.</p>\n</li>\n</ul>\n<p>To deploy this AMI, see the <a href=\"/repos/v0.10.1/package-kafka/examples/kafka-zookeeper-standalone-clusters\" class=\"preview__body--description--blue\">kafka-zookeeper-standalone-clusters example</a>.</p>\n<h2 class=\"preview__body--subtitle\" id=\"quick-start\">Quick start</h2>\n<p>To build the AMI:</p>\n<ol>\n<li>Run the <a href=\"/repos/v0.10.1/package-kafka/modules/generate-key-stores\" class=\"preview__body--description--blue\">generate-key-stores module</a> to create a Key Store, Trust Store, and\nself-signed SSL certificate. You can run the <code>check-for-key-store.sh</code> script to print out example usage for the\n<code>generate-key-stores.sh</code> script.</li>\n<li>Install <a href=\"https://www.packer.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Packer</a>.</li>\n<li>Set up your <a href=\"https://www.packer.io/docs/builders/amazon.html\" class=\"preview__body--description--blue\" target=\"_blank\">AWS credentials as environment variables</a>.</li>\n<li>Set the <code>GITHUB_OAUTH_TOKEN</code> environment variable to a valid GitHub auth token with "repo" access. You can generate\none here: https://github.com/settings/tokens</li>\n<li>Run <code>packer build -var "package_kafka_branch=<branch>" -only <target os> kafka.json</code> to create a new AMI in your AWS account. Note down the ID of this new AMI.</li>\n</ol>\n<p>To deploy the new AMI, plug its ID into the <a href=\"/repos/v0.10.1/package-kafka/examples/kafka-zookeeper-standalone-clusters\" class=\"preview__body--description--blue\">kafka-zookeeper-standalone-clusters\nexample</a>.</p>\n","repoName":"package-kafka","repoRef":"v0.11.0","serviceDescriptor":{"serviceName":"Apache Kafka and Confluent Tools","serviceRepoName":"package-kafka","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a cluster of Kafka brokers. Optionally deploy Confluent tools such as Schema Registry, REST Proxy, and Kafka Connect.","imageUrl":"kafka.png","licenseType":"subscriber","technologies":["Terraform","Bash"],"compliance":[],"tags":[""]},"serviceCategoryName":"Messaging & streaming","fileName":"README.md","filePath":"/examples/kafka-ami","title":"Repo Browser: Apache Kafka and Confluent Tools","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
check-for-kafka-key-store.sh
