Kafka/Zookeeper/Kafka Connect/Schema Registry/REST Proxy Example AMI
This folder contains an example Packer template that can be used to create an Amazon
Machine Image (AMI) with Kafka, ZooKeeper, Kafka Connect,
Schema Registry, REST Proxy, and all of their dependencies installed. A few notes about this Packer template:
It builds two AMIs, one on top of Amazon Linux and one on top of Ubuntu. Pick which ever distro you prefer.
This AMI includes Kafka, ZooKeeper, Kafka Connect, Schema Registry and REST Proxy with the expectation that you will
run all of these co-located in the same cluster. The ZooKeeper dependencies come from terraform-aws-zookeeper. All other dependencies live in this repo.
A colocated cluster is useful for deploying a less expensinve non-production environment. For a production-ready example
that configures these services to run in three separate clusters, see the kafka-ami
, zookeeper-ami, and confluent-oss-ami examples.
This AMI example does not include SSL support.
If you want your Kafka brokers and Kafka Connect Workers to communicate over SSL, see the kafka-ami example.
If you want your Schema Registry and REST Proxy endpoints to communicate over SSL, see the confluent-oss-ami example.
Set the GITHUB_OAUTH_TOKEN environment variable to a valid GitHub auth token with "repo" access. You can generate
one here: https://github.com/settings/tokens
Run packer build -var "package_kafka_branch=<branch>" -only <target os> kafka-zookeeper-confluent-oss.json to create a new AMI in your AWS account. Note down the ID of this new AMI.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"f6a1d049f1988183bd05a90c285533dfcdeaf48a"}]},{"name":".gitignore","path":".gitignore","sha":"e68eece82c5bbdddf63121f38b66cdea255b5567"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"8f0a49e6e74c419dd55216b6397d21c6cc2e1029"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"4be01a6334d39aa5bf6abe6baae701f5e2a8c5ac"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"689cf10ec98e3297a75bdd9b9fb5da10b7a675f8"},{"name":"README.md","path":"README.md","sha":"3b28f35855c132ef18878116e94b623ef2831d0d"},{"name":"examples","children":[{"name":"confluent-oss-ami","children":[{"name":"README.md","path":"examples/confluent-oss-ami/README.md","sha":"675ce60c000b84facb441e96fa16e89614fe2b6b"},{"name":"check-for-kafka-trust-store.sh","path":"examples/confluent-oss-ami/check-for-kafka-trust-store.sh","sha":"e2e7d323e5153471809af13b4434c9edfad1b85b"},{"name":"check-for-key-store.sh","path":"examples/confluent-oss-ami/check-for-key-store.sh","sha":"120539a6d63dfd2ed208e92c0a25a32b6b86a2bd"},{"name":"config","children":[{"name":"README.md","path":"examples/confluent-oss-ami/config/README.md","sha":"50e3000eda90f23880d9f34da435f1b170e213f5"},{"name":"kafka-connect","children":[{"name":"config","children":[{"name":"dev.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/dev.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"},{"name":"prod.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/prod.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"},{"name":"stage.worker-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-connect/config/stage.worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/dev.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/prod.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-connect/log4j/stage.log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"kafka-rest","children":[{"name":"config","children":[{"name":"dev.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/dev.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"},{"name":"prod.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/prod.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"},{"name":"stage.kafka-rest-4.0.x.properties","path":"examples/confluent-oss-ami/config/kafka-rest/config/stage.kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/dev.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/prod.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/kafka-rest/log4j/stage.log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"schema-registry","children":[{"name":"config","children":[{"name":"dev.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/dev.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"},{"name":"prod.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/prod.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"},{"name":"stage.schema-registry-4.0.x.properties","path":"examples/confluent-oss-ami/config/schema-registry/config/stage.schema-registry-4.0.x.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/dev.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"},{"name":"prod.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/prod.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"},{"name":"stage.log4j.properties","path":"examples/confluent-oss-ami/config/schema-registry/log4j/stage.log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]}]}]},{"name":"configure-common-dependencies.sh","path":"examples/confluent-oss-ami/configure-common-dependencies.sh","sha":"a5b7c815d15309c39f3aff9e9ad004a029057de8"},{"name":"configure-kafka-connect.sh","path":"examples/confluent-oss-ami/configure-kafka-connect.sh","sha":"effeed2d32e0ea2878b1d5ad726452e21af72647"},{"name":"configure-kafka-rest.sh","path":"examples/confluent-oss-ami/configure-kafka-rest.sh","sha":"aa9b04bb10946d50a22157a351ba859935b0350c"},{"name":"configure-schema-registry.sh","path":"examples/confluent-oss-ami/configure-schema-registry.sh","sha":"6ecc2c129f17a28e81ff45edade8a57e56bc9e07"},{"name":"confluent-oss.json","path":"examples/confluent-oss-ami/confluent-oss.json","sha":"8d9a8100d0c6823d0e670b0b9b1b32bf43640922"},{"name":"ssl","children":[{"name":"README.md","path":"examples/confluent-oss-ami/ssl/README.md","sha":"2b7b50749a90c78e026f597f677e44ece9f2458d"},{"name":"ca-cert","path":"examples/confluent-oss-ami/ssl/ca-cert","sha":"fb02e172efcdc4ad4c660e137059be86926108f4"},{"name":"cert","path":"examples/confluent-oss-ami/ssl/cert","sha":"0f486b16f80eebe97d9135542d229404a6b48ddc"},{"name":"kafka-connect","children":[{"name":"keystore","children":[{"name":"dev.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/keystore/dev.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"prod.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/keystore/prod.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"stage.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/keystore/stage.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"}]},{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-connect/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]},{"name":"kafka-rest","children":[{"name":"keystore","children":[{"name":"dev.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/keystore/dev.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"prod.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/keystore/prod.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"stage.keystore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/keystore/stage.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"}]},{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka-rest/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]},{"name":"kafka","children":[{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/confluent-oss-ami/ssl/kafka/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]},{"name":"schema-registry","children":[{"name":"keystore","children":[{"name":"dev.keystore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/keystore/dev.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"prod.keystore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/keystore/prod.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"stage.keystore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/keystore/stage.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"}]},{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/confluent-oss-ami/ssl/schema-registry/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]}]}]},{"name":"kafka-ami","children":[{"name":"README.md","path":"examples/kafka-ami/README.md","sha":"692e94969d0352ab15fb3f87a67b44085a1285a5"},{"name":"check-for-kafka-key-store.sh","path":"examples/kafka-ami/check-for-kafka-key-store.sh","sha":"27141a3a4a2ffe07fae4e83503d833ef3e3ec36b"},{"name":"config","children":[{"name":"README.md","path":"examples/kafka-ami/config/README.md","sha":"ec0ff5b551e31f4783712d3ed169dbbb757ca9f9"},{"name":"kafka","children":[{"name":"config","children":[{"name":"dev.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/dev.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"},{"name":"prod.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/prod.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"},{"name":"stage.server-4.0.x.properties","path":"examples/kafka-ami/config/kafka/config/stage.server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"}]},{"name":"log4j","children":[{"name":"dev.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/dev.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"},{"name":"prod.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/prod.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"},{"name":"stage.log4j.properties","path":"examples/kafka-ami/config/kafka/log4j/stage.log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]}]},{"name":"configure-kafka-server.sh","path":"examples/kafka-ami/configure-kafka-server.sh","sha":"e4206f8df32e89810d8eff1be13be1e90df733b4"},{"name":"kafka.json","path":"examples/kafka-ami/kafka.json","sha":"98b106f115513800b58723a8d97f632cdda65c11"},{"name":"ssl","children":[{"name":"README.md","path":"examples/kafka-ami/ssl/README.md","sha":"51859e48ac5ba48f1278f479d38112e69e761fa3"},{"name":"kafka","children":[{"name":"ca-cert","path":"examples/kafka-ami/ssl/kafka/ca-cert","sha":"fb02e172efcdc4ad4c660e137059be86926108f4"},{"name":"cert","path":"examples/kafka-ami/ssl/kafka/cert","sha":"0f486b16f80eebe97d9135542d229404a6b48ddc"},{"name":"keystore","children":[{"name":"dev.keystore.jks","path":"examples/kafka-ami/ssl/kafka/keystore/dev.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"prod.keystore.jks","path":"examples/kafka-ami/ssl/kafka/keystore/prod.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"},{"name":"stage.keystore.jks","path":"examples/kafka-ami/ssl/kafka/keystore/stage.keystore.jks","sha":"6283b3e9b655c2a987192e81b3a6172e6c9ea487"}]},{"name":"truststore","children":[{"name":"dev.truststore.jks","path":"examples/kafka-ami/ssl/kafka/truststore/dev.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"prod.truststore.jks","path":"examples/kafka-ami/ssl/kafka/truststore/prod.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"},{"name":"stage.truststore.jks","path":"examples/kafka-ami/ssl/kafka/truststore/stage.truststore.jks","sha":"9545e6ac795144d714c23f252abe79f4811d4d89"}]}]}]}]},{"name":"kafka-zookeeper-confluent-oss-ami","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/README.md","sha":"1626f60062f2b2ba3d8352a459067b5fae812dbf","toggled":true},{"name":"config","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/config/README.md","sha":"c14625eae0097f9e3428ce11d5a0a39580bde664"},{"name":"kafka-connect","children":[{"name":"config","children":[{"name":"worker-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-connect/config/worker-4.0.x.properties","sha":"7cde3283393b1b511a79b87936a0430c4607641f"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-connect/log4j/log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"kafka-rest","children":[{"name":"config","children":[{"name":"kafka-rest-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-rest/config/kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka-rest/log4j/log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"kafka","children":[{"name":"config","children":[{"name":"server-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka/config/server-4.0.x.properties","sha":"f9ae2462af98e6e98f4b64df21be5bbdd4df56a9"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/kafka/log4j/log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]},{"name":"schema-registry","children":[{"name":"config","children":[{"name":"schema-registry-4.0.x.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/schema-registry/config/schema-registry-4.0.x.properties","sha":"6d499d60b09982b45424f4060e04c586dd39287c"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"examples/kafka-zookeeper-confluent-oss-ami/config/schema-registry/log4j/log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]}]}]},{"name":"configure-kafka-zk-confluent-server.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/configure-kafka-zk-confluent-server.sh","sha":"eaa35dd030e655d16b6e0e170e569aa5aba0a550"},{"name":"docker-compose.yml","path":"examples/kafka-zookeeper-confluent-oss-ami/docker-compose.yml","sha":"8beedfce4e773f452dee733f4f26cf5e8b0cb763"},{"name":"kafka-zookeeper-confluent-oss.json","path":"examples/kafka-zookeeper-confluent-oss-ami/kafka-zookeeper-confluent-oss.json","sha":"996c854d2065434180500881778d184615bbcb61"},{"name":"mock","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/README.md","sha":"d373af41223ad92e574a958e4e17b78da61fe725"},{"name":"bash-commons","children":[{"name":"aws.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/bash-commons/aws.sh","sha":"ce067be902c8c7c49b85bb395f8eb50b87a535e6"},{"name":"docker.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/bash-commons/docker.sh","sha":"7827db443288057e5f2df9f43a955bc2afa464a4"}]},{"name":"modules","children":[{"name":"attach-eni","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/modules/attach-eni","sha":"da052caea4586b27c2dc13e521092e9403fcc327"},{"name":"mount-ebs-volume","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/modules/mount-ebs-volume","sha":"9b81549efc7c94e5baf609918e4831dd780eee2f"}]},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/mock/user-data/user-data.sh","sha":"31964f5f291ce525a04c0aedd3a9c8c84557944a"}]}]},{"name":"wait_for_zk.sh","path":"examples/kafka-zookeeper-confluent-oss-ami/wait_for_zk.sh","sha":"0ac5e9e1bb712d727f14c8777b35c0ccfbfbf59e"}],"toggled":true},{"name":"kafka-zookeeper-confluent-oss-colocated-cluster","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/README.md","sha":"1ab7d9295b492aee332af4c8e507320eef2351ee"},{"name":"main.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/main.tf","sha":"906ff574edf90d0fe06522099ea3166eb762ea36"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/outputs.tf","sha":"331d0632825a6f349d01751717ecf84bb08581c7"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/user-data/user-data.sh","sha":"a63ab90111ce199b179f3db24f04e00e6cfeaf69"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-confluent-oss-colocated-cluster/vars.tf","sha":"b671f2f0d2f1ac9888ec630f42275ccd7e3d6a4e"}]},{"name":"kafka-zookeeper-confluent-oss-standalone-clusters","children":[{"name":"README.md","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/README.md","sha":"c951818155d03b44f3335a54da07f2ba43360bbf"},{"name":"main.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/main.tf","sha":"a6b34d7b2150b7ba907470c6f612eee2f68f5e59"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/outputs.tf","sha":"3c12deec6a6f70912b60bd14f20ac5f40152046e"},{"name":"user-data","children":[{"name":"confluent-tools-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/confluent-tools-cluster-user-data.sh","sha":"01b77ddab8cebc5367c05af25f361113d1d34a5f"},{"name":"kafka-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/kafka-cluster-user-data.sh","sha":"4216a3d67c3fab1db94c78d290eb685952268b46"},{"name":"zookeeper-cluster-user-data.sh","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/user-data/zookeeper-cluster-user-data.sh","sha":"e5edb3e727f377436ae9c991cd2c7b01fc52baba"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-confluent-oss-standalone-clusters/vars.tf","sha":"80ed87cfc775f8e81415d9c6d0d636fa4eec4fed"}]},{"name":"kafka-zookeeper-standalone-clusters","children":[{"name":"README.md","path":"examples/kafka-zookeeper-standalone-clusters/README.md","sha":"9780235b75ceb497d3fd299c4794eb0f2601ef15"},{"name":"main.tf","path":"examples/kafka-zookeeper-standalone-clusters/main.tf","sha":"d4a90c87dc0062972e31385dbc792123ed63a6c2"},{"name":"outputs.tf","path":"examples/kafka-zookeeper-standalone-clusters/outputs.tf","sha":"051d6e5ddda7cef04fd9f5031a57694124762eca"},{"name":"user-data","children":[{"name":"kafka-user-data.sh","path":"examples/kafka-zookeeper-standalone-clusters/user-data/kafka-user-data.sh","sha":"ccb4d9324cabce27d4617c7e32ea10a10014385a"},{"name":"zookeeper-user-data.sh","path":"examples/kafka-zookeeper-standalone-clusters/user-data/zookeeper-user-data.sh","sha":"2e144c5dc55c4e721ecf595ae60df4553024191f"}]},{"name":"vars.tf","path":"examples/kafka-zookeeper-standalone-clusters/vars.tf","sha":"fc70b12e7d2c784e3a491ad08d10005f44ed71a7"}]},{"name":"zookeeper-ami","children":[{"name":"README.md","path":"examples/zookeeper-ami/README.md","sha":"6ebd1619152754561ae480c0ec00945ff7d2df43"},{"name":"configure-zookeeper-server.sh","path":"examples/zookeeper-ami/configure-zookeeper-server.sh","sha":"4faecc54c6dae27a79256f8346b1195de0ebb74b"},{"name":"zookeeper.json","path":"examples/zookeeper-ami/zookeeper.json","sha":"cbe0d0d35c0353dd3d7d6203a8e678e3c9def2c5"}]}],"toggled":true},{"name":"modules","children":[{"name":"bash-commons","children":[{"name":"README.md","path":"modules/bash-commons/README.md","sha":"0b7b7bf23db870999e14ee833d05488ba44a136c"},{"name":"install.sh","path":"modules/bash-commons/install.sh","sha":"bedb09f6eaa00a323ae1dd814afec954ca3efeeb"},{"name":"lib","children":[{"name":"array.sh","path":"modules/bash-commons/lib/array.sh","sha":"2d4e0ef22dc608392e99522e8ff0eb68ed1f708c"},{"name":"assert.sh","path":"modules/bash-commons/lib/assert.sh","sha":"bfaf1740050694ed05d03bcff7dbc99724c4fc43"},{"name":"aws.sh","path":"modules/bash-commons/lib/aws.sh","sha":"e6986c813e1fef28dfd5881b0193c4925e8dc66b"},{"name":"file.sh","path":"modules/bash-commons/lib/file.sh","sha":"196b04006ff622844d6d198f78130b2d2fd7c0c6"},{"name":"java.sh","path":"modules/bash-commons/lib/java.sh","sha":"3cc8614fd91c2d9e0816555e558ee12ba0a8c95b"},{"name":"log.sh","path":"modules/bash-commons/lib/log.sh","sha":"1b5887a63f9e7de613707866753e2cfe910da4d1"},{"name":"os.sh","path":"modules/bash-commons/lib/os.sh","sha":"3371306dc7959874cf6b8935d9454e5c1c942c4d"},{"name":"strings.sh","path":"modules/bash-commons/lib/strings.sh","sha":"67a96995df1886ff0d4ce528b5fb26cfbe7b044d"}]}]},{"name":"confluent-tools-cluster","children":[{"name":"README.md","path":"modules/confluent-tools-cluster/README.md","sha":"658308333cc07a35f46a8ec04ff882c74db52bfe"},{"name":"main.tf","path":"modules/confluent-tools-cluster/main.tf","sha":"ac1bdaf4f075ba6ddfbf5d7462564d8d7f9a025a"},{"name":"outputs.tf","path":"modules/confluent-tools-cluster/outputs.tf","sha":"12f380d0aff454ad6d556c116d84c344a62bb999"},{"name":"vars.tf","path":"modules/confluent-tools-cluster/vars.tf","sha":"8234819f003e0566e96cb3a5fa14f71e6020139d"}]},{"name":"confluent-tools-iam-permissions","children":[{"name":"README.md","path":"modules/confluent-tools-iam-permissions/README.md","sha":"06057008bc666088f319da89baa9d16b68fdce30"},{"name":"main.tf","path":"modules/confluent-tools-iam-permissions/main.tf","sha":"b9d177f5ae0a29b9b57f79452b5093253e279d75"},{"name":"vars.tf","path":"modules/confluent-tools-iam-permissions/vars.tf","sha":"9803fc14dc414f11fc9f236685dbb0db8f5273e6"}]},{"name":"confluent-tools-security-group-rules","children":[{"name":"README.md","path":"modules/confluent-tools-security-group-rules/README.md","sha":"5ee75e53e2ba7c2893e147fe86a2ba48fc8feab0"},{"name":"main.tf","path":"modules/confluent-tools-security-group-rules/main.tf","sha":"552b6ef4bfa9da912df5a834b659c268db65e998"},{"name":"vars.tf","path":"modules/confluent-tools-security-group-rules/vars.tf","sha":"3faf1ad90e4c918aefbec1545235ecf97ecc08cf"}]},{"name":"generate-key-stores","children":[{"name":"README.md","path":"modules/generate-key-stores/README.md","sha":"e4cacd66c8857a552810dc5a8d328a5395659392"},{"name":"generate-key-stores.sh","path":"modules/generate-key-stores/generate-key-stores.sh","sha":"b2076744af89591375d36974d4c3a84ba25f32bb"},{"name":"install.sh","path":"modules/generate-key-stores/install.sh","sha":"33c6e02e94425b4d58050ca7719e70c0292e1ba6"}]},{"name":"install-confluent-tools","children":[{"name":"README.md","path":"modules/install-confluent-tools/README.md","sha":"ca24e1b76b9a0299b4c36e07bff2941e6960e0c1"},{"name":"install.sh","path":"modules/install-confluent-tools/install.sh","sha":"2a0b67e853dfa3f9e6fa760f05fea02a3abe9f87"},{"name":"security","children":[{"name":"confluent.key","path":"modules/install-confluent-tools/security/confluent.key","sha":"1025a2c6dfa66f224c0c45ac172fd8d3efce1744"}]}]},{"name":"install-kafka","children":[{"name":"README.md","path":"modules/install-kafka/README.md","sha":"c94e75bda5a0c753d35a67d785af65d46b551e5b"},{"name":"install.sh","path":"modules/install-kafka/install.sh","sha":"4b0c29e5515b4e94ed4f56c1685faebccbd22471"}]},{"name":"kafka-cluster","children":[{"name":"README.md","path":"modules/kafka-cluster/README.md","sha":"a054f23f1a0aebcc5be1f1ec0368bb67f17e20f6"},{"name":"main.tf","path":"modules/kafka-cluster/main.tf","sha":"27d089f3d898bb01ade5125fcd94d8136435864d"},{"name":"outputs.tf","path":"modules/kafka-cluster/outputs.tf","sha":"d53790ae73bc413f3a21acb52064bddf30d0d517"},{"name":"vars.tf","path":"modules/kafka-cluster/vars.tf","sha":"9bf8a4f71cf0c382192eb5a7fee70fbe333a8f7e"}]},{"name":"kafka-iam-permissions","children":[{"name":"README.md","path":"modules/kafka-iam-permissions/README.md","sha":"3763dd58a4cd52e71b5ffe0d3e5e9fe0cf48053f"},{"name":"main.tf","path":"modules/kafka-iam-permissions/main.tf","sha":"ecfc7111106491e318ca9eed66d258f3b08df09d"},{"name":"vars.tf","path":"modules/kafka-iam-permissions/vars.tf","sha":"d29e8e5a07834701c3050e6369607747f85e43d8"}]},{"name":"kafka-security-group-rules","children":[{"name":"README.md","path":"modules/kafka-security-group-rules/README.md","sha":"63627f3af8842e4af90d0c31337004ef2503fc9c"},{"name":"main.tf","path":"modules/kafka-security-group-rules/main.tf","sha":"a5946f1e5cf4dfa784be01c332c058672ce2fbfe"},{"name":"vars.tf","path":"modules/kafka-security-group-rules/vars.tf","sha":"9c27d84eb7bfd6291acafc3e844f0ab7e1ec970f"}]},{"name":"run-health-checker","children":[{"name":"README.md","path":"modules/run-health-checker/README.md","sha":"8bc3cab46b5eecb25e642364dbd57ea35a0be71c"},{"name":"bin","children":[{"name":"run-health-checker","path":"modules/run-health-checker/bin/run-health-checker","sha":"ee3357d5fcc32957115b32538c00702c20b36a97"}]},{"name":"install.sh","path":"modules/run-health-checker/install.sh","sha":"af927c79f7df2b1d57204e1207a4104cbf0f63a2"}]},{"name":"run-kafka-connect","children":[{"name":"README.md","path":"modules/run-kafka-connect/README.md","sha":"0f180c7e494588f218cfe2ace99c671c8dfcadcf"},{"name":"bin","children":[{"name":"run-kafka-connect","path":"modules/run-kafka-connect/bin/run-kafka-connect","sha":"fb4145cc36ab2f8efc39ad7fc94444fa866ccef1"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka-connect/config/README.md","sha":"5c6f2b6e63f1eba41957dd083c0787ac139c9622"},{"name":"kafka-connect","children":[{"name":"worker-3.3.x.properties","path":"modules/run-kafka-connect/config/kafka-connect/worker-3.3.x.properties","sha":"6bd29bb369f3aaea951e0a00d157ba167e943813"},{"name":"worker-4.0.x.properties","path":"modules/run-kafka-connect/config/kafka-connect/worker-4.0.x.properties","sha":"5b4760ba1b5805ed9ec7eb0f0be06a3597f15001"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka-connect/config/log4j/log4j.properties","sha":"a23dfdbf369c5a8cba498d9016ab239f3c1c18a8"}]}]},{"name":"install.sh","path":"modules/run-kafka-connect/install.sh","sha":"ba03f49f90a63a0e683c7be1b016b14db41bd71c"},{"name":"security","children":[{"name":"README.md","path":"modules/run-kafka-connect/security/README.md","sha":"5242a8435552c50055c926e1ec704545ca2c1b24"},{"name":"confluent-3.3.1-2.11.tar.gz.checksum","path":"modules/run-kafka-connect/security/confluent-3.3.1-2.11.tar.gz.checksum","sha":"c7aed490972e7b1565795221488d18449bd0bae1"},{"name":"confluent-4.0.0-2.11.tar.gz.checksum","path":"modules/run-kafka-connect/security/confluent-4.0.0-2.11.tar.gz.checksum","sha":"27b7a13f188475b4157386aa2761915633b72aa3"}]}]},{"name":"run-kafka-rest","children":[{"name":"README.md","path":"modules/run-kafka-rest/README.md","sha":"93c501f797818e3fee1e078cfbb93cc35577cc91"},{"name":"bin","children":[{"name":"run-kafka-rest","path":"modules/run-kafka-rest/bin/run-kafka-rest","sha":"2d1ff513fef3134b9c066f1e80ef04986ec6d5a2"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka-rest/config/README.md","sha":"772b1a95a54aadd3c0f0dce34eac10f3d1967634"},{"name":"kafka-rest","children":[{"name":"kafka-rest-3.3.x.properties","path":"modules/run-kafka-rest/config/kafka-rest/kafka-rest-3.3.x.properties","sha":"21263ea344efbfa26e1a792b8bb29c33606c0d7a"},{"name":"kafka-rest-4.0.x.properties","path":"modules/run-kafka-rest/config/kafka-rest/kafka-rest-4.0.x.properties","sha":"29c9ca3bd784637597c683b1585bfd52fd0035db"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka-rest/config/log4j/log4j.properties","sha":"43c18e3a2eb5bdf7a49c0336919aac1acf5f6b6d"}]}]},{"name":"install.sh","path":"modules/run-kafka-rest/install.sh","sha":"97c543dd2a175ea3866f92e835494723645c6edc"}]},{"name":"run-kafka","children":[{"name":"README.md","path":"modules/run-kafka/README.md","sha":"b1fcb424860e462141f9a17423b0c3ecc3f2de08"},{"name":"bin","children":[{"name":"run-kafka","path":"modules/run-kafka/bin/run-kafka","sha":"2301427f95fbfb42ac5be080993d5cdd8699e172"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-kafka/config/README.md","sha":"e08702423b137b254ad0a0a070f7300f094ba046"},{"name":"kafka","children":[{"name":"server-3.3.x.properties","path":"modules/run-kafka/config/kafka/server-3.3.x.properties","sha":"b5aa3757d41f0e6fef81799b58c222774a0da63d"},{"name":"server-4.0.x.properties","path":"modules/run-kafka/config/kafka/server-4.0.x.properties","sha":"5ea1bae91e95a50333444d0ffffc94924cfd0483"}]},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-kafka/config/log4j/log4j.properties","sha":"394c539d46d5922b33ba1e8b3a50db2fbed7e6ef"}]}]},{"name":"install.sh","path":"modules/run-kafka/install.sh","sha":"d538e667d2a66004ceacc149b1d8c2f54e639ec7"}]},{"name":"run-schema-registry","children":[{"name":"README.md","path":"modules/run-schema-registry/README.md","sha":"343ddd7b1e9b24054c05722b3cb6ea4e63d88419"},{"name":"bin","children":[{"name":"run-schema-registry","path":"modules/run-schema-registry/bin/run-schema-registry","sha":"07a457889d54a9385de0bcc4d4d5dc51b3636b19"}]},{"name":"config","children":[{"name":"README.md","path":"modules/run-schema-registry/config/README.md","sha":"93a30f8adf3f778682463bbc8715a38f70908866"},{"name":"log4j","children":[{"name":"log4j.properties","path":"modules/run-schema-registry/config/log4j/log4j.properties","sha":"28fa60645b6ba0ab402433aebbedec8a8a9533e3"}]},{"name":"schema-registry","children":[{"name":"schema-registry.properties","path":"modules/run-schema-registry/config/schema-registry/schema-registry.properties","sha":"e6541005171b9f0de27e7f177f915b08399f9404"}]}]},{"name":"install.sh","path":"modules/run-schema-registry/install.sh","sha":"81b4a8b7c8b26d6b4a60e524bcda30e95a2777b7"}]}]},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"cf72a9f58a3b36aee8053b37dfc6e1a617f93f7b"},{"name":"generate_key_stores_test.go","path":"test/generate_key_stores_test.go","sha":"fceb03a8da4583eff59b8c4da1c070b9ef0c980a"},{"name":"go.mod","path":"test/go.mod","sha":"73214126337cfa12cfd00919689fb51ea7493b3a"},{"name":"go.sum","path":"test/go.sum","sha":"93a629ddbce948e976f8b110943947df31c0975e"},{"name":"kafka_zookeeper_confluent_colocated_cluster_test.go","path":"test/kafka_zookeeper_confluent_colocated_cluster_test.go","sha":"d5ac28bbeef65dfaf784f7339e516510c9c22ba4"},{"name":"kafka_zookeeper_confluent_standalone_clusters_test.go","path":"test/kafka_zookeeper_confluent_standalone_clusters_test.go","sha":"55445cf91cd5534aa0fdb148b96bd5d1013b4915"},{"name":"kafka_zookeeper_standalone_clusters_test.go","path":"test/kafka_zookeeper_standalone_clusters_test.go","sha":"55abe10b05b0cba29f54df6f83b02d849fe19c83"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"f01924f4c18c7595d58621e5e3dcc2128996f44e"},{"name":"test_helpers_kafka.go","path":"test/test_helpers_kafka.go","sha":"474caf79574b27637d376fe5ab65abea9af7eb3c"},{"name":"test_helpers_kafka_connect.go","path":"test/test_helpers_kafka_connect.go","sha":"1ecef92e9a45501fb4bf834579d02ddcc05e7103"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"02d88327f4021955dca74c5311eb916bed5c7afa"},{"name":"test_helpers_rest_proxy.go","path":"test/test_helpers_rest_proxy.go","sha":"39a5c7d2f96a873615c856cd56861ad5e7920e1c"},{"name":"test_helpers_schema_registry.go","path":"test/test_helpers_schema_registry.go","sha":"b15a39916cd41441705265ccabeb23e0460b260f"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"kafka-zookeeper-kafka-connect-schema-registry-rest-proxy-example-ami\">Kafka/Zookeeper/Kafka Connect/Schema Registry/REST Proxy Example AMI</h1><div class=\"preview__body--border\"></div><p>This folder contains an example <a href=\"https://www.packer.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Packer</a> template that can be used to create an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon\nMachine Image (AMI)</a> with Kafka, ZooKeeper, Kafka Connect,\nSchema Registry, REST Proxy, and all of their dependencies installed. A few notes about this Packer template:</p>\n<ul>\n<li>\n<p>It builds two AMIs, one on top of Amazon Linux and one on top of Ubuntu. Pick which ever distro you prefer.</p>\n</li>\n<li>\n<p>This AMI includes Kafka, ZooKeeper, Kafka Connect, Schema Registry and REST Proxy with the expectation that you will\nrun all of these co-located in the same cluster. The ZooKeeper dependencies come from <a href=\"/repos/terraform-aws-zookeeper\" class=\"preview__body--description--blue\">terraform-aws-zookeeper</a>. All other dependencies live in this repo.</p>\n</li>\n<li>\n<p>A colocated cluster is useful for deploying a less expensinve non-production environment. For a production-ready example\nthat configures these services to run in three separate clusters, see the <a href=\"/repos/v0.11.0/package-kafka/examples/kafka-ami\" class=\"preview__body--description--blue\">kafka-ami</a>\n, <a href=\"/repos/v0.11.0/package-kafka/examples/zookeeper-ami\" class=\"preview__body--description--blue\">zookeeper-ami</a>, and <a href=\"/repos/v0.11.0/package-kafka/examples/confluent-oss-ami\" class=\"preview__body--description--blue\">confluent-oss-ami</a> examples.</p>\n</li>\n<li>\n<p>This AMI example does not include SSL support.</p>\n<ul>\n<li>If you want your Kafka brokers and Kafka Connect Workers to communicate over SSL, see the <a href=\"/repos/v0.11.0/package-kafka/examples/kafka-ami\" class=\"preview__body--description--blue\">kafka-ami</a> example.</li>\n<li>If you want your Schema Registry and REST Proxy endpoints to communicate over SSL, see the <a href=\"/repos/v0.11.0/package-kafka/examples/confluent-oss-ami\" class=\"preview__body--description--blue\">confluent-oss-ami</a> example.</li>\n</ul>\n</li>\n</ul>\n<p>To deploy this AMI, see the <a href=\"/repos/v0.11.0/package-kafka/examples/kafka-zookeeper-confluent-oss-colocated-cluster\" class=\"preview__body--description--blue\">kafka-zookeeper-confluent-oss-colocated-cluster example</a>.</p>\n<h2 class=\"preview__body--subtitle\" id=\"quick-start\">Quick start</h2>\n<p>To build the AMI:</p>\n<ol>\n<li>Install <a href=\"https://www.packer.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Packer</a>.</li>\n<li>Set up your <a href=\"https://www.packer.io/docs/builders/amazon.html\" class=\"preview__body--description--blue\" target=\"_blank\">AWS credentials as environment variables</a>.</li>\n<li>Set the <code>GITHUB_OAUTH_TOKEN</code> environment variable to a valid GitHub auth token with "repo" access. You can generate\none here: https://github.com/settings/tokens</li>\n<li>Run <code>packer build -var "package_kafka_branch=<branch>" -only <target os> kafka-zookeeper-confluent-oss.json</code> to create a new AMI in your AWS account. Note down the ID of this new AMI.</li>\n</ol>\n<p>To deploy the new AMI, plug its ID into the <a href=\"/repos/v0.11.0/package-kafka/examples/kafka-zookeeper-confluent-oss-colocated-cluster\" class=\"preview__body--description--blue\">kafka-zookeeper-confluent-oss-colocated-cluster example</a>.</p>\n","repoName":"package-kafka","repoRef":"v0.8.0","serviceDescriptor":{"serviceName":"Apache Kafka and Confluent Tools","serviceRepoName":"package-kafka","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a cluster of Kafka brokers. Optionally deploy Confluent tools such as Schema Registry, REST Proxy, and Kafka Connect.","imageUrl":"kafka.png","licenseType":"subscriber","technologies":["Terraform","Bash"],"compliance":[],"tags":[""]},"serviceCategoryName":"Messaging & streaming","fileName":"README.md","filePath":"/examples/kafka-zookeeper-confluent-oss-ami","title":"Repo Browser: Apache Kafka and Confluent Tools","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
configure-kafka-zk-confluent-server.sh
