Browse the Repo

file-type-icon.circleci
file-type-iconexamples
file-type-iconmodules
file-type-iconbash-commons
file-type-iconconfluent-tools-cluster
file-type-iconconfluent-tools-iam-permissions
file-type-iconconfluent-tools-security-group-rules
file-type-icongenerate-key-stores
file-type-iconREADME.md
file-type-icongenerate-key-stores.sh
file-type-iconinstall.sh
file-type-iconinstall-confluent-tools
file-type-iconinstall-kafka
file-type-iconkafka-cluster
file-type-iconkafka-iam-permissions
file-type-iconkafka-security-group-rules
file-type-iconrun-health-checker
file-type-iconrun-kafka-connect
file-type-iconrun-kafka-rest
file-type-iconrun-kafka
file-type-iconrun-schema-registry
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconLICENSE.txt
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...

Browse the Repo

file-type-icon.circleci
file-type-iconexamples
file-type-iconmodules
file-type-iconbash-commons
file-type-iconconfluent-tools-cluster
file-type-iconconfluent-tools-iam-permissions
file-type-iconconfluent-tools-security-group-rules
file-type-icongenerate-key-stores
file-type-iconREADME.md
file-type-icongenerate-key-stores.sh
file-type-iconinstall.sh
file-type-iconinstall-confluent-tools
file-type-iconinstall-kafka
file-type-iconkafka-cluster
file-type-iconkafka-iam-permissions
file-type-iconkafka-security-group-rules
file-type-iconrun-health-checker
file-type-iconrun-kafka-connect
file-type-iconrun-kafka-rest
file-type-iconrun-kafka
file-type-iconrun-schema-registry
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconLICENSE.txt
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...
Apache Kafka and Confluent Tools

Apache Kafka and Confluent Tools

Deploy a cluster of Kafka brokers. Optionally deploy Confluent tools such as Schema Registry, REST Proxy, and Kafka Connect.

Code Preview

Preview the Code

mobile file icon

README.md

down

Generate Key Stores

This folder contains a script that you can use to generate:

  • A Java Key Store: A repository used to store security certificates the current server should use to identify itself.
  • A Java Trust Store: A Key Store used to store security certificates the current server should trust.
  • A self-signed SSL certificate stored in the Key Store.

These three items can be used to configure Kafka brokers to use SSL for communication.

Install

You can install the generate-key-stores.sh script using the Gruntwork Installer:

gruntwork-install --module-name "generate-key-stores" --repo "https://github.com/gruntwork-io/terraform-aws-kafka" --tag "0.0.1"

Quick start

Here is how you run the script:

export KEY_STORE_PASSWORD=(password to use for the Key Store)
export TRUST_STORE_PASSWORD=(password to use for the Trust Store)

generate-key-stores.sh \\
  --key-store-path kafka.server.keystore.jks \\
  --trust-store-path kafka.server.truststore.jks \\
  --cert-path cert \\
  --ca-path ca-cert \\
  --org Gruntwork \\
  --org-unit Engineering \\
  --city Phoenix \\
  --state Arizona \\
  --country US

This will generate four files:

  • kafka.server.keystore.jks: This is the Key Store. It will be protected with the password you specified in KEY_STORE_PASSWORD. It has a self-signed SSL certificate stored inside of it.

  • kafka.server.truststore.jks: This is the Trust Store. It will be protected with the password you specified in TRUST_STORE_PASSWORD.

  • ca-cert: This is the public key of the the CA certificate. You won't need this for use with Kafka itself, but you may need it to connect to Kafka from a non-Java client (i.e., a client that doesn't use a Trust Store).

  • cert: This is the public key of the SSL certificate stored in the Key Store. You won't need htis for use with Kafka itself.

Now that you have these files, here's how you use them:

  1. Pass the paths to the Key Store and Trust Store to the install-kafka script using the --key-store-path and --trust-store-path arguments, respectively. See kafka-ami for an example.

  2. When running Kafka with the run-kafka script, set --enable-ssl=true and provide your Key Store and Trust Store passwords using the --key-store-password and --trust-store-password arguments, respectively. See kafka-user-data.sh for an example.

  3. You will also need to provide the Trust Store file (but NOT the Key Store file) to each of your Kafka clients (the producers and consumers) as documented here.

How the script works

This script implements the steps for generating a Key Store and Trust Store as described in the Kafka Encryption and Authentication using SSL documentation. Under the hood, we are using keytool to create the Key Store and Trust Store and openssl to sign the certificate.

A key point about this script: we generate a CA to sign the SSL certificate and then delete the CA private key. This ensures that no one will be able to steal the CA key and sign fake certificates with it. However, it also means that you cannot sign any more certificates with the same CA key. If you generate other certificates in the future, you will have to create new CAs and add those CAs to your Trust Store.

If you're new to SSL, make sure to read our TLS/SSL background documentation as a primer.

What if I want my cert to validate IP addresses and domain names?

If you have just one domain name, using the common-name (CN) field is the way to go. If you have multiple domain names, or a domain name and some IPs, using SAN (Subject Alternative Name) is probably the way to go.

Fortunately the generate-key-stores has you covered.

Generating a self-signed certificate with domains or ip's in the SAN field

export KEY_STORE_PASSWORD=(password to use for the Key Store)
export TRUST_STORE_PASSWORD=(password to use for the Trust Store)

generate-key-stores.sh \\
  --key-store-path kafka.server.keystore.jks \\
  --trust-store-path kafka.server.truststore.jks \\
  --cert-path cert \\
  --ca-path ca-cert \\
  --org Gruntwork \\
  --org-unit Engineering \\
  --city Phoenix \\
  --state Arizona \\
  --country US \\
  --domain myexample-domain.com
  --domain another-valid-domain.io
  --ip 127.0.0.1
  --ip 192.168.2.23

Notes:

  • You can specify multiple --domain and --ip arguments
  • The presence of either --domain or --ip arguments will automatically create a certificate with that entry in the SAN field.
  • If you do use --domain or --ip arguments then the CN field should NOT include a domain name as the CN field will not be examined if a domain or IP is specified. For more info see the exact rules here

Additional Info:

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?