core-concepts.md

Browse the Repo

.circleci

.github

.patcher

_ci

_docs

codegen

generate-aws-config

generate-aws-guardduty

generate-ebs-encryption

generate-multiregion-iam-access-analyzer

generate-multiregion-kms-grant

generate-multiregion-kms

generator

logging

README.adoc

core-concepts.md

generate-all.sh

go.mod

go.sum

examples

modules

test

.editorconfig

.gitignore

.pre-commit-config.yaml

CODEOWNERS

LICENSE.txt

README.adoc

terraform-cloud-enterprise-private-module-...

Browse the Repo

.circleci

.github

.patcher

_ci

_docs

codegen

generate-aws-config

generate-aws-guardduty

generate-ebs-encryption

generate-multiregion-iam-access-analyzer

generate-multiregion-kms-grant

generate-multiregion-kms

generator

logging

README.adoc

core-concepts.md

generate-all.sh

go.mod

go.sum

examples

modules

test

.editorconfig

.gitignore

.pre-commit-config.yaml

CODEOWNERS

LICENSE.txt

README.adoc

terraform-cloud-enterprise-private-module-...

ssh-grunt

Manage SSH access to EC2 Instances using groups in AWS IAM or your Identity Provider (e.g., ADFS, Google, Okta, etc).

Generating modules for multiple AWS regions

How to use a multi-region module

To use a multi-region module, such as aws-config-multi-region, you must perform the following steps:

Instantiate a provider block for each AWS region. In your Terraform or Terragrunt code, in the "root" module (i.e., the one on which you run apply), you MUST instantiate one provider block for each AWS region. Don't worry if you're not using all the regions or some are not enabled in your account: we'll configure opt_in_regions shortly to only use the regions you want. However, you still have to create a provider block for every region, whether you're using it or not. The easiest way to do this is to copy/paste one of the following:
1. If you use Terragrunt, copy/paste these locals and generate blocks into your terragrunt.hcl.
2. If you use pure Terraform, copy/paste this providers.tf into your module.

Pass in a providers map. You must pass a providers = { ... } map to the multi-region module that contains all of your provider blocks, as shown in the example below. Note that this is only necessary if you're using pure Terraform; Terragrunt users can skip this step.

module "aws_config" {
  source = "git::git@github.com:gruntwork-io/terraform-aws-security.git//modules/aws-config-multi-region?ref=v0.50.0"

  # You MUST create a provider block for EVERY AWS region (see providers.tf) and pass all those providers in here via
  # this providers map. However, you should use var.opt_in_regions to tell Terraform to only use and authenticate to
  # regions that are enabled in your AWS account.
  providers = {
    aws.af_south_1     = aws.af_south_1
    aws.ap_east_1      = aws.ap_east_1
    aws.ap_northeast_1 = aws.ap_northeast_1
    aws.ap_northeast_2 = aws.ap_northeast_2
    aws.ap_northeast_3 = aws.ap_northeast_3
    aws.ap_south_1     = aws.ap_south_1
    aws.ap_southeast_1 = aws.ap_southeast_1
    aws.ap_southeast_2 = aws.ap_southeast_2
    aws.ap_southeast_3 = aws.ap_southeast_3
    aws.ca_central_1   = aws.ca_central_1
    aws.cn_north_1     = aws.cn_north_1
    aws.cn_northwest_1 = aws.cn_northwest_1
    aws.eu_central_1   = aws.eu_central_1
    aws.eu_north_1     = aws.eu_north_1
    aws.eu_south_1     = aws.eu_south_1
    aws.eu_west_1      = aws.eu_west_1
    aws.eu_west_2      = aws.eu_west_2
    aws.eu_west_3      = aws.eu_west_3
    aws.me_south_1     = aws.me_south_1
    aws.sa_east_1      = aws.sa_east_1
    aws.us_east_1      = aws.us_east_1
    aws.us_east_2      = aws.us_east_2
    aws.us_gov_east_1  = aws.us_gov_east_1
    aws.us_gov_west_1  = aws.us_gov_west_1
    aws.us_west_1      = aws.us_west_1
    aws.us_west_2      = aws.us_west_2
  }

  # ... (other params omitted) ...

}

Configure the regions to use via opt_in_regions. You MUST set the opt_in_regions variable to the list of regions you wish to use in your AWS account. This will typically be all the enabled regions in your account. You cannot leave the variable as null or empty. To get the list of regions enabled in your AWS account, you can use the AWS CLI: aws ec2 describe-regions.

variable "opt_in_regions" {
  description = "Creates resources in the specified regions. The best practice is to enable AWS Config in all enabled regions in your AWS account. This variable must NOT be set to null or empty. Otherwise, we won't know which regions to use and authenticate to, and may use some not enabled in your AWS account (e.g., GovCloud, China, etc).  To get the list of regions enabled in your AWS account, you can use the AWS CLI: aws ec2 describe-regions. The value provided for global_recorder_region must be in this list."
  type        = list(string)
  default = [
    "eu-north-1",
    "ap-south-1",
    "eu-west-3",
    "eu-west-2",
    "eu-west-1",
    "ap-northeast-2",
    "ap-northeast-1",
    "sa-east-1",
    "ca-central-1",
    "ap-southeast-1",
    "ap-southeast-2",
    "eu-central-1",
    "us-east-1",
    "us-east-2",
    "us-west-1",
    "us-west-2",

    # By default, skip regions that are not enabled in most AWS accounts:
    #
    #  "af-south-1",     # Cape Town
    #  "ap-east-1",      # Hong Kong
    #  "eu-south-1",     # Milan
    #  "me-south-1",     # Bahrain
    #  "us-gov-east-1",  # GovCloud
    #  "us-gov-west-1",  # GovCloud
    #  "cn-north-1",     # China
    #  "cn-northwest-1", # China
    #
    # This region is enabled by default but is brand-new and some services like AWS Config don't work.
    # "ap-northeast-3", # Asia Pacific (Osaka)
  ]
}

Set other input variables as usual. Browse variables.tf for the module and configure it to fit your use case.
Run plan and apply as usual. Run the standard Terraform commands you use to deploy any other module.

Default provider

Some of the multi-region modules require a provider that is aliased as default. This is the provider configuration for resources that need to be deployed into a single region. This provider is also used for detecting the AWS Cloud partition to use when computing ARNs. Note that even though it's the default, we explicitly expect a named provider ensuring that users will explicitly configure the provider to exactly what is needed, rather than having an implicit one get used accidentally.

Installation

Each generator is a single, self-contained, statically compiled binary written in Go. The easiest way to get it onto your servers is to use the Gruntwork Installer. For example, to install generate-aws-config (make sure to replace <VERSION> below with the latest version from the releases page):

gruntwork-install --binary-name generate-aws-config --repo https://github.com/gruntwork-io/terraform-aws-security --tag <VERSION>

Alternatively, you can download the binary from the Releases Page.

Usage

The generator command should be run in your infrastructure-modules repository, or any repository where you store your Terraform modules. This command will output a Terraform module that you can then deploy with various input parameters to enable the single region module on all regions.

For example, generate-aws-config will generate a Terraform module named aws-config-multi-region which exposes input parameters to adjust various options exposed in the single region module. Suppose you had an infrastructure-modules repository that mimics the Gruntwork Reference Architecture, and has the following structure:

infrastructure-modules
├── README.md
├── data-stores
│   └── rds
│       ├── README.md
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── security
    ├── iam-groups
    │   ├── README.md
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    └── cloudtrail
        ├── README.md
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

Suppose that you now want to add the module aws-config-multi-region as generated by this script in the security folder. To do so, first install the binary so that it is available in your PATH as described in the Installation section of this docs.

Next, run the generator in the infrastructure-modules directory, passing in the target directory and main region. Note that you will need to be authenticated to your AWS account. Refer to the Comprehensive Guide to Authenticating to AWS on the Command Line for recommended ways to authenticate on the command line.

Note that one region must be marked as the one to store the global recorder, which includes resources that don't have regions in AWS such as IAM.

generate-aws-config --target-directory ./security/aws-config --global-recorder-region us-east-1

This will:

Look up all the available regions for the authenticated account.
Generate a terraform module in the folder security/aws-config that will deploy AWS Config on each enabled region.
Set the us-east-1 as the one to store the global recorder.

At the end of this command, you should see the aws-config module generated in your infrastructure-modules repository:

infrastructure-modules
├── README.md
├── data-stores
│   └── rds
│       ├── README.md
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── security
    ├── aws-config
    │   ├── README.md
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    ├── iam-groups
    │   ├── README.md
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    └── cloudtrail
        ├── README.md
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

You can now invoke the module using Terragrunt, or any other production Terraform frontend that you are currently using.

Building the Binary

We use packr to compile the templates into the binary so that it is portable. To do so, we need to run packr2 prior to building the binaries.

Here are the steps for compiling from source:

Install the packr2 binary: go get -u github.com/gobuffalo/packr/v2/packr2
Run $GOPATH/bin/packr2. This will convert the template files into go files so that they are available in the go binary.
Build the generate-aws-config binary: go build -o $BIN_PATH .

Contributions

For generic guidance on contributions, please refer to contributions guidelines for this repository.

Adding a new module

Adding a new module to this repo is a task that requires deep understanding of the module you're about to create. Below are listed a few steps and suggestions on how to add a new module.

The list is not exclusive and aims to give a starting point for contributors, rather than step-by-step instructions.

Add a new module folder for your module - e.g. modules/new-module.
Add the necessary code to create the desired AWS resources.
For multi-region modules:
1. Add a new folder for your module under the codegen directory.
2. Add the necessary code (it might be best to follow as an example one of the other modules in the codegen directory).
  - for documentation, follow the pattern of having a README.adoc and core-concepts.md in the static folder. This is following this RFC.
3. Update the codegen/core-concepts.md file to contain a reference to the new module.
4. Run generate-all.sh to generate a multi-region version of the new module - that will be generated in the modules directory of the repo.
5. Confirm all is working well by trying to run terraform apply on your generated multi-region module.
Add examples demosntrating using the new module
Add tests
Add necessary permissions to any CI users as needed.

Alternatives considered

Native Terraform

As of 0.12.16, Terraform still does not support:

The combination of these limitations makes it impossible to natively write Terraform code that invokes the aws-config module across all regions.

Manually generating the code

If this tool didn't exist, users will have to manually create the module with each of the enabled regions replicated. This involves a lot of copy paste and with 10+ regions, it becomes prone to operator error. Additionally, there are several manual steps required:

The user will have to use the aws CLI to look up all enabled regions.
For each enabled region, the user will have to add in the provider config and module call.
One of the regions need to be designated the global recorder and have the input is_global_recorder = true.

This is especially painful if the module needs to be updated, as all references need to be replaced. When the config is almost exactly the same (except for the region and the global recorder), this can be unnecessarily painful to replicate the changes in each block.

Generating Terragrunt live config

Instead of generating a single module that manages all the configs, we could also have generated Terragrunt live config files for each region in the Terragrunt folder structure. Ultimately we decided to generate the module calls in Terraform for the following reasons:

We don't want to be opinionated to Terragrunt and support a wide range of possible use cases including Terraform Enterprise.
Removing a region requires running terragrunt destroy. If you forget to run terragrunt destroy on the removed region, then you may never remove that Config until you manually remove it from the console. In contrast, the current approach will eventually ensure the Config is removed when terraform apply is run after the code for the region is removed.

Additionally, replicating the Terraform module blocks in a single module has a natural progression to migrate to using for_each on modules once that is ready to be implemented.

Supporting iteration in Terragrunt

This ended up being too complex to justify supporting the feature. See the RFC for Terragrunt iteration for more information.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?

Buy Now

"https://cdn.gruntwork.io/gruntwork-website/"

{"index":{"js":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.c7884255553b53fbca3a.js","map":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.1b14c1b7d19f1f5eb35d6e118e838255.map"},"styles":{"css":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.f22938926651ddec7c49.css","js":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.e782420e74a20dcb8691.js","map":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.d5e2af49807c6ca33f8367d621ece507.map"},"vendors":{"css":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.29f7d0366a0978763f96.css","js":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.fa8174a130797d75d12c.js","map":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.57243d94deeeb29d5061288a338b4eb6.map"}}

ssh-grunt

Code Preview

Preview the Code

Generating modules for multiple AWS regions

How to use a multi-region module

Default provider

Installation

Usage

Building the Binary

Contributions

Adding a new module

Alternatives considered

Native Terraform

Manually generating the code

Generating Terragrunt live config

Supporting iteration in Terragrunt

Questions? Ask away.

Ready to hand off the Gruntwork?