mobile file icon

README.md

close sidebar menu icon

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconaccounts
file-type-iconcommands
file-type-icondns
file-type-iconformat
file-type-icongcloud
file-type-iconoptions
file-type-iconprompt
file-type-iconrepos
file-type-iconretry
file-type-icontls
file-type-iconwizard
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md
file-type-icongo.mod
file-type-icongo.sum
file-type-iconmain.go

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconaccounts
file-type-iconcommands
file-type-icondns
file-type-iconformat
file-type-icongcloud
file-type-iconoptions
file-type-iconprompt
file-type-iconrepos
file-type-iconretry
file-type-icontls
file-type-iconwizard
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md
file-type-icongo.mod
file-type-icongo.sum
file-type-iconmain.go
gruntwork

gruntwork

Code Preview

View on GitHub:

https://github.com/gruntwork-io/gruntwork/tree/v0.1.5

Preview the Code

mobile file icon

README.md

down

Gruntwork

This repo contains a CLI tool to perform Gruntwork tasks. The currently supported functionality includes:

AWS

  1. Creating the AWS accounts and permissions for the Gruntwork Reference Architecture.
  2. Creating the GitHub repos and GitHub teams for the Gruntwork Reference Architecture.
  3. Registering domain names for the Gruntwork Reference Architecture.
  4. Creating TLS certs for the Gruntwork Reference Architecture.

GCP

  1. Creating the GCP Admin Project and GSuite Groups for the Gruntwork Reference Architecture.

Install

  1. Download the latest binary for your OS on the releases page.
  2. Move the binary to a folder on your PATH. E.g.: mv gruntwork_darwin_amd64 /usr/local/bin/gruntwork.
  3. Add execute permissions to the binary. E.g.: chmod u+x /usr/local/bin/gruntwork.
  4. Test it installed correctly: gruntwork --help.

Credentials

In order for the gruntwork CLI tool to access your AWS and Git repos, you will need to provide your credentials:

  1. AWS credentials
  2. GCP credentials
  3. Git credentials

AWS credentials

To make any changes in your AWS account(s), such as creating new accounts, giving Gruntwork access to those accounts, or registering domain names, you will need to provide your AWS credentials. There are two ways to do this:

  1. Use one of the standard AWS CLI credential mechanisms, such as setting environment variables:

    export AWS_ACCESS_KEY_ID="your access key"
export AWS_SECRET_ACCESS_KEY="your secret key"

  2. Allow the gruntwork CLI to prompt you interactively for your credentials. E.g.,:

    > gruntwork aws grant --account prod --managed-policy ReadOnlyAccess

INFO[2017-12-17T22:56:56Z] Checking for AWS credentials...
INFO[2017-12-17T22:57:02Z] Did not find any AWS credentials in the environment. Will prompt for credentials interactively.

Enter your AWS Access Key ID: xxx

Enter your AWS Secret Access Key: yyy

Note that if are using the gruntwork CLI to create new AWS accounts, then:

  1. The CLI uses AWS Organizations to create the new AWS accounts as "children" of a "master" AWS account. So the credentials you provide must be of an admin user in what will become your "master" account.

  2. If you haven't enabled AWS Organizations already for your AWS account, follow these instructions to do so. Make sure to enable "All features" (and not just consolidated billing) for your AWS organization!

GCP credentials

The gruntwork utility will auth to the GCP and GSuite APIs using gcloud. gcloud is Google's official CLI for interacting with GCP. gcloud bakes in routines for getting Oauth credentials, including a login flow through the browser. These credentials are then stored on the file system, and can be used to setup the various SDKs to access the APIs.

During the setup process, the gruntwork utility will start a subprocess that runs gcloud auth application-default login, which will obtain application default credentials. These credentials can be used outside of gcloud, unlike the user credentials obtained in gcloud auth login.

To ensure the credentials are temporary, the CLI will create a temporary directory to store the credentials in. This directory is automatically deleted by the CLI when access to GCP is no longer needed.

This means that you must have gcloud installed in order for the gruntwork utility to be able to auth to GCP. Follow the instructions in the official docs to install it.

Required permissions of authenticated user

The authenticated user must have the following permissions on the GCP org in order to use this utility:

  • Organization Viewer IAM Role
  • Project Creator IAM Role
  • Billing Account User IAM Role

Additionally, you need to be able to create and manage GSuite Groups on the corresponding GSuite Org.

Git credentials

If you use GitHub, you will need to create a GitHub personal access token with the following permissions:

  • repo
  • admin:org

You can then provide your key to the gruntwork CLI by:

  1. Using an environment variable:

    export GITHUB_TOKEN="your github token"

  2. Allowing the gruntwork CLI to prompt you interactively for it: E.g.,:

    > gruntwork github grant --github-org gruntwork-io

INFO[2017-12-17T22:56:56Z] Checking for AWS credentials...
INFO[2017-12-17T22:57:02Z] Did not find any GitHub credentials in the environment. Will prompt for credentials interactively.

Enter your personal access token for GitHub: xxx

If you use a different Git host (e.g., BitBucket or GitLab), you'll have to create BitBucket and GitLab repos manually, as the gruntwork CLI currently only supports GitHub. See Bootstrap the Git repos for instructions.

NOTE: To grant the Gruntwork team access, you will temporarily need at least 7 seats available in your GitHub org. As soon as the deployment is done, you can use the github revoke command below to remove the Gruntwork team and get those seats back!

Bootstrapping the Reference Architecture

AWS

Once you've configured your credentials, there are two main ways to bootstrap the Git repos and AWS accounts for the Gruntwork Reference Architecture:

  1. Bootstrap using the wizard (recommended)
  2. Bootstrap manually

GCP

For GCP, your only option currently is to bootstrap manually. See Bootstrap the GCP Org for the details on how to get setup.

Bootstrap using the wizard (recommended)

The easiest way to bootstrap your GitHub repos and AWS accounts for the Gruntwork Reference Architecture is to run the wizard command:

gruntwork wizard

This is an interactive wizard that will take you through the process step by step, including:

  1. Bootstrapping your AWS accounts
  2. Bootstrapping your Git repos
  3. Bootstrapping your domain names (only AWS)
  4. Bootstrapping your TLS certs (only AWS)

Bootstrap manually

If you want more control over the bootstrap process, you can run a separate gruntwork command for each step:

  1. Bootstrap the AWS accounts
  2. Bootstrap the GCP Org
  3. Bootstrap the Git repos
  4. Bootstrap the domain names (only AWS)
  5. Bootstrap the TLS certs (only AWS)

Bootstrap the AWS accounts

We deploy the Reference Architecture into new AWS accounts to ensure we cannot have any effect on any infrastructure you already have running. If you bought the Standard Reference Architecture, everything will be deployed into a single new AWS account. If you bought the Multi-Account add-on for the Reference Architecture, each environment will be deployed into a separate AWS account.

To bootstrap your AWS accounts, you need to:

  1. Create the AWS accounts
  2. Grant Gruntwork access to the AWS accounts
Create the AWS accounts

You can use the create command to create these new AWS accounts as child accounts of your AWS organization.

To create these accounts for the Standard Reference Architecture (one AWS account):

gruntwork aws create --account "main=<EMAIL_ADDRESS_FOR_ROOT_USER>"

Note that you must specify a unique email address not already associated with any other AWS account!

To create these accounts for the Standard Reference Architecture with the Multi-Account Upgrade (multiple AWS accounts):

gruntwork aws create \
  --account "dev=<EMAIL_ADDRESS_FOR_DEV_ROOT_USER>" \
  --account "stage=<EMAIL_ADDRESS_FOR_STAGE_ROOT_USER>" \
  --account "prod=<EMAIL_ADDRESS_FOR_PROD_ROOT_USER>" \
  --account "security=<EMAIL_ADDRESS_FOR_SECURITY_ROOT_USER>" \
  --account "shared-services=<EMAIL_ADDRESS_FOR_SHARED_ROOT_USER>"

Note that you must specify a unique email address for each of these accounts that is not already associated with any other AWS account! If you use GMail or Google Apps, a handy trick is that you can create aliases of your email address using a plus sign (e.g., if your email is jon@doe.com, you can use jon+any-text@doe.com as an alias)!

To see all available options, run gruntwork aws create --help.

Grant Gruntwork employees access to the AWS accounts

To Grant Gruntwork employees access to your AWS account(s), you need to create an IAM role in each account that can be assumed from a Gruntwork AWS account.

To create this IAM role for the Standard Reference Architecture (one AWS account):

gruntwork aws grant \
  --managed-policy "AdministratorAccess" \
  --iam-user-account-name "main"  \
  --account "main"

To create this IAM role for the Standard Reference Architecture with the Multi-Account Upgrade (multiple AWS accounts):

gruntwork aws grant \ 
  --managed-policy "AdministratorAccess" \
  --iam-user-account-name "security"  \
  --account "dev" \
  --account "stage" \
  --account "prod" \
  --account "security" \
  --account "shared-services"

If you don't have access to the root account, or you want to grant access to a single child account, you can authenticate directly to the child account and run the grant command with --account set to __current__. Note that in this case, you must also specify the ID of the AWS account where your IAM users are defined: that is, the ID of the security account in the Multi-Acccount Reference Architecture, or the main account in the Standard Reference Architecture:

gruntwork aws grant --managed-policy "AdministratorAccess" \
  --account "__current__" \
  --iam-user-account-id "<SECURITY_ACCOUNT_ID>"

The grant commands above give Gruntwork employees administrator access to your AWS accounts, which is necessary to deploy the Reference Architecture. If you just want to give Gruntwork employees access for help with debugging or troubleshooting, then you can grant read-only access by setting the --managed-policy flag to ReadOnlyAccess.

To see all available options, run gruntwork aws grant --help.

Bootstrap the GCP Org

We deploy the GCP Reference Architecture into multiple GCP Projects (you can learn more about Projects in the official documentation). As part of the Reference Architecture, we provide the code for managing and setting up new Projects. However, in order to use IaC to manage Projects, we need a bootstrapping Project that we can use to store the metadata and enable API services, such that credentials obtained through that Project will have the right APIs enabled.

Additionally, in the Gruntwork GCP Reference Architecture, we recommend managing permissions using GSuite Groups. GSuite Groups provides a lightweight way to group and classify members of your Organization so that they get a consistent set of permissions in GCP. You can bind various IAM roles and permissions to a Google Group, and all members of the group will inherit those permissions when they are members of the group. This allows you to grant access to GCP directly in the GSuite admin, streamlining the onboarding set up for users.

The Gruntwork Reference Architecture binds a set of predefined recommended IAM roles to Groups. As such, the Reference Architecture needs these Groups to be precreated before deployment. This tool assists the bootstrapping step by creating the set of groups the Reference Architecture expects.

As such, to bootstrap the GCP Org in preparation for deploying the Reference Architecture, you need to:

  1. Create the Admin GCP Project
  2. Create the default GSuite Groups
Create the Admin GCP Project

You can use the gcp create-admin-project command to create the admin project in your GCP Org.

To create the admin project for the GCP Reference Architecture:

gruntwork gcp create-admin-project \
    --org-id <ID_OF_YOUR_GCP_ORGANIZATION> \
    --billing-account <ID_OF_THE_BILLING_ACCOUNT_TO_USE>

This command will:

  • Create a new Project under the Org named admin, with the ID admin-xyz where xyz is a randomly generated string to uniquely identify the Project.
  • Associate the provided Billing Account to the Project (you can learn more about Billing Accounts in the official documentation).
  • Enable API services used by the Project factory IaC module:
    • cloudresourcemanager.googleapis.com
    • cloudbilling.googleapis.com
    • iam.googleapis.com
    • admin.googleapis.com

To see all available options, run gruntwork gcp create-admin-project --help.

Create the default GSuite Groups

NOTE: This step depends on the admin Project.

You can use the gcp create-groups command to create the set of default GSuite Groups we recommend for the Reference Architecture.

To create the default GSuite groups for the GCP Reference Architecture:

gruntwork gcp create-groups \
    --oauth-credentials-file-path <PATH_TO_OAUTH_CREDENTIALS> \
    --admin-project-id <ID_OF_THE_ADMIN_PROJECT> \
    --org-domain <EMAIL_DOMAIN_OF_YOUR_GCP_ORGANIZATION>

The following Groups are created in your GSuite Org when you run this command:

  • gcp-org-admin: Used to grant admin level privileges on the GCP org to the users. This translates to the Organization Admin and Organization Role Admin IAM roles.
  • gcp-project-creator: Used to grant the minimal level of privileges necessary to setup a new Project in GCP. This translates to the Organization Viewer, Project Creator, and Billing Account User IAM roles.
  • gcp-deployment-projects-full-access: Used to grant admin level privileges on all projects BUT the admin Project in GCP. See here for more info on why the admin project is omitted (LINK TODO). This corresponds to the Project Editor IAM role.
  • gcp-deployment-projects-read-only: Used to grant read only access on all projects BUT the admin Project in GCP. This corresponds to the Project Viewer IAM role.
  • For each project (admin, shared-services, dev, stage, prod):
    • gcp-{project}-full-access: Used to grant admin level privileges on the GCP Project. This translates to the Project Editor IAM role.
    • gcp-{project}-read-only: Used to grant read only access to the GCP Project. This translates to the Project Viewer IAM role.

To see all available options, run gruntwork gcp create-groups --help.

Oauth Credentials

This command uses APIs that are not activated by default when using gcloud. Specifically, the Admin SDK API is not a part of GCP, and is actually an API on GSuite. This means that the API credentials need to be sourced from a different Project from the default one, and that Project needs to have this API enabled. Luckily, the admin Project is setup to support this.

In order to obtain credentials through the admin Project, we need to create Oauth credentials on the Project. Unfortunately, there is currently no programmatic way to create Oauth credentials via the API (see https://issuetracker.google.com/issues/116182848 for the feature request). As such, you need to manually create the Oauth credentials.

To create the Oauth credentials:

  1. Go to the Google Cloud Console for managing Oauth credentials on the Project.

  2. Configure the Oauth consent screen by clicking the "Oauth consent screen" tab.

    click oauth consent screen tab

  3. On the consent screen configuration page, select "Internal" for the "Application type" and set an application name. The application name is what shows up on the Oauth permissions screen, where the app asks for permissions to access your resources. You can use any name that is recognizable (e.g "Gruntwork Wizard"). Click "save".

    configure oauth consent screen tab

  4. On the credentials tab, click "Create Credentials" and select "Oauth client ID".

    select create credentials

  5. Select "Other" and choose a name. You can use any name that is recognizable (e.g "Gruntwork Wizard"). Click "Create".

    configure oauth credentials

  6. Ignore the pop up showing your credentials and click "Ok". We will get the credentials through another way.

  7. The new credentials you created should show up in the list view. Click the download link for the credential you just created to download the JSON file.

    download oauth credentials

Bootstrap the Git repos

By default, the Gruntwork Reference Architecture uses 4 Git repos:

  1. infrastructure-modules: Reusable modules that define the entire infrastructure.
  2. infrastructure-live: Uses the modules in infrastructure-modules to deploy your live environments.
  3. sample-app-frontend: Sample frontend app that is deployed in the Reference Architecture.
  4. sample-app-backend: Sample backend app that is deployed in the Reference Architecture.

If you're using GitHub, you can use the gruntwork CLI to create these repos automatically:

  1. Create the GitHub repos
  2. Grant Gruntwork access to the GitHub repos

If you're using a different Git host (e.g., BitBucket or GitLab), you will need to create the repos listed above manually and give Gruntwork employees read + write access to them.

Create the GitHub repos

To create these four repos in your GitHub organization:

gruntwork github create --github-org "<GITHUB_ORG_NAME>"

If you want to use your personal GitHub account instead of a GitHub organization, just omit the --github-org param:

gruntwork github create

To see all available options, run gruntwork github create --help.

Grant Gruntwork access to the GitHub repos

To grant Gruntwork access to your GitHub repos in a GitHub organization:

gruntwork github grant --github-org "<GITHUB_ORG_NAME>"

This will create a team called "Gruntwork" in your GitHub organization, add the GitHub IDs of Gruntwork employees to the team, and grant the team access to the Git repos from the previous step.

If you are using a personal GitHub account instead of a GitHub organization, just omit the --github-org param:

gruntwork github grant

This will add Gruntwork employees directly as collaborators on each of the Git repos from the previous step.

To see all available options, run gruntwork github grant --help.

Bootstrap the domain names

The Reference Architecture uses Route 53 to manage DNS, configuring domain names for the public load balancers, Jenkins, OpenVPN server, Bastion Host, and CloudFront. We strongly recommend buying placeholder domain names from Route 53 so we can get everything working with the Reference Architecture without affecting any of your production domain names. Most .com domain names are just $12! You can find the latest pricing on this page.

To check if a domain name is available, you can run:

gruntwork dns check --domain "<DOMAIN_NAME>"

If the domain name is available and the pricing listed on this page for that domain looks reasonable, you can register it using the register command.

For the single-account Reference Architecture, where the account is called "main", you would run:

gruntwork dns register \
  --account "main=<DOMAIN_NAME>" \
  --first "<FIRST_NAME>" \
  --last "<LAST_NAME>" \
  --org "<COMPANY_NAME>" \
  --address "<ADDRESS>" \
  --city "<CITY>" \
  --state "<STATE_CODE>" \
  --zip "<ZIP_CODE>" \
  --country "<COUNTRY_CODE>" \
  --email "<EMAIL>" \
  --phone "<PHONE>"

For the multi-account Reference Architecture, you should register a different domain name in each of the dev, stage, prod, and shared-services accounts:

gruntwork dns register \
  --account "dev=<DEV_DOMAIN_NAME>" \
  --account "stage=<STAGE_DOMAIN_NAME>" \
  --account "prod=<PROD_DOMAIN_NAME>" \
  --account "shared-services=<SHARED_SERVICES_DOMAIN_NAME>" \
  --first "<FIRST_NAME>" \
  --last "<LAST_NAME>" \
  --org "<COMPANY_NAME>" \
  --address "<ADDRESS>" \
  --city "<CITY>" \
  --state "<STATE_CODE>" \
  --zip "<ZIP_CODE>" \
  --country "<COUNTRY_CODE>" \
  --email "<EMAIL>" \
  --phone "<PHONE>"

Here's a concrete example so you can see the types of values you have to specify:

gruntwork dns register \
  --account "dev=acme-corp-dev.com" \
  --account "stage=acme-corp-stage.com" \
  --account "prod=acme-corp-prod.com" \
  --account "shared-services=acme-corp-shared.com" \
  --first "Jane" \
  --last "Smith" \
  --org "Acme Corp" \
  --address "123 Main St" \
  --city "San Francisco" \
  --state "CA" \
  --zip "94016" \
  --country "US" \
  --email "jane.smith@acme-corp.com" \
  --phone "+1.5551234567"

To register a domain name in the "current" account—the one you are authenticated to in the terminal—use the name __current__ (i.e., --account "__current__=acme-corp.com").

To see all available options, run gruntwork dns register --help.

WARNING: Using the register command will buy one or more domain names and charge your account. Make sure to check the pricing listed on this page to make sure the price is reasonable!

Bootstrap the TLS certs

To secure your public endpoints with TLS/SSL, the Reference Architecture uses TLS Certificates from Amazon's Certificate Manager (ACM). These certificates are managed by AWS, automatically renewed for you, and completely free!

To request a certificate for the single-account Reference Architecture, run:

gruntwork tls request \
  --account "main=*.<DOMAIN_NAME>" \
  --region "<REGION>"

To request a certificate for each of the accounts in the multi-account Reference Architecture, run:

gruntwork tls request \
  --account "dev=*.<DEV_DOMAIN_NAME>" \
  --account "stage=*.<STAGE_DOMAIN_NAME>" \
  --account "prod=*.<PROD_DOMAIN_NAME>" \
  --account "shared-services=*.<SHARED_SERVICES_DOMAIN_NAME>" \
  --region "<REGION>"

Here's a concrete example so you can see the types of values you have to specify:

gruntwork tls request \
  --account "dev=*.acme-corp-dev.com" \
  --account "stage=*.acme-corp-stage.com" \
  --account "prod=*.acme-corp-prod.com" \
  --account "shared-services=*.acme-corp-shared.com" \
  --region "us-east-1"

To request a TLS cert in the "current" account—the one you are authenticated to in the terminal—use the name __current__ (i.e., --account "__current__=*.acme-corp.com").

A few important notes:

  1. Please request a wildcard certificate: e.g., *.your-company.com. This allows us to use the same cert for multiple subdomains (e.g., www.your-company.com, stage.your-company.com, jenkins.your-company.com, etc).

  2. For the --region parameter, specify the primary AWS region you wish to use. Note that CloudFront can only use TLS certs in us-east-1, so if you want to use CloudFront with your Reference Architecture, you must request certs for us-east-1, in addition to any other region you may be using.

  3. If your domain names are registered with a different registrar and not Route 53 (e.g., GoDaddy), specify the --skip-route53-check flag. Note that to use an external domain name with the Reference Architecture, you will (a) have to create Route 53 Hosted Zones for that domain name, as documented here and (b) we will use email validation with the TLS certs, so the owner of the domain will receive confirmation emails from AWS and will have to click the 'confirm' link for each domain name before AWS will issue a certificate.

To see all available options, run gruntwork tls request --help.

Revoking access

Once the Reference Architecture is deployed and you are fully up and running on it, it can be a good idea to revoke Gruntwork's access to your accounts (this reduces your surface area to potential attackers):

  1. Revoke Gruntwork's access from the AWS accounts
  2. Revoke Gruntwork's access from the GitHub repos
  3. Revoke Gruntwork's access from the BitBucket or GitLab repos

Note: If you ever need Gruntwork to access your accounts in the future (e.g., to help with troubleshooting), you can re-run the commands from the Grant Gruntwork access to the AWS accounts and Grant Gruntwork access to the Git repos sections.

Revoke Gruntwork's access from the AWS accounts

To remove the IAM role that gives Gruntwork access to the Standard Reference Architecture (one AWS account):

gruntwork aws revoke --account "main"

To remove the IAM roles for the Standard Reference Architecture with the Multi-Account Upgrade (multiple AWS accounts):

gruntwork aws revoke \
  --account "dev" \
  --account "stage" \
  --account "prod" \
  --account "security" \
  --account "shared-services"

To revoke Gruntwork's access from the "current" account—the one you are authenticated to in the terminal—use the name __current__ (i.e., --account "__current__").

To see all available options, run gruntwork aws revoke --help.

Revoke Gruntwork's access from the GitHub repos

To revoke Gruntwork's access from the GitHub repos in a GitHub organization:

gruntwork github revoke --github-org "<GITHUB_ORG_NAME>"

This will delete the GitHub team called "gruntwork" (by default).

If you are using a personal GitHub account instead of a GitHub organization, just omit the --github-org param:

gruntwork github revoke

This will remove Gruntwork employees as collaborators from each of the Git repos.

To see all available options, run gruntwork github revoke --help.

Revoke Gruntwork's access from the BitBucket or GitLab repos

The gruntwork CLI does not currently support BitBucket or GitLab, so for now, you'll have to remove Gruntwork's access from your Git repos manually.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Contact Us

Ready to hand off the Gruntwork?

Buy Now