Browse the Repo

file-type-icon_docs
file-type-icon_images
file-type-icon01-architecture-overview.md
file-type-icon02-whats-deployed.md
file-type-icon03-security-compliance-compatibility.md
file-type-icon04-how-code-is-organized.md
file-type-icon05-dev-environment.md
file-type-icon06-ci-cd.md
file-type-icon07-monitoring-alerting-logging.md
file-type-icon08-ssh-vpn.md
file-type-icon09-accounts-and-auth.md
file-type-icon10-gruntwork-tools.md
file-type-icon11-deploying-a-docker-service.md
file-type-icon12-migration.md
file-type-icon13-deploying-the-reference-architecture-fr...
file-type-icon14-undeploying-the-reference-architecture.md
file-type-icon15-adding-new-environments-regions-and-acc...
file-type-iconREADME.md
file-type-icondev
file-type-iconmaster
file-type-iconprod
file-type-iconsecurity
file-type-iconshared-services
file-type-iconstage
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md

Browse the Repo

file-type-icon_docs
file-type-icon_images
file-type-icon01-architecture-overview.md
file-type-icon02-whats-deployed.md
file-type-icon03-security-compliance-compatibility.md
file-type-icon04-how-code-is-organized.md
file-type-icon05-dev-environment.md
file-type-icon06-ci-cd.md
file-type-icon07-monitoring-alerting-logging.md
file-type-icon08-ssh-vpn.md
file-type-icon09-accounts-and-auth.md
file-type-icon10-gruntwork-tools.md
file-type-icon11-deploying-a-docker-service.md
file-type-icon12-migration.md
file-type-icon13-deploying-the-reference-architecture-fr...
file-type-icon14-undeploying-the-reference-architecture.md
file-type-icon15-adding-new-environments-regions-and-acc...
file-type-iconREADME.md
file-type-icondev
file-type-iconmaster
file-type-iconprod
file-type-iconsecurity
file-type-iconshared-services
file-type-iconstage
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md
Multi-account Reference Architecture

Multi-account Reference Architecture

End-to-end tech stack designed to deploy into multiple AWS accounts. Includes VPCs, EKS, ALBs, CI / CD, monitoring, alerting, VPN, DNS, and more.

Code Preview

Preview the Code

mobile file icon

08-ssh-vpn.md

down

SSH and VPN

In the previous section, you saw how to use Monitoring, Alerting, and Logging to diagnose issues. Sometimes, that's not enough, and you need to connect directly to your servers using:

SSH

You can use SSH to connect to any of your EC2 Instances as follows:

The traditional way: EC2 Key Pairs

When you launch an EC2 Instance in AWS, you can specify an EC2 Key Pair that can be used to SSH into the EC2 Instance. This suffers from an important problem: usually more than one person needs access to the EC2 Instance, which means you have to share this key with others. Sharing secrets of this sort is a security risk. Moreover, if someone leaves the company, to ensure they no longer have access, you'd have to change the Key Pair, which requires redeploying all of your servers.

The better way: ssh-grunt

To solve the "key sharing" problem, Gruntwork implemented ssh-grunt, a tool that enables each member of your team to log in to every EC2 Instance with their own IAM user name and their own SSH key. Here's how it works:

One-Time Setup

  1. Log in to the AWS Web Console with your IAM User account.

  2. Go to your IAM User profile page, select the Security credentials tab, and click Upload SSH public key. Now upload your public SSH key (e.g. ~/.ssh/id_rsa.pub). Do NOT upload your private key.

  3. Now make sure your IAM User account is a member of either the ssh-grunt-users or ssh-grunt-sudo-users group. By being a member of one of these IAM Groups, any EC2 Instance configured to use these IAM Groups will permit you to login as either a non-sudo user or sudo user, depending on which IAM Group you're in.

  4. Note that your linux username is based on your IAM User name according to the ssh-grunt guidelines. For example:

    • The IAM User name josh will be the linux username josh.
    • The IAM User name josh@gruntwork.io will be the linux username josh
    • The IAM User name _gruntwork.josh.padnick will be the linux username gruntwork_josh_padnick.

For more information, see the ssh-grunt documentation

SSH to an EC2 Instance

As an example, suppose that:

  • Your IAM User name is josh.
  • You've uploaded your public SSH key to your IAM User profile.
  • Your private key is located at /Users/josh/.ssh/id_rsa on your local machine.
  • Your EC2 Instance's IP address is 1.2.3.4.

Then you can SSH to the EC2 Instance as follows:

# Do this once to load your SSH Key into the SSH Agent
ssh-add /Users/josh/.ssh/id_rsa

# Every time you want to login to an EC2 Instance, use this command
ssh josh@1.2.3.4

VPN

For security reasons, just about all of your EC2 Instances run in private subnets, which means they do not have a public IP address, and cannot be reached directly from the public Internet. This reduces the "surface area" that attackers can reach. Of course, we still need access into the VPCs, so we expose a single entrypoint into the network: an OpenVPN server.

The idea is that you use an OpenVPN client to connect to the OpenVPN server, which gets you "in" to the network, and you can then connect to other resources in the account as if you were making requests from the OpenVPN server itself.

Here are the steps you'll need to take:

VPN one-time setup

The very first time you want to use OpenVPN, you need to:

Install an OpenVPN client

There are free and paid OpenVPN clients available for most major operating systems:

Join the OpenVPN IAM group

To get access to an OpenVPN server, your IAM user need access to SQS queues used by that OpenVPN server. Since our IAM users are defined in one AWS account (security) and the OpenVPN servers are defined in separate AWS accounts (stage, prod, etc), that means you need to "switch" to the accounts with the OpenVPN servers by assuming an IAM role that has access to the SQS queues in those accounts.

To be able to assume an IAM role, your IAM user needs to be part of an IAM group with the proper permissions, such as _account.xxx-full-access or _account.xxx-openvpn-users, where xxx is the name of the account you want to access (stage, prod, etc).

Use openvpn-admin to generate a configuration file

To connect to an OpenVPN server, you need an OpenVPN configuration file, which includes a certificate that you can use to authenticate. To generate this configuration file, do the following:

  1. Install the latest openvpn-admin binary for your OS.

  2. Set up your AWS credentials using any of the options supported by AWS CLI tools. Typically, environment variables are the easiest and most secure.

  3. Use aws-auth to assume an IAM role in the AWS account with the OpenVPN server. This IAM role must have access to the SQS queues used by OpenVPN server. Typically, the allow-full-access-from-other-accounts or openvpn-server-allow-certificate-requests-for-external-accounts IAM role is what you want. For example, if your prod account had account ID 1111111111111, you would run:

    aws-auth --role-arn arn:aws:iam::1111111111111:role/allow-full-access-from-other-accounts
    

    If you have MFA enabled (recommended!), and your IAM users and MFA settings are defined in account 222222222222, you would run:

    aws-auth --role-arn arn:aws:iam::1111111111111:role/allow-full-access-from-other-accounts --serial-number arn:aws:iam::222222222222:mfa/jondoe --token-code 123456    
    
  4. Run openvpn-admin request --aws-region us-east-1 --username <YOUR_IAM_USERNAME>.

  5. This will create your OpenVPN configuration file in the current folder.

  6. Load this configuration file into your OpenVPN client.

Connect to the OpenVPN server

To connect to the OpenVPN server, simply click the "Connect" button next to your configuration file in the OpenVPN client! After a few seconds, you should be connected.

Connect to other resources

Now that you're connected to VPN, you can connect to other resources in your AWS account. For example, if you followed the ssh-grunt setup instructions above, you can SSH to an EC2 Instance with private IP address 1.2.3.4 as follows:

ssh <your_username>@1.2.3.4

# Example:
ssh josh@1.2.3.4

Similarly, non-production resources, such as a load balancer in the staging environment, or Jenkins in the mgmt environment, should now be accessible to you.

Note: we run OpenVPN in "split tunnel" mode. That means that only the IP addresses we have explicitly opted into (namely, the private IP addresses in your AWS account) will be routed over VPN. Other IP addresses, such as requests you make from your computer to YouTube, GMail, Spotify, etc, are NOT routed over VPN. This dramatically reduces the load on your OpenVPN server and your bandwidth usage in AWS.

Next steps

Now that you know how to connect to your servers, let's talk about auth for your AWS account(s).

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?