all blog posts


The Ideal SSH Config for Instances Behind a Bastion Host

If you're working with public cloud environments, you're most likely familiar with bastion hosts (or jump boxes). If you're managing multiple environments, you probably also know that keeping track of keys, ports and usernames can quickly become very hard. This blog post will help you clear the clutter.

TL;DR - If you just want to see the config, jump to the bottom of the article.

In public cloud environments, best practice is to put sensitive infrastructure like web servers and databases in private subnets. But if you need to access them, for example for bug tracing or troubleshooting, you might still want to SSH into an instance. A common solution to achieve this is to implement a bastion host.

A bastion host is a single point of access into your environment. You place it in a public subnet, but you limit its firewall to only your home, office or VPN IP address. You make sure that the instance is hardened and has as little software as possible installed; less packages means less chance of vulnerabilities.

Bastion

This simple setup is quite maintainable, especially when the bastion host and the web server share the same key. But you're already facing an issue: how to make sure that the SSH key to connect to the web server (2) is present on the bastion host? You could copy the key to the bastion host, but that would be a very bad solution.

SSH Agent

The solution to this problem is to use ssh-agent. However, we will see that this is not the perfect solution in a later section.

ssh-agent allows you to store private keys in the agent. When connecting to an instance, the ssh command will retrieve the private key from the agent.

Here's an example for connecting to the bastion host:

# lucvandonkersgoed at macbook.local in ~ [22:07:16] 
→ ssh-add -D
All identities removed.

# lucvandonkersgoed at macbook.local in ~ [22:07:18] 
→ ssh-add ~/Downloads/public-key-pair.pem
Identity added: /Users/lucvandonkersgoed/Downloads/public-key-pair.pem (/Users/lucvandonkersgoed/Downloads/public-key-pair.pem)

# lucvandonkersgoed at macbook.local in ~ [22:07:23] 
→ ssh ec2-user@52.215.235.28

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-0-222 ~]$ 

The ssh-add -D clears any existing keys from the agent. ssh-add ~/Downloads/public-key-pair.pem adds a key to the agent. In the ssh command, we don't specify the key, but the connection succeeds because the key is available in the agent.

Forwarding the ssh-agent

Let's assume the private instance uses the same username and ssh key. In that case, connecting to it should be as easy as ssh 10.0.1.72, right? You might think the key is already present in the agent, so the ssh command should just use that. Unfortunately, it won't. You will be greeted with a error like this:

[ec2-user@ip-10-0-0-222 ~]$ ssh 10.0.1.72
Permission denied (publickey).

You want to make sure the bastion host has access to your ssh agent. To achieve this, edit ~/.ssh/config and make sure it contains the following segment.

Host *
  ForwardAgent Yes

Update: After releasing this blog post, /u/aquatictortoise commented that using ForwardAgent poses a security risk:

One month ago at matrix.org, they were severely compromised because their team thought this was a good idea (and they only trusted their own infra while at it). When using SSH agent forwarding, you're making a compromise of your bastion host equal to a compromise of all your systems. That's the exact opposite of what you actually want to achieve with a bastion host, right? You have to use ProxyJump or -J if you're gonna have a bastion host. You don't want agent forwarding. In fact, you should probably just forget that it even exists.

As such, we need to actively advise against using ForwardAgent. However, we will still retain the original content of this blog post for posterity.

After you have updated your ssh config, disconnect from your bastion host, and ssh to it again. Now connect to the web server, and tadaaa: we're connected.

# lucvandonkersgoed at macbook.local in ~ [22:18:06] 
→ ssh ec2-user@52.215.235.28

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-0-222 ~]$ ssh 10.0.1.72

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-1-72 ~]$ 

Web server with a different ssh key

If the web server in the private subnet uses a different ssh key then the bastion, you can just add multiple ssh keys to the ssh agent:

# lucvandonkersgoed at macbook.local in ~ [22:25:17] 
→ ssh-add -D                                         
All identities removed.

# lucvandonkersgoed at macbook.local in ~ [22:25:28] 
→ ssh-add ~/Downloads/public-key-pair.pem 
Identity added: /Users/lucvandonkersgoed/Downloads/public-key-pair.pem (/Users/lucvandonkersgoed/Downloads/public-key-pair.pem)

# lucvandonkersgoed at macbook.local in ~ [22:25:38] 
→ ssh-add ~/Downloads/private-key-pair.pem 
Identity added: /Users/lucvandonkersgoed/Downloads/private-key-pair.pem (/Users/lucvandonkersgoed/Downloads/private-key-pair.pem)

Configuring the ssh key in ~/.ssh/config

The problem with using ssh-agent described above is that it doesn't scale. If you maintain multiple environments, with multiple servers, each with their own key, you will need to remember when to use which key. This might work for two environments, maybe even for five, but imagine doing this for tens or hundreds of servers.

Also, as described above, using ForwardAgent poses security risks. So what is the alternative?

The solution is to store which key (and username) to use in ~/.ssh/config. Open the config file and change its contents to the following:

Host 52.215.235.28
  User ec2-user
  IdentityFile ~/Downloads/public-key-pair.pem

If you still had keys stored in your agent, now is the time to clear them with ssh-add -D. Then connect with a simple ssh 52.215.235.28.

# lucvandonkersgoed at macbook.local in ~ [22:34:19] 
→ ssh-add -D       
All identities removed.

# lucvandonkersgoed at macbook.local in ~ [22:34:44] 
→ ssh 52.215.235.28

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-0-222 ~]$ 

Epic win! You didn't need to specify the username or identity file. Instead, ssh fetched it from your config.

However, adding configuration for the web server to your local ssh config file won't work. We will solve this in the last section of this blog post. But first: aliases.

Using aliases for your hosts

A very cool feature of ~/.ssh/config is the ability to give an easy-to-remember name to your hosts. An example:

Host my-bastion
  User ec2-user
  IdentityFile ~/Downloads/public-key-pair.pem
  Hostname 52.215.235.28

Change your config to this, then run ssh my-bastion and hurrah! We're connected.

# lucvandonkersgoed at macbook.local in ~ [22:40:46] 
→ ssh my-bastion

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-0-222 ~]$ 

Bringing it all together with ProxyJump

ProxyJump is a super powerful configuration option when working with bastion hosts. Take a look at this config:

Host my-bastion
  Hostname 52.215.235.28
  User ec2-user
  IdentityFile ~/Downloads/public-key-pair.pem

Host private-web-server
  Hostname 10.0.1.157
  User ec2-user
  ProxyJump my-bastion
  IdentityFile ~/Downloads/private-key-pair.pem

As you can see, the private web server has a private IP address, and should be inaccessible without jumping through a bastion host. But by configuring the ProxyJump property, we can do exactly that. Update your config, then run ssh private-web-server:

# lucvandonkersgoed at macbook.local in ~ [22:47:02] 
→ ssh private-web-server

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
[ec2-user@ip-10-0-1-157 ~]$ 

No ssh-agent, no difficult to remember IP addresses or hostnames; just ssh and an alias. The entire bastion just became invisible. The username, hostname and private key are all fetched from the config. If you have a custom ssh port, you can configure that in the config as well.

If you're managing multiple environments, I'd advise to stick to a simple naming pattern:

Host customer1-bastion
  Hostname 52.215.235.28
  User ec2-user
  IdentityFile ~/Downloads/customer1-bastion.pem

Host customer1-web-server
  Hostname 10.0.1.157
  User ec2-user
  ProxyJump customer1-bastion
  IdentityFile ~/Downloads/customer1-web.pem

Host customer2-bastion
  Hostname 38.158.100.12
  User ec2-user
  IdentityFile ~/Downloads/customer2-bastion.pem

Host customer2-mysql-server
  Hostname 192.168.2.123
  User ubuntu
  ProxyJump customer2-bastion
  IdentityFile ~/Downloads/customer2-db.pem

Conclusion

I've been using this ssh config for years and couldn't be happier with it. If I've missed any awesome config options, or if you have any questions, reach out to me on Twitter.


Related blog posts


all blog posts