Improving Your Daily Development Workflow - Custom SSH Configurations

This post describes the SSH configuration I use to simplify the daily work of connecting to remote hosts with SSH and public key authentication.

Introduction

At Entwine we use SSH to connect to a wide range of different servers: local development VMs, servers running in dedicated data centers, client installations (running on their own infrastructure), QA/testing hosts running in AWS and so on.

Due to the heterogeneity in the setup of the different hosts, we usually have to know quite a bit about each of the hosts we want to connect to:

• what is the hostname?
• on which port does SSH listen?
• which user do we use to connect to the host?
• which key do I have to use and what’s the passphrase?

I don’t like to read through the system documentation every time I want to connect to a host. Therefore I’ve setup a custom SSH configuration which takes away most of the burden.

The SSH configuration file

The best reference about the SSH configuration is its man page.

At the very beginning you find this information in the man page:

ssh obtains configuration data from the following sources in the following order:

1. command line options
2. user’s configuration file (~/.ssh/config)
3. system-wide configuration file (/etc/ssh/ssh_config)

I would recommend to put all the configuration options into the user’s configuration file to avoid entering command line options constantly. This way also allows other users of the systems to setup their own set of SSH configuration options without conflicts.

Configuration options for a host

Within your SSH config file, you can now start to add configuration options. I usually makes sense to group them by host. A very basic section could look like this:

Host foo.bar
  User demo
  Port 2222

This makes demo the default user and 2222 the default port when a connection is made to the host foo.bar. Instead of entering in all of the options on the command line like so:

$> ssh foo.bar -l demo -p 2222

you can now omit the login as well as the port command line option and just type

$> ssh foo.bar

Working with several keys

For several reasons, we don’t use the same SSH key to connect to all the hosts we manage. All the private & public keys I use are stored in the ~/.ssh directory and are organized like this:

$> ll ~/.ssh
-rw-------+  client_a
-rw-------+  client_b
-rw-------+  id_rsa

Instead of logging in directly you now have to tell SSH which key to use. Of course, this can be done at the command line using the -i option (i stands for identity file):

$> ssh foo.bar -i ~/.ssh/client_a

Again, I don’t want to have to type (and remember) the identity file to use each and every time I have to connect to a certain host. There are two options to prevent this:

Option 1: Set IdentityFile in the config

The config file allows to specify an identity file per host. The host configuration block then looks like this:

Host foo.bar
  User demo
  Port 2222
  IdentityFile ~/.ssh/client_a

Connecting to the remote host is now again simple by running:

$> ssh foo.bar

Option 2: Add the keys to your SSH agent

Option 1 is easy to setup, although there are two drawbacks:

1. If your private key is protected by a passphrase (which is definitely recommended), you have to type it each and every time you try to connect to the remote host. If you have several different keys all having a different passphrase, this gets very cumbersome.
2. You don’t have your local keys available on the remote host (this feature is called agent forwarding and is not part of this post; GitHub has some good information about it)

So, I usually go with option 2, which is to use the ssh-agent. Wikipedia writes about the ssh-agent:

ssh-agent is a program that, used together with OpenSSH or similar SSH programs, provides a secure way of storing the private key. For private keys that require a passphrase, ssh-agent allows the user to connect multiple times without having to repeatedly type the passphrase.

First of all, you have to add your private keys to the ssh-agent:

$> ssh-add ~/.ssh/client_a

If you’re a OS X user, you may want to add the option -K. Your passphrase will then be stored in the keychain and you don’t have to type it in again. With option -l (list) you can check which keys are part of your ssh-agent:

$> ssh-add -l
2048 41:5f:00:(...):e2:bc:89 /Users/user/.ssh/id_rsa (RSA)
2048 c4:17:e3:(...):23:ff:13 /Users/user/.ssh/client_a (RSA)

Now, ssh automatically tries to use the identity files which are in your ssh-agent when connecting to a remote host and you never have to type your passphrase!

Host name alias

Another difficult thing to remember are all the exact host names. Since most of the hosts at client side are managed by their own IT departments, we cannot enforce any naming conventions. This results in a large variety of host names. e.g.

• mhadmin.dev.dep.co.uk
• mh-storage-1-dev.client.com
• worker1.mh.client.ch

To make the host names a bit more predictable, I’ve setup alias hosts in my SSH config which all follow a certain naming convention.

Host client_a-mhadmin
  HostName mhadmin.dev.dep.co.uk
Host client_a-mhworker1
  HostName mhworker-1.dev.dep.co.uk

With this, connecting to the Matterhorn worker host at a client is easy as:

$> ssh client_a-mhworker1

Proxy

In some cases hosts are not directly accessible by SSH but only through a proxy host that is externally available. For opening a connection to the target host, I first have to open a connection to the proxy host and then open a second SSH connection to the target host. For example you might need to execute the following:

$> ssh proxy.foo.bar -l demo -p 2222
proxy $> ssh target.foo.bar

Wouldn’t it be much nicer if we could connect to the target host just with:

$> ssh target.foo.bar

ProxyCommand to the rescue! Just add the first command as proxy command to your host configuration:

Host foo.bar
  User demo
  Port 2222
  ProxyCommand ssh proxy.foo.bar -l demo -p 2222

Done!

Roundup

My ssh configuration file now looks like this:

Host *.foo.bar
  User demo
  Port 2222

Host worker1.foo.bar
  ProxyCommand ssh proxy.foo.bar -l demo -p 2222

Host client_a-mhadmin
  HostName mh-admin.foo.bar
Host client_a-mhworker1
  HostName mh-worker1.foo.bar

The configuration file in combination with the ssh-agent allows me now to connect to all kinds of different hosts in a straight-forward and transparent way.