tl;dr: Well-known yet underused OpenSSH features and their applications in building secure systems.
My last post on this
blog was about the
dangers of using SSH Agent Forwarding. In that post I recommended using OpenSSH
ProxyCommand as not only a workaround but actually a superior way of hopping
between hosts. This feature is clearly documented and intended to be used as
such, still a lot of the discussions which ensued on Reddit and Netsec showed me
that there are quite a number of users who were previously unaware of these
features. That’s what prompted me to do a follow-up on some additional
“tricks”, even if it’s just the application of something clearly documented in
the manpages. But then, not everybody reads the manpages.
This post will cover some of these features and also provide some references for further reading. I’ll start by introducing the basic features and then presenting some ways to use them. The biggest takeaway is this:
OpenSSH is a lot more than a tool to securely to connect to your VPS. Think of it as a simple, well-understood building-block for constructing secure distributed systems, for both automated and interactive applications. With the right workflow, employing OpenSSH will come to you very naturally and comfortably.
Me, April 2015
We start of by applying reasonable security settings to our OpenSSH workflow. In the recent months there were enough posts written on this topic, so I’ll just reference them here. It boils down to this:
The most current and comprehensive post on this topic has been written by @stribika in his blog-post Secure Secure Shell. His observations are spot-on, including the section on “System hardening”: Keep a clean system.
What I like to do on top of this is to employ some way of monitoring the SSH logs and rate-limiting connection attempts in a very crude way. Monitoring SSH connection attempts can be done by logging TCP connections to your SSH ports and then ingesting these logs into something like Logstash and Kibana. To rate-limit connection attempts I use a very simple iptables ruleset because I don’t trust systems like fail2ban enough.
Next we turn our attention to your
~/.ssh/config file. This file can be used
to configure per-host parameters, most commonly a combination of a custom port
You should consider
~/.ssh/config your authoritative and only sources about
your hosts. Whenever you get access to a new host just add it to this file and
don’t worry about remembering it. The
Host line can be used to create aliases
which are easy to remember (e.g.
ams-vps instead of a raw IP).
Let’s examine a contrived example to show the options I find myself most frequently using:
1 2 3 4 5 6 7 8 9 10 11 ServerAliveInterval 10 TCPKeepAlive no Host ams-vps Hostname 22.214.171.124 Port 12345 User root IdentityFile ~/.ssh/vps_key IdentitiesOnly yes LocalForward 8080 localhost:80 DynamicForward 9090
Let’s go through these lines one by one, ignoring the “Alive” lines.
ams-vpsso I can remember it easily.
Hostnameis the actual IP or hostname of the host.
vps_keyand I only want to use that one.
tcp/80on the remote host via
tcp/9090terminated with the remote host.
~/.ssh/config is currently ~250 lines long, just to give you an idea. The
newest versions of OpenSSH even allow include statements in this file.
ControlMaster is one of the options that I use globally in my
ControlMaster does is create (and subsequently use) a control socket
for each connection to a remote server. SSH supports multiple independent
“channels” which can be multiplexed over a single existing SSH connection. On
the first connection to a host, OpenSSH will create a control socket in
With each subsequent connection, if a ControlMaster exists for a given
user/host/port, OpenSSH will use it to create a new channel (pty or scp) within
the existing SSH connection without going through the SSH handshake and shared
secret agreement. The socket will be destroyed when the last SSH session
The biggest reason for me to use
ControlMaster is performance. The SSH
handshake does take a while, even more so with increasing latency. While this
might not be an issue for the occasional login, it is really annoying or
downright prohibitive when trying to connect more frequently. The prime example
here is remote scp-completion, something that zsh is capable of doing out of
the box. Unless you’re on your local network, scp-completion just plain sucks
ControlMaster. The problem is even amplified when considering
scenarios with more than one hop (
ProxyCommand). In this case, each full SSH
session establishment can take several seconds.
ControlPersist keyword is optional. I don’t use it because I like to keep
track of the open session, but software like Ansible employs it to have the
control socket linger for a while after the last/initial SSH connection is
I’m not going to talk about
ProxyCommand at length since I mentioned its
benefits in my last
post. As a quick recap:
ProxyCommand allows your system to connect to an otherwise inaccessible system
via one or multiple intermediate “hops”. It has undeniable advantages in terms
of usability and security over using something like SSH agent forwarding. The
number of ways you could get pwned when not using
ProxyCommand is not
something to be dismissed lightly.
Suffice to say, none of the features I mentioned in this post (
ControlMaster) work if you use agent forwarding or manual
hopping as opposed to using
ProxyCommand. One more reason to reconsider.
authorized_keys file specifies which public key is allowed to login to
the current account. You’ve probably used it when adding your own public key on
a remote server. The basic format of this is very simple: It takes one SSH
public key per line to allow login via that key.
Some people might not know that you can actually add a number of options per
key. The complete description is in
man sshd, I’ll only cover the options I
commandwill execute a fixed command for each login with this public key, ignoring any other commands supplied by the client.
no-port-forwardingwill disallow agent/port forwarding.
no-ptywill disallow pty allocations, useful for automation.
permitopen="host:port"will only allow port forwards to this host/port. Can be repeated.
Using these options, one can use OpenSSH with a passphrase-less dedicated key for some unsupervised applications:
For a constant connection you can simply run
ssh via some process supervisor
The intention of this post was to show that OpenSSH is much more than just a remote-login tool. It can be used for various automated applications. The nice thing about using SSH for these cases is that it is cross-platform, dead-simple to setup, test and to actually understand the security and features offered by such a solution.
Want to forward a single port with some low-bandwidth yet high-value traffic over the Internet? Why set up a complicated VPN solution when you know how to use SSH? Want to have automated backups? Use SSH! Even if you only use SSH interactively, this post might have shown you a few tricks to improve your workflow.
Having something that a user already understands and which does not introduce a new attack surface is a huge security win in my opinion. The tendency of some modern software is to just bind to localhost and omit any form of authentication or transport security. Here, SSH can be used as a security layer, even if only during development. The same is true for supposedly secure protocols which you still don’t trust entirely. Used correctly, OpenSSH is a very robust system that does authentication, authorization and proper transport security and is part of every conceivable distribution. Also consider that OpenSSH is one of the pieces of your userland toolkit which is most closely reviewed and at the same time still being actively developed by the OpenBSD community.
Topics not mentioned here include: