VanDyke Software

Secure Shell is widely used to provide secure access to remote systems, and everybody who uses it is familiar with routine password authentication. This is the easiest authentication method to set up and is available by default, but using passwords alone suffers from a number of limitations. These include both security and usability issues, and Steve covers them in this tip. Steve then explores the various forms of authentication available to Secure Shell users and contrasts the security and usability tradeoffs of each. He then discusses the extra functionality of agent key forwarding, making the case that using SSH public-key authentication is a substantial win.

Ordinary Password Authentication

SSH supports authentication with a username and password, and this is little more than an encrypted Telnet. Access is, in fact, just like Telnet, with the normal username/password exchange.

We'll note that this exchange, and all others in this paper, assumes that an initial exchange of host keys has been completed successfully. Though an important part of session security, host validation is not material to the discussion of agent key forwarding.

All examples start from a user on homepc (perhaps a Windows workstation) connecting with SecureCRT to a server running OpenSSH. The particular details (program names, mainly) vary from implementation to implementation, but the underlying protocol has been proven to be highly interoperable.

1	The user makes an initial TCP connection and sends a username. We'll note that unlike Telnet, where the username is prompted as part of the connected data stream (with no semantic meaning understood by Telnet itself), the username exchange is part of the SSH protocol itself.
2	The SSH daemon on the server responds with a demand for a password; access to the system has not yet been granted in any way.
3	The SecureCRT prompts the user for a password that is relayed through the encrypted connection to the server where it is compared against the local user base.
4	If the user's password matches the local credential, access to the system is granted and a two-way communications path is established, usually to a login shell.

The main advantage of allowing password authentication is that it's simple to set up – usually the default – and is easy to understand. Systems which require access for many users from many varying locations often permit password authentication simply to reduce the administrative burden and to maximize access.

The substantial downside is that by allowing a user to enter a password, it means anybody is allowed to enter a password. This opens the door to wholesale password guessing by users or bots alike, and this has been an increasingly common method of system compromise.

Unlike prior-generation SSH worms, which attempted just a few very common passwords with common usernames, modern badware has very extensive dictionaries of usernames and passwords and has proven to be most effective in penetrating even systems with "good" passwords. Only one compromised account is required to gain entry to a system.

But even putting security issues aside, the other downside of password authentication is that passwords must be remembered and entered separately upon every login. For users with just one system to access, this may not be such a burden, but users connecting to many systems throughout the day may find repeated password entry tedious.

And having to remember a different password for every system is not conducive to choosing strong passwords.

Public-Key Authentication

To counteract the shortcomings of password authentication, SSH supports public-key authentication: the user creates a pair of public and private keys, installs the public key in his account on the target server, and access is granted on a much more secure (and more convenient) basis.

The keys are very long strings of numbers and are mathematically related. Access is granted by the server when it proves that the SSH client attempting to authenticate has the private key. In this protocol, the private key itself never travels over the wire, so there's nothing for an attacker to intercept. It's really very clever.

Both keys are stored in files, and the user's private key is further protected by a secret passphrase; when the private key is needed for authentication, the user must present this passphrase to unlock the key. This is an added protection so that a user's private key file is of no value if it falls into the wrong hands.

The public key, on the other hand, is truly public: it's not sensitive information and need not be protected. Keys are represented in ASCII as long strings of digits along with some accessory information such as the key type and a comment about the key (often the owner's name). There are a few formats for these keys:

---- BEGIN SSH2 PUBLIC KEY ----
Subject: steve
Comment: "steve@unixwiz.net"
AAAAB3NzaC1yc2EAAADAAQABAAAAgQCiR1Jo2hMbZp+XOIS6R+U9TrSxJjIMfk/SikL4
v38KINrPBUfmC2QxhhyP3zlthXRpzqJHbdp1wRodRiCr7Dx6JcxDQRh85IO0AzlKdQUi
LwQ41Gz5WOtSGCbQ1Eag7EZaJ0XLdHvWYrNOo0mWcjYsuDlDgoeRSHWRsnrwGvMPv1==
	---- END SSH2 PUBLIC KEY ----

ssh-rsa AzAAB3NzaC1yc2EaaaabiWaaaieaX9AyNR7xWnW0eI3x2NGXrJ4gkQpK/EqpkveGCvvbM \
  oH84zqu3Us8jSaQD392JZAEAhGSoe0dWMBFm9Y41VGZYmncwkfTQPFH1P07vDw49aTAa2RJNFyV \
  QANZCbSocDeuT0Q7usuUj/v8h27+PqsUUl9XVQSDIhXBkWV+bJawc1c= Steve's key

The OpenSSH server consults the keys in the file $HOME/.ssh/authorized_keys in the one-line format, and it grants access to the holder of the private key, all without a password. Note: older OpenSSH installations look in the file authorized_keys2 for version 2 keys, which is all we're considering here.

For VShell, the entire (small) public key file is placed into the per-user public-key directory, and it can contain as many public keys as desired.

Transferring the public-key files can be done in several ways, including cut-and-paste, file transfer, or emailing it to the system's administrator with a request to install the key.

But, when SecureCRT is connected to a SSH server supporting the Secure Shell Public-Key Subsystem (VShell, or OpenSSH with the Secure Shell Public-Key Subsystem patch), it's even easier. The Public Key Assistant... (from the Tools menu) provides a mechanism to transfer a public key automatically. It could hardly be easier.

However the public-key files are generated and placed on the remote system, it's a one-time operation; after that, the keys "just work". And now we'll look at the mechanism which takes place to provide this access.

1	The user makes an initial connection and sends a username along with a request to use a key.
2	The SSH daemon (sshd) on the server looks in the user's authorized_keys file, constructs a challenge based on the public key found there, and sends this challenge back to SecureCRT.
3	SecureCRT receives the key challenge. It finds the user's private key on the local system, but it's protected by an encrypting passphrase. SecureCRT uses "Identity" and "Identity.pub" as filenames for the private and public key files, respectively, regardless of key type. The user is free to rename these files as the last step in the key-generation wizard. Other clients or keytypes use different filename formats. An RSA key file is named id_rsa on OpenSSH.
4	SecureCRT prompts the user for the passphrase, and the comment on the private key is displayed to remind the user which key is requested.
5	SecureCRT uses the private key to construct a key response, and sends it to the waiting sshd on the other end of the connection. It does not send the private key itself!
6	The sshd validates the key response, and if valid, grants access to the system.

This process involves more steps behind the scenes, but the user experience is mostly the same; you're prompted for a passphrase rather than a password. But, unlike setting up access to multiple computer systems (each of which may have a different password), using public-key authentication means you type the same passphrase no matter which system you're connecting to.

This has a substantial, but non-obvious, security benefit: since you're now responsible for just one secret phrase instead of many passwords, you type it more often. This makes you more likely to remember it, and therefore, pick a stronger passphrase to protect the private key than you otherwise might.

Trying to remember many separate passwords for different remote systems is difficult, and does not lend itself to picking strong ones. Public key authentication solves this problem entirely.

We'll note that though public-key accounts can't generally be cracked remotely, the mere installation of a public key on a target system does not disable the use of passwords system wide. Instead, the server must be explicitly configured to allow only public-key encryption by the use of the PasswordAuthentication no keyword in the sshd_config file.

Public Key Authentication with Agent Support

Now that we've taken the leap into public-key authentication, we'll take the next step to enable agent support. In the previous section, the user's private key was unlocked at every connection request; this is not functionally different from typing a password, and though it's the same passphrase every time (which makes it easier to remember), it nevertheless gets tedious in the same way.

Fortunately, most Secure Shell applications provide a broker known as a "key agent", which can hold and manage private keys on your workstations and respond to requests from remote systems to verify your keys. Agents provide a tremendous productivity benefit, because once you've unlocked your private key (one time when you launch the agent), subsequent authentication works with the agent without prompting.

This works much like the key authentication seen previously, but with a twist.

1	The user makes an initial connection and sends a username along with a request to use a key.
2	The SSH daemon (sshd) on the server looks [1] in the user's authorized_keys file, constructs a challenge based on the key, and sends it [2] back to SecureCRT.
3	SecureCRT receives the key challenge, and forwards it to the waiting agent. The agent, though part of the SecureCRT process, is nevertheless a separate entity. The agent opens the user's private key and discovers that it's protected by a passphrase.
4	SecureCRT prompts the user for the passphrase to unlock the private key, just as with standalone key support.
5	The agent constructs the key response and hands it back [1] to SecureCRT, which sends it off [2] to the sshd waiting on the other end. Unlike the previous example, SecureCRT never sees the private key directly, only the key response.
6	The sshd validates the key response, and if valid, grants access to the system. Note: the agent still retains the private keys in memory, though it's not participating in the ongoing conversation.

As far as the user is concerned, this first exchange is little different from key authentication shown in the previous section the only difference is which program prompts for the private key (the SSH client itself versus the agent). With SecureCRT, the passphrase prompts are identical.

But, where agent support shines is at the next connection request made while the agent is still resident. Since it remembers the private keys from the first time it was unlocked with the passphrase, it's able to respond to the key challenge immediately without prompting. The user sees an immediate, direct login without having to type anything.

Many users only unlock their private keys once in the morning when they launch their SSH client and agent, and they don't have to enter it again for the rest of the day because the resident agent is handling all the key challenges. It's wonderfully convenient, as well as secure.

It's very important to understand that private keys never leave the agent; instead, the clients ask the agent to perform a computation based on the key, and it's done in a way which allows the agent to prove that it has the private key without having to divulge the key itself. We discuss the challenge/response in a later section.

Once agent support is enabled, all prompting has now been bypassed, and one can consider performing scripted updates of remote systems. This contrived example queries several systems for disk space.

# scripted query of several remote systems

for svr in server1 server2 server3 server4
do
	vsh $svr df           # ask about disk space
done

Without agent support, each server would prompt for a password or passphrase. With agent support, there is no prompting at all.

However, these benefits only accrue to outbound connections made from the local system to SSH servers elsewhere: once logged into a remote server, connecting from there to yet a third server requires either password authentication, or setting up the user's private key on the intermediate system to pass to the third.

Having agent support on the local system is certainly an improvement, but many of us working remotely often must copy files from one remote system to another. Without installing and initializing an agent on the first remote system, the secure file transfer operation will require a password or passphrase every time. In a sense, this just pushes the tedium back one link down the SSH chain.

Fortunately, there's a great solution which solves all these issues.

Public-Key Authentication With Agent Forwarding

With our key agent in place, it's time to enable the final piece of our puzzle: agent forwarding. In short, this allows a chain of SSH connections to forward key challenges back to the original agent, obviating the need for passwords or private keys on any of the intermediate machines.

1	This all starts with an already established connection to the first server, with the agent now holding the user's private key. The second server plays no part yet.
2	The user launches the SSH command-line client on the first server with a request to connect to server2, and this passes the username and a use-key request to the sshd. This could be done with the command-line clients as well.
3	The sshd searches the the user's public key folder for a matching public key or authorized_keys file [1], constructs a key challenge from the key, and sends it [2] back down the channel to the client which made the request.
4	This is where the magic occurs; SecureCRT on server receives the key challenge from the target system, and it forwards [1] that challenge to the sshd server on the same machine acting as a key agent. The sshd in turn relays [2] the key challenge down the first connection to the original SecureCRT. Once back on homepc, SecureCRT takes the final step in the relay process by handing the key challenge off [3] to the resident agent, which knows about the user's private key.
5	The agent running on homepc constructs the key response and hands it [1] back to SecureCRT, which in turn passes it [2] down the channel to the sshd running on server. Since sshd is acting as a key agent, it forwards [3] the key response off to the requesting SSH client, which sends it [4] to the waiting sshd on the target system (server2). This forwarding action is all done automatically and near instantly.
6	The sshd on server validates the key response, and if valid, grants access to the system.

This process can be repeated with even more links in the chain (say, if the user wanted to SSH from server2 to server3), and it all happens automatically. Agent authentication is available with most Secure Shell utilities including ssh and sftp (secure file transfer). VanDyke Software's command-line utilities (vcp, vsftp, and vsh) participate as well.

Agent Forwarding

Pro:	exceptional convenience
Con:	requires installation of public keys on all target systems
Con:	requires a Tech Tip to understand
Pro:	an excellent Tech Tip is available

This does require the one-time installation of the user's public – not private! – keys on all the target machines, but this setup cost is rapidly recouped by the added productivity provided. Those who use public keys with agent forwarding rarely go back.

Configuring VShell Authentication

One of the difficulties with forbidding password authentication is installation of public keys for new users; since they cannot login without a key, we're in a chicken-and-egg situation with respect to their own ability to set up their own accounts. This usually means that the administrator has to do it, and this is two interruptions for each new user rather than just one. This doesn't scale so well for larger enterprises.

VShell provides a clever solution to this: even with password authentication disabled, it can be configured to permit a small number – usually three – of password attempts before it reverts to key-only mode. This gives the user a couple of tries to login with a password to install a public key before the key is the only way to get in.

Password attempts are maintained on a per-user basis, so allowing the new users to make a go of keyless logins doesn't open this door for the existing accounts.

Configuring this requires coordination of a few settings, but it's easy to do:

1. Open the VShell setup applet and navigate to the Authentication category. Uncheck both Allow and Require settings for Password; this disables password authentication entirely for the user base as a whole, requiring other, more secure mechanisms to get in.
2. Check the Allow [3] password attempts check box; this grants a limited exception to the no-password provision in step #1 just for this user. Tune the number of attempts to taste.

It's inevitable that some users will have problems logging in even within the allotted attempts, and this will certainly require administrative intervention, but for the majority of users it means self-service installation of public keys.

How Key Challenges Work

One of the more clever aspects of the agent is how it can verify a user's identity (or more precisely, possession of a private key) without revealing that private key to anybody. This, like so many other things in modern secure communications, uses public-key encryption.

When a user wishes access to an SSH server, he presents his username to the server with a request to set up a key session. This username helps locate the list of public keys allowed access to that server; with VShell, it's found in the public key folder assigned to that user, or with OpenSSH it's typcially found in the $HOME/.ssh/authorized_keys file.

The server creates a "challenge" which can only be answered by one in possession of the corresponding private key; it creates and remembers a large random number, then encrypts it with the user's public key. This creates a buffer of binary data which is sent to the user requesting access. To anybody without the private key, it's just a pile of bits.

When the agent receives the challenge, it decrypts it with the private key. If this key is the "other half" of the public key on the server, the decryption will be successful, revealing the original random number generated by the server. Only the holder of the private key could ever extract this random number, so this constitutes proof that the user is the holder of the private key.

The agent takes this random number, appends the SSH session ID (which varies from connection to connection), and creates an MD5 hash value of the resultant string: this result is sent back to the server as the key response.

The server computes the same MD5 hash (random number + session ID) and compares it with the key response from the agent; if they match, the user must have been in possession of the private key, and access is granted. If not, the next key in the list (of any) is tried in succession until a valid key is found, or no more authorized keys are available. At that point, access is denied.

Curiously, the actual random number is never exposed in the client/agent exchange — it's sent encrypted to the agent, and included in an MD5 hash from the agent. It's likely that this is a security precaution designed to make it harder to characterize the properties of the random number generator on the server by looking at the client/agent exchange.

More information on MD5 hashes can be found in An Illustrated Guide to Cryptographic Hashes.

Security Issues With Key Agents

One of the security benefits of agent forwarding is that the user's private key never appears on remote systems or on the wire, even in encrypted form. But, the same agent protocol which shields the private key may nevertheless expose a different vulnerability: agent hijacking.

Each SSH implementation has to provide a mechanism for clients to request agent services, and on UNIX/Linux this is typically done with a UNIX domain socket stored under the /tmp/ directory. On our Linux system running OpenSSH, for instance, we find the file /tmp/ssh-CXkd6094/agent.6094 associated with the SSH daemon servicing a SecureCRT remote client.

This socket file is as heavily protected as the operating system allows (restricted to just the user running the process, kept in a protected subdirectory), but nothing can really prevent a root user from accessing any file anywhere.

If a root user is able to convince his SSH client to use another user's agent, root can impersonate that user on any remote system which authorizes the victim user's public key. Of course, root can do this on the local system as well, but he can do this directly anyway without having to resort to SSH tricks.

Several environment variables are used to point a client to an agent, but only SSH_AUTH_SOCK is required in order to use agent forwarding. Setting this variable to a victim's agent socket allows full use of that socket if the underlying file is readable. For root, it always is.

# ls -l /tmp/ssh*      – look for somebody's agent socket
/tmp/ssh-CXkd6094:
total 24
srwxr-xr-x    1 steve    steve           0 Aug 30 08:46 agent.6094=

# export SSH_AUTH_SOCK=/tmp/ssh-CXkd6094/agent.6094

# ssh steve@remotesystem

remote$                  – Gotcha! Logged in as "steve" user on remote system!

One cannot tell just from looking at the socket information which remote systems will accept the user's key, but it doesn't take too much detective work to track it down. Running the ps command periodically on the local system may show the user running SSH remotesystem, and the netstat command may well point to the user's home base.

Furthermore, the user's $HOME/.ssh/known_hosts file contains a list of machines which the user has a connection: though they may not all be configured to trust the user's key, it's certainly a great place to start looking. Modern versions (4.0 and later) of OpenSSH can optionally hash the known_hosts file to forestall this.

There is no technical method which will prevent a root user from hijacking an SSH agent socket if he has the ability to access it, so this suggests that agent forwarding might not be such a good idea when the remote system cannot be entirely trusted. All SSH clients provide a method to disable agent forwarding.

The Command-Line Utilities

Most users spend the bulk of their time in SecureCRT terminal windows, but SecureCRT also includes a suite of standalone command-line utilities. These utilities work and play well with all the other parts, including the key agents.

vcp.exe

This is the secure file transfer program, and it's able to copy files to and from remote systems from the command line. Though it appears to work like the UNIX scp, the UNIX version uses the rcp/remote execute mechanism while vcp uses the SFTP protocol. It's able to fully use any resident key agents.

C:\Temp> vcp file*.jpg steve@unixwiz.net:/home/www/htdocs/images
Processing directory: .
ls: .\file1.jpg
ls: .\file2.jpg
ls: .\file3.jpg
Copying: .\file3.jpg
  100% 57KB     57KB/s 00:00:01
Copying: .\file2.jpg
  100% 327KB     40KB/s 00:00:08
Copying: .\file1.jpg
   62% 216KB     43KB/s 00:00:02 ETA   – still in progress; shows estimated time

Copy operations can go in either direction, subject to the rights granted by the remote server.

vsh.exe
This is a full-strength SSH client, providing remote login and remote command execution services from the command line; in this respect it's very much like the OpenSSH ssh command.

vsftp.exe
This provides an FTP-like front end to the Secure Shell File Transfer protocol, and it's easier for many users to navigate than the low-level vcp operations (users wishing a GUI interface should consider SecureFX).

Each takes a similar set of command-line parameters which direct most of the important behaviors, and this makes them all highly amenable to scripting.

Of particular utility in the scripting arena is the ability to provide a specific private key file. It's common to create a separate key which provides only limited powers on the target server (for instance, it could allow the user to connect, but only to run a single, specified command via vsh).

Since scripts often run from Scheduled Tasks (and do not allow user interaction), all the credentials necessary for remote access — the username, the key, and the passphrase must be provided with the script. This is not terribly secure, so by creating a key configured with purpose-built limited powers, compromise of the credentials by those with access to the machine doesn't provide unrestricted access to the remote system.

Additional Resources

Up to this point, we've provided essentially no practical how-to information on installing or configuring any particular SSH implementation. Our feeling is that this information is covered better elsewhere, and we're happy to provide some links here to those other resources.

The Secure Shell: The Definitive Guide