Suppose your company has a gateway host, G, which is your only gateway to the Internet. You are logged into a client host, C, and want to reach a server host, S, outside the company network, as shown in Figure 11-11. We assume that all three machines have SSH installed.

Figure 11-11. Proxy gateway

To make a connection from client C to server S now requires two steps:

This works, and using agent forwarding and public-key authentication on both hosts, you can avoid a second authentication prompt on gateway G.

Now, an obvious simplification would be this single command:

    $ ssh G ssh S

If you do this, though, you’ll have a problem: first, you’ll see this warning:

    Pseudo-terminal will not be allocated because stdin is not a terminal.

...and next, your shell on S will behave very strangely! You won’t get a prompt, or any fancy line-editing—but if you persist and enter some commands, they will get executed. The problem is that ssh only creates a pseudo-terminal, needed for interactive terminal-based programs, if you explicitly request a remote shell session; to ssh, this means that you do not specify a remote program to run. By default, it assigns no terminal when running remote commands like ssh host uname -a. Most of the time this is a reasonable default, but sometimes you’ll run a remote command that actually needs a terminal—in this case, the shell! You can fix this with the -t switch for force a pseudo-terminal:

    $ ssh -t G ssh S

But this introduces yet another messy aspect: to reach hosts through the gateway, you not only have to use double-ssh commands, but furthermore, specify -t in some cases but not others. Not a big burden for occasional use, perhaps, but cumbersome if large numbers of hosts or automation are involved.

Fortunately, SSH configuration is flexible enough to afford a neat solution, which we now present using OpenSSH features and syntax.^[149] We use public-key authentication to take advantage of the options of the authorized_keys file, and ssh-agent with agent forwarding so that authentication passes on transparently to the second SSH connection (see Figure 11-12).

Figure 11-12. Chained SSH connections through a proxy gateway

Suppose your account on gateway G is gilligan, and on server S it is skipper. First, set up your SSH client configuration file so that the name S is a nickname for accessing your account on gateway G:

    # ~/.ssh/config on client C

    host S
      hostname G
      user gilligan

Next, on gateway G, associate a forced command with your chosen key to invoke an SSH connection to server S: [8.2.3]

     # ~/.ssh/authorized_keys on gateway G
    command="ssh -l skipper S" ...key..

Now, when you invoke the command ssh S on client C, it connects to gateway G, runs the forced command automatically, and establishes a second SSH session to server S. And thanks to agent forwarding, authentication from G to S happens automatically, assuming you’ve loaded the appropriate key. This can be the same key you used to access gilligan@G or a different one.^[150]

This trick not only provides a transparent connection from client C to server S, it also sidesteps the fact that the name S might not have any meaning on client C. Often in this kind of network situation, your internal network naming scheme is cut off from the outside world (e.g., split DNS with internal roots). After all, what’s the point of allowing you to name hosts you can’t reach? Thanks to the Host configuration keyword for SSH clients, you can create a nickname S that instructs SSH to reach that host transparently via G. [7.1.2.5]

You’ll soon notice a problem, though. Interactive logins work fine, but remote commands are ignored! And worse, the missing terminal problem rears its head again:

    $ ssh S echo Hello
    Pseudo-terminal will not be allocated because stdin is not a terminal.

You’re left talking to a mute shell, and no “Hello” appears. The problem now is that we’ve done nothing to pass along any remote command to S; the forced command on G simply ignores it and always tries to start a remote-login SSH connection (hence provoking the missing terminal problem, as before). We can fix this using another OpenSSH feature:^[151]

    command="ssh -l skipper S $SSH_ORIGINAL_COMMAND" ...key...

If a remote command is used, sshd stores it in the environment variable SSH_ORIGINAL_COMMAND; we use that here to pass it along to the next ssh command. The variable is not set, however, if there is no remote command. Some shells consider this an error, so you might have to augment this in some way to accommodate the shell’s predilections. For example, some shells have this syntax:

    command="ssh -l skipper S ${SSH_ORIGINAL_COMMAND:-}" ...key...

where ${foo:-bar} evaluates to “bar” if the variable foo is not set. And remember, the shell used here is the one belonging to the remote account; to be especially robust, it might be best to use a particular shell explicitly:

    command="/bin/bash -c 'ssh -l skipper S ${SSH_ORIGINAL_COMMAND:-}'" ...key...

This technique also neatly solves the “missing terminal” problem at the same time!

Recall that the command:

    $ scp ... S:file ...

actually runs ssh in a subprocess to connect to S and invoke a remote scp server. [3.7] Now that we’ve gotten ssh working from client C to server S, you’d expect that scp would work between these machines with no further effort. Well, it almost does, but it wouldn’t be software if there weren’t a small problem to work around, in this case authentication. You can’t provide a password or passphrase to the second ssh program, since there is no pseudo-terminal on the first ssh session--ssh requires a terminal for user input. So, you need a form of authentication that doesn’t require user input: either hostbased, or public-key authentication with agent forwarding. Hostbased works as is, so if you plan to use it, you can skip to the next section. Public-key authentication, however, may have a problem: some versions of scp run ssh with the -a switch to disable agent forwarding. [6.3.5.3] You need to reenable agent forwarding for this to work, and this is surprisingly tricky.

Normally, you could turn on agent forwarding in your client configuration file:

    # ~/.ssh/config on client C, but this FAILS
    ForwardAgent yes

but this doesn’t help because as it happens, the -a on the command line takes precedence. Alternatively, you might try the -o option of scp, which can pass along options to ssh, such as -o ForwardAgent yes. But in this case, scp places the -a after any -o options it passes where it takes precedence, so that doesn’t work either.

There is a solution, though. scp has a -S option to indicate a path to the SSH client program it should use, so you create a “wrapper” script that tweaks the SSH command line as needed, and then make scp use it with -S. Place the following script in an executable file on client C—say, ~/bin/ssh-wrapper:

    #!/usr/bin/perl
    exec '/usr/bin/ssh', map {$_ eq '-a' ? () : $_} @ARGV;

This runs the real ssh, removing -a from the command line if it’s there. Now, give your scp a command like this:

    scp -S ~/bin/ssh-wrapper ...S:file ...

and it should work.

Instead of using a forced command, here’s another way to connect by SSH through a gateway: forward a port on client C to the SSH server on S, using an SSH session from C to G, and then run a second SSH session through the first (see Figure 11-13).

Figure 11-13. Forwarded SSH connection through a proxy gateway

That is:

    # Execute on client C
    $ ssh -L2001:S:22 G

    # Execute on client C in a different shell
    $ ssh -p 2001 -o HostKeyAlias=S localhost

This connects to server S by carrying the second SSH connection (from C to S) inside a port-forwarding channel of the first (from C to G ). Note the use of HostKeyAlias, so ssh will look up S’s host key with the name “S.” Otherwise, it would try to use the key for “localhost,” which would be the wrong key.

You can make this more transparent by creating a nickname S in your client configuration file:

    # ~/.ssh/config on client C
    Host S
      Hostname localhost
      Port 2001
      HostKeyAlias S

Now the earlier commands become:

    # Execute on client C
    $ ssh -L2001:S:22 G

    # Execute on client C in a different shell
    $ ssh S

Because this technique requires a separate, manual step to establish the port forwarding, it is less transparent than the one in [11.4.1]. However, it has some advantages. If you plan to use port or X forwarding between C and S with the first method, it’s a little complicated. scp not only gives the -a switch to ssh to turn off agent forwarding, but also it gives -x and -o “ClearAllForwardings yes”, turning off X and port forwarding. So, you need to modify the earlier wrapper script to remove these unwanted options as well. [11.4.2] Then, for port forwarding you need to set up a chain of forwarded ports that connect to one another. For example, to forward port 2017 on client C to port 143 (the IMAP port) on server S:

    # ~/.ssh/config on client C
    host S
      hostname G
      user gilligan

    # ~/.ssh/authorized_keys on gateway G
    command="ssh -L1234:localhost:143 skipper@S" ...key...

    # Execute on client C
    $ ssh -L2017:localhost:1234 S

This works, but it’s difficult to understand, error-prone, and fragile: if you trigger the TIME_WAIT problem [9.2.9.1], you have to edit files and redo the tunnel just to pick a new ephemeral port to replace 1234.

Using the SSH-in-SSH technique instead, your port and X-forwarding options operate directly between client C and server S in the usual, straightforward manner. The preceding example becomes:

    # ~/.ssh/config on client C
    Host S
      Hostname localhost
      Port 2001
      HostKeyAlias S

    # Execute on client C
    $ ssh -L2001:S:22 G

    # Execute on client C in a different shell
    $ ssh -L2017:localhost:143 S

This final command connects to server S, forwarding local port 2017 to the IMAP port on S.

Here’s yet another way to implement the tunneled SSH technique:

    # ~/.ssh/config on client C
    Host S
      ProxyCommand "ssh -qax G nc S 22"

If a ProxyCommand value is set, OpenSSH uses this command to get a communication channel to the remote host, rather than using the network directly. The command, in turn, can do anything at all—it could connect to an SSH server at the other end of a serial line, for example! In this case, we actually use a second ssh command to connect through gateway G to the SSH server TCP port on server S. The trick is that we really want a kind of connection that OpenSSH doesn’t provide. Ideally, we’d like to be able to say something like ssh --tcp S:22 G (note: this syntax does not currently exist), the meaning of which would be: “connect to G via SSH, instruct G to make a TCP connection to host S port 22, and connect the local stdin/stdout to that stream.” Making remote TCP connections is already something sshd can do; that’s how local TCP forwarding is done. Unfortunately, no SSH clients we know of provide this useful feature. So, we must have a separate program on G, which just makes a simple TCP connection for us; here, we use netcat (nc).

Expanding on this a bit...instead of a single host on the other side of the gateway, suppose you have many you want to access. If their names follow a pattern, you may be able to express this behavior very succinctly using OpenSSH. Suppose the machines in question are a cluster with hostnames beowulf-1, beowulf-1, etc. Then you can use this:

    # ~/.ssh/config on client C
    Host beowulf-*
      ProxyCommand "ssh -qax G nc %h %p"

This Host directive will match any of the cluster hostnames, and use an ssh subprocess to reach the host in question through the gateway: OpenSSH substitutes the %h and %p in the ProxyCommand with the host and port to use.

The ProxyCommand technique is simpler than port forwarding: there’s no extra SSH command to start separately and no ad hoc port numbers to coordinate and possibly have to change. It also gains in security, since port forwarding always has the problem of unauthenticated access to the forwarded connection. And, we need no HostKeyAlias statements. However, we lose the speed advantage gained over chained ssh commands, since once again we end up waiting for two SSH connections every time. A compromise approach would be to use the ProxyCommand method together with an OpenSSH connection server. [7.4.4.2]

We’ve presented several methods of SSH access through a gateway. There are various trade-offs, but overall we think tunneling is usually the best way to go. Here’s why.