In our earlier example, we had an IMAP server running on machine S, and an email reader on home machine H, and we wanted to secure the IMAP connection using SSH. Let’s delve into that example in more detail.

IMAP uses TCP port 143; this means that an IMAP server listens for connections on port 143 on the server machine. To tunnel the IMAP connection through SSH, we need to pick a local port on home machine H (between 1024 and 65535) and forward it to the remote socket (S,143). Suppose you randomly pick local port 2001. The following command then creates the tunnel:^[130]

    $ ssh -L2001:localhost:143 S

The -L option specifies local forwarding, in which the TCP client is on the local machine with the SSH client. The option is followed by three values separated by colons: a local port to listen on (2001), the remote machine name or IP address (S), and the remote, target port number (143).

The previous command logs you into S, just like ssh S does. However, this SSH session has also forwarded TCP port 2001 on H to port 143 on S; the forwarding remains in effect until you log out of the session. To make use of the tunnel, the final step is to tell your email reader to use the forwarded port. Normally, your email program connects to port 143 on the server machine—that is, the socket (S,143). Instead, it’s configured to connect to port 2001 on home machine H itself, i.e., socket (localhost,2001). So the path of the connection follows the list shown next.

Port forwarding can be specified only when you create an SSH connection. You can’t add a forwarding to an existing SSH connection with any SSH implementation we know of, though there’s nothing intrinsic to the SSH protocol that would prevent it, and it would sometimes be a useful feature. Instead of using the -L option to establish a local forwarding, you can use the LocalForward keyword in your client configuration file:

    #  OpenSSH
    LocalForward 2001 localhost:143
    # Tectia
    LocalForward "2001:localhost:143"

Note the small syntactic differences. In OpenSSH, there are two arguments: the local port number, and the remote socket expressed as host:port. In Tectia, the expression is just as on the command line, except that it must be enclosed in double quotes. If you forget the quotes, ssh doesn’t complain, but it doesn’t forward the port, either.

Our example with home machine H and IMAP server S can be set up like this:

    #  OpenSSH
    Host local-forwarding-example
     HostName S
     LocalForward 2001 localhost:143

    # Run on home machine H
    $ ssh local-forwarding-example

    # OpenSSH
    $ ssh -g -L<localport>:<remotehost>:<remoteport> hostname

    #  OpenSSH
    GatewayPorts yes

    $ ssh -R2001:localhost:143 H

    # OpenSSH
    RemoteForward 2001 S:143

    # Tectia
    RemoteForward "2001:S:143"

    # Tectia
    remote-forwarding-example:
     Host H
     RemoteForward "2001:S:143"

    $ ssh remote-forwarding-example

If you use LocalForward or RemoteForward in your configuration file, you might run into a subtle problem. Suppose you have set up a section in your configuration file to forward local port 2001 to an IMAP server:

    # OpenSSH  syntax used for illustration
    Host server.example.com
     LocalForward 2001 server.example.com:143

This configuration works fine if you connect once:

    $ ssh server.example.com

But if you try to open a second ssh connection to server.example.com at the same time—perhaps to run a different program in another window of your workstation—the attempt fails:

    $ ssh server.example.com
    Local: bind: Address already in use

Why does this happen? Because your configuration file section tries to forward port 2001 again but finds that port is already in use (“bound” for listening) by the first instance of ssh. You need some way to make the connection but omit the port forwarding.

OpenSSH provides a solution, the client configuration keyword ClearAllForwardings. From the name, you might think it terminates existing forwardings, but it doesn’t. Rather, it nullifies any forwardings specified in the current ssh command. In the previous example, you can connect without forwardings to server.example.com with:

    # OpenSSH
    $ ssh -o ClearAllForwardings=yes server.example.com

The original tunnel, set up by the first invocation, continues to exist, but ClearAllForwardings prevents the second invocation from attempting to re-create the tunnel. To illustrate the point further, here’s a rather silly command:

    # OpenSSH
    $ ssh -L2001:localhost:143 -o ClearAllForwardings=yes mymachine

The -L option specifies a forwarding, but ClearAllForwardings cancels it. This silly command is identical in function to:

    $ ssh mymachine

ClearAllForwardings may also be placed in your client configuration file, of course. It seems more useful on the command line, however, where it can be used on the fly without editing a file.

The differences between local and remote forwarding can be subtle. It can get a bit confusing to know which kind of forwarding to use in a given situation. The quick rule is look for the TCP client application.

The rest of this section is devoted to dissecting the forwarding process in detail and understanding where this rule comes from.

9.2.3.1 Common elements

Local and remote forwarding can be confusing because of overloaded terminology. In a given port-forwarding situation, there are two clients and two servers lying around. We have the SSH client and server programs (e.g., ssh and sshd), plus the TCP application’s client and server programs whose connection you want to protect by port forwarding.

An SSH session has a direction of establishment. That is, you run an SSH client on one machine, and it initiates a session with an SSH server on another. Likewise, a forwarded connection has a direction of establishment: you run an application client on one machine, and it initiates a session with a service on another. These two directions may or may not match. This is the difference between local and remote forwarding. Let’s introduce some terminology and provide some diagrams to make sense of this.

To begin with, we have an application client and server running on two hosts, A and B (Figure 9-2).

The application server is listening on a well-known port W for incoming client connections. Without SSH, you can tell the application client that its server is on host B, port W. The client makes a direct connection to the server, and all application protocol data goes in the clear over the network (Figure 9-3).

Figure 9-2. Application client and server

Figure 9-3. Direct client/server connection (no forwarding)

To protect the application protocol data by forwarding, you establish an SSH session between these two hosts. When setting up the SSH session, you select an unused port number P on the application client side (host A), and request SSH port forwarding from the socket (A,P) to the socket (B,W). Once the session is established, the SSH process on A is listening for incoming TCP connection requests on port P. Tell the application client that its server is on (A,P) instead of (B,W), and the stage is now set for port forwarding (Figure 9-4).

There are now two cooperating SSH processes with an established, encrypted SSH session between them; you don’t distinguish between the SSH client and server. Inside that session, SSH creates multiple channels, or logical streams for carrying data. It uses channels to carry and distinguish the input, output, and error streams for an interactive login or remote command run via SSH, and similarly creates a new channel for each use of a port forwarding, to carry the forwarded data inside the protected SSH session.

Figure 9-4. A forwarded port

Figure 9-5 shows that now, when the application client tries to connect to its server, it connects instead to the listening SSH process (1). The SSH listener notices this and accepts the connection. It then notifies its partner SSH process that a new instance of this port forwarding is starting up, and they cooperate to establish a new channel for carrying the data for this forwarding instance (2). Finally, the partner SSH process initiates a TCP connection to the target of the port forwarding: the application server listening on (B,W) (3). Once this connection succeeds, the port-forwarding instance is in place. The SSH processes cooperate to pass back and forth any data transmitted by the application client and server, over the channel inside the SSH session. This allows them to communicate and secures the application’s activities on the network.

Figure 9-5. A forwarded connection

9.2.3.2 Local versus remote forwarding: the distinction

With this general framework in place, you can distinguish between local and remote forwarding. First we introduce some terms. In the generic port-forwarding description in the last section, you saw that one SSH process listens for connections, while the other is ready to initiate connections in response to connections accepted on the other side, to complete the forwarded path. We call the first side the listening side of the SSH session with respect to this forwarding, and the other, the connecting side. For example, in Figure 9-4, host A is the listening side, while host B is the connecting side. Note that these terms aren’t mutually exclusive. Since a single SSH session may have multiple forwardings in place, the same side of a session may be the listening side for some forwardings, and simultaneously the connecting side for others. But with respect to any particular forwarding, it’s one or the other.

Now, recall that in the last section we didn’t label the SSH processes according to which was the SSH client and which was the SSH server, but simply referred to two cooperating SSH processes. We do so now and can state succinctly the local versus remote distinction:

• In a local forwarding (Figure 9-6), the application client and hence the listening side are located with the SSH client. The application server and connecting side are located with the SSH server.

• In a remote forwarding (Figure 9-7), the situation is reversed: the application client and listening side are located with the SSH server, while the application server and connecting side are located with the SSH client.

Figure 9-6. Local forwarding

So, as we said at the beginning of this section: use a local forwarding when the application client is on the local side of the SSH connection, and a remote forwarding when it’s on the remote side.

In all our discussions of port forwarding so far, the application client and server have been located on the machines on the ends of the SSH session. This is reflected in our always using “localhost” in naming the target socket of a forwarding:

    $ ssh -L2001:localhost:143 server.example.com

Figure 9-7. Remote forwarding

Since the application server is located on the same machine as the connecting side of the SSH port forwarding, the target host can be “localhost.” But the connections between the application client and the SSH listening side, and between the application server and the SSH connecting side, are themselves TCP connections. For convenience, TCP implementations allow programs to make connections between two sockets on the same host. The connection data is simply transferred from one process to another without actually being transmitted on any real network interface. However, in principle, either the application client or server—or both—could be on different machines, potentially involving as many as four hosts in a single forwarding (Figure 9-8).

Figure 9-8. Off-host port forwarding

Although this situation is possible, you generally don’t want to do it for security reasons—namely, privacy and access control.

    $ ifconfig -a
    ...
    lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST> mtu 8232
            inet 127.0.0.1 netmask ff000000

    $ ssh -g -L P:S:W B

Let’s tackle a more complicated example of port forwarding. Figure 9-9 returns us to the same company situation as in Figure 6-5, when we discussed agent forwarding. [6.3.5] Your home machine H talks to work machine W via a bastion host, B, and you want to access your work email from home. Machine W runs an IMAP server, and your home machine H has an IMAP-capable email reader, but you can’t hook them up. Your home IMAP client expects to make a TCP connection directly to the IMAP server on W, but unfortunately that connection is blocked by the firewall. Since host B is inside the firewall, and it’s running an SSH server, there should be some way to put all the pieces together and make the IMAP connection from H to W.

Figure 9-9. Port forwarding through a firewall

Port forwarding can solve this problem. As before, the IMAP server is on port 143, and we select a random local port number, 2001. This time, however, we use a slightly different command to set up forwarding:

    # Executed on home machine H
    $ ssh -L2001:W:143 B

This establishes an interactive SSH session from home machine H to bastion host B and also creates an SSH tunnel from local host H to the email server machine W. Specifically, in response to a connection on port 2001, the local SSH client directs the SSH server running on B to open a connection to port 143 on W, that is, socket W:143. The SSH server can do this because B is inside the firewall. If you configure your email reader to connect to local port 2001, as before, the communication path is now as follows:

You have now bypassed the firewall by tunneling the IMAP traffic through SSH.

It may happen that you’d like to forward a port via SSH but don’t want an SSH login session to the remote host. For example, if you’re using the IMAP forwarding example we’ve been harping on, you may want only to read email, not open an unnecessary terminal connection at the same time. With Tectia, this is simple: just provide the -f option to ssh in your port-forwarding command:

    # Tectia
    $ ssh -f -L2001:localhost:143 server.example.com

or use the GoBackground keyword for the same effect:

    # Tectia
    GoBackground yes

As a result, ssh puts itself into the background and handles connections to the forwarded port 2001, and that is all. It doesn’t create an interactive terminal session with standard input, output, and error channels. The -S option also avoids starting a terminal session, but unlike -f, it doesn’t put the session in the background (in other words, the -f option implies -S):

    # Tectia
    $ ssh -S -L2001:localhost:143 server.example.com

The -f option is also supported by OpenSSH, but by default it still requires a command to execute. This usage is intended more for executing remote commands that don’t require terminal interaction, such as graphical programs using X. Specifically, it causes the backgrounded ssh to connect the local end of the terminal session to /dev/null (that is, -f implies the -n option).

For example, if X forwarding is turned on (which we’ll discuss later), the following command puts itself into the background, popping up a graphical clock on your local display, with the clock program running on the remote host zwei.uhr.org:

    #  OpenSSH
    $ ssh -f zwei.uhr.org xclock

This is similar to the background command:

    #  OpenSSH
    $ ssh -n zwei.uhr.org xclock &

but -f is better because it performs any needed user interaction—like prompting for a password—before forking into the background. If you want to background an OpenSSH session without a remote command, as with Tectia earlier, then add the -N switch as well:

    $ ssh -fN -L2001:localhost:143 server.example.com

Technically, this means the client will not create a “shell channel” in the SSH protocol. Tectia doesn’t require the extra option, it just does the right thing whether you give a remote command or not; with OpenSSH, you must use the -N option if you don’t provide a command. If you forget the option, you’ll see:

    #  OpenSSH
    $ ssh -f -L2001:localhost:143 server.example.com
    Cannot fork into background without a command to execute.

The old SSH-1 protocol always requires the remote command, so as a workaround, provide one that does nothing for a long time, such as sleep:

    #  An SSH-1 client
    $ ssh -f -L2001:localhost:143 server.example.com sleep 1000000

    # Tectia
    $ ssh -fo -L2001:localhost:143 server

    Waiting for forwarded connections to terminate...
    The following connections are open:
      port 2001, connection from localhost port 143

Earlier, we suggested selecting any unused port for the listening side of a forwarding. Port numbers are encoded in a 16-bit field and can have any value from 1 to 65535 (port 0 is reserved). On multiuser operating systems such as Unix, ports 1 through 1023 are called privileged and are reserved for processes run by the superuser (user ID zero). If a nonprivileged process tries to bind a privileged port for listening, it fails with an error message such as “insufficient permission.”^[136]

When setting up the listening side of a tunnel, you generally must select a port number between 1024 and 65535, inclusive. This is because an SSH program running under your user ID, not the superuser’s, is responsible for listening on that port. If SSH reports that your chosen port is already in use, just choose another; it shouldn’t be hard to find a free one.

For the target side of the tunnel, you can specify any port number, privileged or not. You are attempting to connect to the port, not listen on it. In fact, most of the time the target side is a privileged port, since the most common TCP services have ports in the privileged range.

If you are the superuser on a machine with SSH clients, you can perform local forwarding with a privileged port. Likewise, you can forward a remote privileged port if your remote account has superuser privileges.

Some TCP applications hardcode the server port numbers and don’t permit them to be changed. These applications aren’t usable with port forwarding if the operating system has a privileged port restriction. For example, suppose you have an FTP client that’s hardwired to connect to the server on the standard FTP control port, 21. To set up port forwarding, you have to forward the local port 21 to the remote port 21. But since port 21 is privileged, you can’t use it as a listening port number unless you are the superuser. Fortunately, most Unix TCP-based programs let you set the destination port number for connections.

Suppose you want to forward a connection from your local machine to remote.host.net. Both of the following commands work:

    $ ssh -L2001:localhost:143 remote.host.net
    $ ssh -L2001:remote.host.net:143 remote.host.net

The forwarded connection is made from the remote machine to either the loopback address or remote.host.net, and in either case, the connection stays on the remote machine and doesn’t go over the network. However, the two connections are perceptibly different to the server receiving the forwarded connection. This is because the source sockets of the connections are different. The connection to localhost appears to come from source address 127.0.0.1, whereas the connection to remote.host.net is from the address associated with that name.

Most of the time this difference doesn’t matter, but sometimes you must take it into account. The application server (e.g., the IMAP daemon) might be doing access control based on the source address and may not be configured to accept the loopback address. Or it might be running on a multihomed host, and have bound only a subset of the addresses the host has, possibly not including the loopback address. Each of these situations is usually an oversight, but you might not be able to do anything about it. If you’re getting “connection refused” from the connecting side of the forwarding, but you’ve verified that the server appears to be running and responding to normal clients, this might be the problem. If the server machine is running Unix, the command netstat -a -n should list all the network connections and listeners on that machine. Look for listeners on the relevant port, and the addresses on which they are listening.

Sometimes, the problem can be more acute if the server uses the source IP address itself as part of whatever protocol it’s speaking. This problem crops up when trying to forward FTP over SSH. [11.2]

In general, we recommend using localhost as the forwarding target whenever possible. This way, you are less likely to set up an insecure off-host forwarding by accident.

What happens to forwardings when an SSH connection terminates? The ports simply cease being forwarded; that is, SSH is no longer listening on them, and connection attempts to those ports will fail with the error “connection refused.”

What happens if you try to terminate an SSH session while it still has active forwarded connections? SSH notices and waits for them to disconnect before stopping the session. The details of this behavior differ among implementations.

In Tectia, if you log out of a session that has an active forwarded connection, the session stays open but sends itself into the background:

    remote$ logout
    warning: ssh[7021]: number of forwarded channels still open, forked to background to
    wait for completion.
    local$

The ssh process now waits in the background until the forwarded connections terminate, and then it exits. In contrast, with OpenSSH, if you disconnect a session with active forwardings, you get a warning, but the session stays in the foreground:

    remote$ logout
    Waiting for forwarded connections to terminate...
    The following connections are open:
      port 2002, connection from localhost port 1465

To send it into the background and return to your local shell prompt, type the escape sequence Return-tilde-ampersand: [7.4.6.8]

      ~& [backgrounded]
      local$

and as with Tectia, the connection exits only after its forwarded connections terminate. Be careful not to use the SSH ^Z escape for this purpose. That sends ssh into the background, but in a suspended state, unable to accept TCP connections to its forwarded ports. If you do this accidentally, use your shell’s job control commands (e.g., fg and bg) to resume the process.

    $ ssh -L2001:localhost:21 server.example.com
    Local: bind: Address already in use

    $ netstat -an | grep 2001
    tcp    0   0   127.0.0.1:2001   127.0.0.1:1472    TIME_WAIT

We’ve seen several keywords and command-line options for configuring SSH clients for port forwarding, such as -L and -R. In addition, the SSH server can be configured for port forwarding. We’ll cover compile-time, serverwide, and per-account configuration.

    AllowTcpForwarding no

    # Tectia
    AllowTcpForwardingForUsers
    AllowTcpForwardingForGroups

    # OpenSSH
    no-port-forwarding ...key...

    # Tectia
    Key mykey.pub
    Options no-port-forwarding

SSH port forwarding works best with protocols that make simple use of TCP: those which operate over a single TCP connection and are not sensitive to its network-related details such as IP addresses or ports—in other words, they could operate just as well over a serial line or other similar path. Many common protocols fall in this category, but not all. The exceptions tend to be older protocols designed before the rise of firewalls and NAT on the Internet, which degrade true peer-to-peer connectivity and make some techniques problematic. As we have already mentioned, a prime example is FTP, which exhibits several forwarding problems all at once:

This is all pretty disastrous from a forwarding perspective in the presence of NAT, and we’ll share those gory details later. [11.2.6] Some SSH implementations, though, have an FTP-specific forwarding feature designed to work around these problems. This protocol-specific forwarding involves the SSH client watching the tunneled FTP protocol as it operates, creating dynamic forwardings to accommodate it, and possibly altering some FTP messages as they pass through in order to accommodate this hacking. Tectia has an FTP forwarding mode, while OpenSSH doesn’t. The Tectia usage is:

    # Tectia
    $ ssh -L ftp/2001:localhost:21 S

This logs into server S, forwarding local port 2001 with the FTP workaround magic, to the FTP server running on S (the normal FTP control port is 21). To use the forwarding, point your FTP client at localhost:2001. FTP programs vary in syntax for this; some examples are:

    $ ftp localhost 2001
    $ ftp -P 2001 localhost

As long as the FTP and SSH clients are together on one host, the servers are together on another, and “localhost” is used as shown in the commands, both active and passive FTP now work. This is normally the way you want it, since if the clients or servers are split up, then FTP data transfers (which include directory listings) pass in the clear over a portion of the path, unprotected by SSH. However, in some circumstances you might be forced to split one side up. As a result of the way Tectia FTP forwarding works, the rule is:

Observe that if you split up both sides so that four separate hosts (technically, addresses) are in the picture, then neither mode works, and FTP won’t work at all beyond the initial connection and login.

This rule applies because when Tectia forwards ports to accommodate FTP data connections, the ports listen on the loopback address only, forcing both participants on one side or the other to be on the same host. Which side depends on which mode: in active mode the FTP server makes the data connections, so the SSH forwardings are remote, forcing the servers to be together. In passive mode, the FTP client makes the connections, so the SSH forwardings are local, forcing the clients to be together.