In this recipe, we will troubleshoot an infrequent yet very persistent issue that can occur when setting up a VPN connection. When the redirect-gateway directive is used to redirect the default gateway on an OpenVPN client, it sometimes causes the client to lose all the Internet connections. This particularly occurs when the client machine on which OpenVPN is running is connected to the rest of the network or with the Internet using a PPP-based connection, such as PPPoE or PPPoA, especially, when using a GPRS/UMTS connections via a mobile phone.
When this occurs, OpenVPN sometimes is not capable of determining the default gateway before it is redirected. After the default gateway is redirected to the OpenVPN tunnel, the whole tunnel collapses on itself, as all the traffic, including the encrypted tunnel traffic itself, is redirected into the tunnel, causing the VPN to lock up.
This recipe will show how to detect this situation and what can be done about it. In this recipe, we will not use a GPRS/UMTS connection but we will use a PPP-over-SSH connection, which behaves in a similar fashion and is more readily available.
We use the following network layout:
Set up the client and server certificates using the first recipe from Chapter 2, Client-server IP-only Networks. For this recipe, the server computer was running CentOS 6 Linux and OpenVPN 2.3.11. The client was running Fedora 22 Linux and OpenVPN 2.3.11. Keep the configuration file basic-udp-server.conf from the Server-side routing recipe from Chapter 2, Client-server IP-only Networks.
Make sure the client is connected to the network using a PPP connection, as otherwise the issue described in the title of this recipe will not occur. For this recipe, a PPP-over-SSH connection and the default route was altered to point to the ppp0 device.
- Start the server and add an extra parameter to direct the default gateway:
[root@server]# openvpn --config basic-udp-server.conf --push "redirect-gateway"
- Create the client configuration file:
client proto udp # next is the IP address of the VPN server via the # PPP-over-SSH link remote 192.168.222.1 port 1194 dev tun nobind ca /etc/openvpn/cookbook/ca.crt cert /etc/openvpn/cookbook/client1.crt key /etc/openvpn/cookbook/client1.key tls-auth /etc/openvpn/cookbook/ta.key 1 user nobody verb 5
Save it as
example7-9-client.conf. - Check the system routes before starting the client:
[root@client]# netstat -rn 172.30.0.10 172.30.0.1 255.255.255.255 UGH 0 0 0 eth0 192.168.222.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0 0.0.0.0 192.168.222.1 0.0.0.0 UG 0 0 0 ppp0
- Now, start the client:
[root@client]# openvpn --config example7-9-client.confThe connection will start but after a few seconds will stop and the log file will contain a warning message:
... OpenVPN ROUTE: omitted no-op route: 192.168.222.1/255.255.255.255 -> 192.168.222.1 - Check the system routes again:
[client]$ netstat -rn 172.30.0.19 172.30.0.1 255.255.255.255 UGH 0 0 0 eth0 192.168.222.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0 192.16.186.192 0.0.0.0 255.255.255.192 U 0 0 0 eth0 10.200.0.0 0.0.0.0 255.255.248.0 U 0 0 0 tun0 10.198.0.0 10.200.0.1 255.255.0.0 UG 0 0 0 tun0 0.0.0.0 10.200.0.1 0.0.0.0 UG 0 0 0 tun0
The default gateway is now the VPN tunnel but the original route to the gateway is now gone.
All the connections on the client have stopped. What's worse, when the OpenVPN client is aborted (by pressing Ctrl +C in the terminal window) the default route is not restored, as the OpenVPN process does not have the proper rights to do so:
TCP/UDP: Closing socket /sbin/ip route del 10.198.0.0/16 RTNETLINK answers: Operation not permitted ERROR: Linux route delete command failed: external program exited with error status: 2 /sbin/ip route del 192.168.222.1/32 RTNETLINK answers: Operation not permitted ERROR: Linux route delete command failed: external program exited with error status: 2 /sbin/ip route del 0.0.0.0/0 RTNETLINK answers: Operation not permitted ERROR: Linux route delete command failed: external program exited with error status: 2 /sbin/ip route add 0.0.0.0/0 via 192.168.222.1 RTNETLINK answers: Operation not permitted ERROR: Linux route add command failed: external program exited with error status: 2 Closing TUN/TAP interface
The result is that the default gateway on the client machine is gone. The only solution is to reload the network adapter so that all the system defaults are restored.
The solution to the above problem is to use the following as is done in the Redirecting the default gateway recipe from Chapter 2, Client-server IP-only Networks:
push "redirect-gateway def1"
When the OpenVPN client initializes, it always tries to create a direct route to the OpenVPN server via the existing system gateway. Under certain circumstances this fails, mostly due to an odd network configuration. It is seen most often when the default gateway is a dial-up or PPPoE connection, which is used in certain ADSL/VDSL setups and especially when using GPRS/UMTS connections.
When the OpenVPN client is instructed to redirect all the traffic over the VPN tunnel, it normally sends the encrypted VPN traffic itself over a direct link to the OpenVPN server. You can think of the encrypted VPN traffic as outside of the tunnel. However, when this direct route is missing, then this outside traffic is also sent into the tunnel, creating a tunneling loop from which the VPN can never recover.
In the example used in this recipe, the situation is made worse by using the client configuration directive:
user nobody
This tells the OpenVPN process to drop all the privileges after starting. When the client is aborted because the tunnel is not functioning properly, the client is not capable of restoring the original gateway and the system is left in a non-functioning state:
[client]$ netstat -rn 194.171.96.27 192.16.186.254 255.255.255.255 UGH 0 0 0 eth0 192.168.222.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0 192.16.186.192 0.0.0.0 255.255.255.192 U 0 0 0 eth0
Only by adding a new default gateway can the network be restored.
The proper fix is to use:
push "redirect-gateway def1"
This will not overwrite the existing default gateway but will add two extra routes:
0.0.0.0 10.200.0.1 128.0.0.0 UGH 0 0 0 tun0 128.0.0.0 10.200.0.1 128.0.0.0 UGH 0 0 0 tun0
Both of which cover half the available network space. These two routes effectively replace the existing default route whilst not overwriting it.
This "biting your own tail" problem was much more common with older versions of OpenVPN. In current versions of OpenVPN, the detection of the default gateway was much improved and this problem now rarely occurs anymore. However, it is useful to see what happens when the problem does occur.
- The Redirecting the default gateway recipe from Chapter 2, Client-server IP-only Networks, which explains how to properly redirect all traffic via the VPN tunnel