Next, let's take a look at monitoring networking services. Networking services in general usually stay running, and things that go wrong are happening inside the running service. We will go ahead and put a service status check on each of them and add additional checks to make sure things are working across the board. Start with giving each of the network services a service status check – the same checks that the control services got:

neutron-dhcp-agent
neutron-l3-agent
neutron-lbaas-agent
neutron-metadata-agent
neutron-metering-agent
neutron-openvswitch-agent
neutron-ovs-cleanup
openvswitch

Now, let's look at what can be monitored to make sure that when these services say that they are running, the network service is actually running. The configuration we have used in this book uses VXLAN tunnels to build overlay networks for OpenStack tenants. What this means is that each compute node is connected to the network node and to each other with VXLAN tunnels that encapsulate the traffic so that the network that actually connects the nodes doesn't directly handle the network traffic within Open vSwitch. When a packet is put on the network by an instance, it is tagged with a local VLAN to the compute node. When the packet moves from the compute node to the tunnel between that compute node and either another compute node or the network node, it gets retagged and the encapsulation header is added to the packet. This header somewhat hides the VLAN tag given to the packet to move across the tunnel from the actual network that connects the nodes that the packet is moving between. After the packet reaches the destination node it was sent to, the header is removed, and the packet is then retagged again to move around locally within this next node. These tunnels are handled by a running OVS process, and the ports and interfaces that are added and removed are handed by the running neutron OVS agent running on each node. Here is where, just because an agent is running, it does not mean that traffic is flowing from node to node without issue. To monitor that traffic is actually flowing, we can build our networking resources in OpenStack that mock the process followed when an instance is attached. Instead of attaching an instance to it, we will expose an interface to the node that we want to make sure is properly networking and send a ping across it. If the ping succeeds across the interface exposed to the node, then we know that the entire encapsulation just described is working properly.

The first step to set up the networking is to create a network specifically to do the tunnel monitoring on. Make sure you have sourced your admin's keystonerc file, and create the network. Refer to Chapter 2, Identity Management, if you need to revisit the keystonerc file. Here's the command that is being discussed:

control# neutron net-create tun-mon
control# neutron subnet-create tun-mon 10.0.0.0/24

Take note of the network ID from the net-create command; you will need that at the end of this process. Next, manually create a neutron port for each node that you want to monitor tunnel connectivity on. This is most likely each of your compute nodes. You don't need to do this for the network node or the control nodes. The control node has nothing to do with your networking in OpenStack, and the network node is the node that you will be pinging to verify tunnel connectivity. Here's the command that is being discussed:

control# neutron port-create --name moncompute --binding:host_id=compute
$NETWORK_ID

NETWORK_ID is the ID of the network that was just created. You should be able to use grep to port out of a neutron port list now:

control# neutron port-list | grep moncompute

This will include a port ID, a MAC address, and an IP address. You will need those for the final step. Finally add a port in the OVS on the target machine and give it an IP address:

compute# ovs-vsctl -- --may-exist add-port br-int moncompute 
-- set Interface moncompute type=internal 
-- set Interface moncompute external-ids:iface-status=active 
-- set Interface moncompute external-ids:attached-mac=${PORT_MAC} 
-- set Interface moncompute external-ids:iface-id=${PORT_ID}
compute$ ip link set dev moncompute address ${PORT_MAC}
compute$ ip a add ${PORT_IP}/24 dev moncompute

Now, on the compute node, you can verify that all is in place. First, look at the interface:

compute# ip a s moncompute

You should see an interface that has the IP address and MAC address that corresponds to the Neutron port you created for the node. Next, look at the routing table:

compute# ip r

You should see a routing entry for the subnet that you gave to the tun-mon network. In the example command earlier, it was given as 10.0.0.0/24, so the routing entry should look like the following line of code:

compute# 10.0.0.0/24 dev moncompute proto kernel scope link src
${PORT_IP}

Finally, on the compute node, you can look at the port in the OVS:

compute# ovs-vsctl show

Look for a port named moncompute. It will have an interface named moncompute, which is the interface you just looked at, and it will have a VLAN tag number. The last thing to do is get the DHCP address from the network you created and ping it. To get the DHCP agent's IP address, show the interfaces in the network namespace for your network on the network node:

control# ip netns exec qdhcp-${NETWORK_ID} ip a

You will see 127.0.0.1 and another address, probably 10.0.0.2 or 10.0.0.3 if you used the same subnet as the example. This address is the DHCP agent for the network you created. Now, try and ping that address from the compute node:

compute# ping 10.0.0.3 -c 3

If you get a reply ping when you do this, your tunnel is working. The way this traffic is funneled over the wire ensures that the VXLAN tunnels in your OpenStack cluster are working properly. These resources should stay in place unless you delete them but the OVS interface on the compute node will have to be recreated if the node is rebooted. You will have to get creative about how to persist or re-establish the interface if the node is rebooted. The ping can be added to Nagios so that you get your tunnel status with the rest of your checks. Let's move on to compute services and take care of the ping there.