Network Routing
Because machines in a data center usually have multiple network interfaces, questions will sometimes arise about which interfaces particular kinds of traffic should traverse. For example, it’s relatively common to see a machine with a front-end network interface connected to one VLAN for communication to the web servers and a back-end network interface connected to a different VLAN for communication to the database servers. In this case, the server must be told which interface to use in order to reach a particular destination IP address.
In the case of nearby servers, the routes are probably easy; they’ll just be based on the subnet addresses. In the example of the application server, the back-end interface probably shares a subnet with the database server, while the front-end interface probably shares a subnet with the web servers. Routing gets a bit more complicated when distant services—perhaps third-party services—are involved.
Modern operating systems strive to make routing automatic and invisible. When a machine brings up its primary NIC (whichever one it happens to think is primary, anyway), it uses the main IP address for that NIC as its “default gateway.” That becomes the first entry in the routing table for the host. As the host gets cozier with its switches, they gossip about routes and the host updates its routing table. That table tells the operating system which NIC to use to reach a destination address or network. When an application sends a packet, the host checks the destination IP address against the routing table to see if it knows how to move that packet a hop closer to its destination.
Most of the time, this “just works.” Occasionally, though, you can run into problems when multiple routes seem plausible to the host but aren’t actually equivalent. Consider the case of a service provided by a close business partner. If the integration includes personally identifiable information (PII), then you might set up a VPN rather than send sensitive data straight over the public Internet. Depending on a ton of configuration options that are outside your control, both the VPN and the primary switch may advertise routes that could reach the destination address.
In the best case, you’ll discover this problem during testing because nothing will reach the partner’s service. Your service won’t be able to open a socket and will get a “destination unreachable” response.[28] How is that the best case? A consistent error is much better than intermittent success. If the host happens to receive route advertisements in the right order, it might send those sensitive packets over the VPN. If it gets them in the wrong order, it may try to send them over the front-end—in other words, the public—network. Here’s hoping the partner is better at networking and won’t accept connections. Otherwise, that PII will be sent in cleartext over the public Internet. Worse still, your service will appear to be working normally so you won’t even know it’s happening.
One solution is static route definitions. Network admins officially frown on static routes, but sometimes they’re the only way.
Another increasingly common solution to routing is software-defined networking. This goes hand-in-hand with virtualized infrastructure and container-based infrastructure. Containers and VMs use virtual IP addresses, VLAN tagging, and virtual switches to create a kind of “network on a network.” The packets still run over the same wires, but the host machine’s IP address is not involved. This lets the virtual switches operate independently of the physical ones. They can assign IPs from private pools, attach DNS names to those IPs to identify services, and dynamically create firewalls and subnets.
Getting these routing issues right requires paying attention to each and every integration point. Getting them wrong risks reduced availability or, worse, exposure of customer data. For each connection to a remote system, I recommend keeping a record in a spreadsheet or a database with the destination name, address, and desired route. Someday, somebody is going to need that information to write firewall rules anyway.