Online Strategic TE Design

In the section “Tactical TE Design,” you saw that the farther away from a congested link TE tunnels are deployed, the more fine-grained your traffic manipulation can be, and the more likely you are to avoid congestion.

The logical progression of this model is to have a full mesh of TE LSPs at some level in the network, typically the WAN core routers. Doing so gives you the most control over the traffic that's entering your WAN, because you can manipulate traffic at all points where it enters the core.

This is the strategic method of deploying MPLS TE.

In this model, you decide where the boundaries of your TE cloud are, and you build a full mesh of TE LSPs between them. These TE-LSPs reserve bandwidth commensurate with what they're actually carrying, rather than reserving only enough bandwidth to have the proper forwarding ratios. The actual bandwidth value is reserved because it makes things simple. As traffic demands between routers change, the traffic demands can be measured and the tunnels changed accordingly. Periodically, these TE-LSPs are resized to account for the amount of traffic they're actually carrying.

The strategic online model has some advantages over the tactical model. Because you're supposed to have LSPs between every node at the edge of the cloud, you won't be constantly stumbling across LSPs that you didn't expect to find. Also, full-mesh models tend to make more optimal use of your bandwidth than tactical ones, which can save you more money.

The full-mesh model definitely has some trade-offs. Table 9-3 compares the strategic model with the tactical model.

LSP Scalability

The most important issue on the mind of any self-respecting network engineer is scalability—“How well does this scale?”

The answer to this question depends on where you want to put your TE cloud. In this chapter's sample network architecture, three choices exist:

• Full mesh between WRs

• Full mesh between CRs

• Somehow involving the DRs in the equation

All three of these choices scale the same way (a full mesh is n * (n – 1) LSPs), but their scaling numbers are different. CRs outnumber WRs, which means more TE tunnels, which means more signalling load on your network.

You should also ask a few specific scalability questions:

• How many LSPs can a router be a headend for?

• How many LSPs can a router be a midpoint for?

• How many LSPs can a router be a tailend for?

The question of “how many” is murkier than it seems. For example, how do you decide when you have too many? One way to set this ceiling is to set a limit for network convergence time. Find the maximum number of LSPs at each head/tail/mid role that converge in x amount of time, and that's your scalability limit. Of course, this number of LSPs vary from network to network, because convergence time is affected by things such as router CPU utilization, which in turn is influenced by what else is going on in the network, the size of the network, and so forth.

However, Cisco has put a stake in the ground and has done some initial convergence testing. Appendix B,“CCO and Other References,” has a pointer to a document titled “MPLS Traffic Engineering (TE)—Scalability Enhancements,” which gives the following scalability guidelines:

• 600 LSPs as a headend

• 10,000 LSPs as a midpoint

• 5000 LSPs as a tail

The number of LSPs as a tail is fairly irrelevant. Because a full-mesh architecture is the basis here, the number of LSPs for which a router is a head and for which it is a tail are the same, so the gating factor in head/tail scalability is the number of headends.

The numbers given here are neither the minimum number of tunnels that can be supported nor the maximum. Scalability depends tremendously on what else the router and the network are doing. Some routers and networks can deal with far more TE tunnels, because they're otherwise relatively unencumbered, and the network is stable. Others can't hit the numbers listed here because the routers are overworked and the network is unstable.

These numbers, however, serve as a starting point. As you'll see in the upcoming discussion of scalability, these numbers are quite large and give you plenty of room to expand your network.

Data and Variables

Before you begin exploring scalability in great depth, review the sample network. The sample network we're using is built out of the pieces listed in Table 9-4.

Now that the data is defined, Table 9-5 defines some variables to enable discussion about this data.

The factors to consider with scalability differ depending on the role a given router plays in the network. Table 9-6 lists the router types in the sample network.

Full Mesh Between WRs

The first type of strategic TE to look at is a full mesh of TE LSPs between WRs. This design probably makes the most sense, because the WRs are the devices that connect other POPs. WAN links are generally the most expensive links in a network, and as such, you get the most bang for your buck by applying MPLS TE to optimize your bandwidth utilization in the WAN.

As far as the number of LSPs that a router is a headend or tail for, this is easy to figure.

There are 14 POPs, with two WRs each, so Wn = 28.

If the full mesh is between the 28 WRs, Ln = 28 * 27 = 756 LSPs.

The other fact you need is the number of LSPs for which a router can be the midpoint. This is a tricky fact to determine accurately, because this number depends on the degree of connectivity of a given router (the number of physical links it has to neighbors), as well as the bandwidth of that router's links, the router's location in the network, and the overall network topology. However, it's possible to do some rough calculations that give you an approximation of the required load on any router so that you can get an idea of whether MPLS TE will scale to your requirements.

Continue using the sample network that's already established. First, you need to calculate M, the number of LSPs for which a router can be a midpoint. Clearly M <= Ln, because you can't have a midpoint for more LSPs than are present in the network. However, to find the value of M, you can start with Ln and subtract the following:

• All LSPs for which a router is a headend (Wn – 1)

• All LSPs for which a router is a tail (Wn – 1)

because clearly a router will not be a midpoint for LSPs for which it is a head or tail.

So:

This is the maximum number of LSPs a router can be a midpoint for. However, because not all LSPs go through the same router (does traffic from New York to Boston need to transit through Denver?), M is much smaller. One decent number to consider is the average number of LSPs per router. Let's redefine M as follows:

This is simply the previous calculation for M divided by the number of WRs in the network.

The result of this calculation varies quite a lot, because links, bandwidth, and connectivity all figure into the equation, but assume that it's stable because it's a nice rough estimate.

In summary, a full mesh of Wn routers has the following characteristics:

• Each router is a headend and tail for Wn – 1 LSPs.

• There is a total of Wn * (Wn – 1) LSPs in the network.

• Each router will be a midpoint for approximately (Ln – (2 * (Wn – 1))) / Wn LSPs.

In the sample network of 28 WRs, this means

• Each router is a headend for 27 LSPs and a tail for 27 more.

• There are 756 LSPs in the entire network.

• Each router is a midpoint for approximately 25 LSPs.

You might notice that the scalability numbers for head/tail/mid are much larger than what is stated here. Given the earlier formulae and the presumed limits of 600 heads/5,000 tails/10,000 midpoints, the largest MPLS TE network of WRs you can build is a network of roughly 300 POPs with two WRs each. (Each router is a headend for 599 LSPs, a tail for 599 LSPs, and a midpoint for about 597 LSPs.)

Clearly, this is a large network. Not many networks have 300 POPs. You'll hit other scaling limits if you try to get this big with this model—limits such as port density and IGP scaling. But the important point (because this is a book about TE and not about IP design) is that MPLS TE scalability is not the gating factor in your design!

At least, not in a cloud made up of a full mesh of WRs. Are the scaling limits the same with the number of CRs? The formulae are different, so let's look at a CR mesh separately.

Full Mesh Between CRs

A full mesh of TE LSPs between WRs scales quite nicely, as discussed in the preceding section. But what about between CRs?

Why would you want to do a CR full mesh? Perhaps to get more exacting control over your traffic. Also, if some of the CRS are not in the same physical location as their adjacent WRs or DRs, but are instead in remote sub-POPs (so a MAN network), there might be some utility in optimizing the MAN bandwidth. Or maybe you want to do CR-to-CR accounting, or run MPLS VPNs over end-to-end TE tunnels between PEs (which are the CRs). Or maybe you're just curious to see how well TE really scales.

In the CR full mesh, it's important to understand that the CRs are the only routers that are TE headends or tails. DRs and WRs are only midpoints, because in this architecture, traffic enters and exits the network only on CRs. If external connections were brought in on DRs or WRs (such as high-speed customer links or public/private peers), you'd need to include those routers in the mesh. In this architecture, all traffic external to the network enters and exits on CRs.

This network has three different types of routers: CRs, DRs, and WRs. Not all POPs have DRs, but those that do need to consider LSP scalability within the DRs.

It should be obvious what the scaling factor is for CRs. Because the network model discussed here is a full mesh of CRs, and because CRs are only ever headend or tail routers and never midpoints, each CR has to deal with Cn – 1 LSPs. Because 128 CRs exist, each one has an LSP to all 127 of the others; this, hopefully, is clear.

So far, you're well within the stated scalability limits. A router can be a headend for 600 LSPs and a tailend for 5000, and you're designing for only 127 in either category. The more complicated question in a CR mesh is the DRs and WRs. DRs and WRs are midpoints for some number of TE tunnels. With 128 CRs in a full mesh, that's 16,256 LSPs (128 * 127). Because the stated scalability limit for a midpoint router is 10,000 LSPs, you need to make sure that it's not likely that a midpoint will approach that limit. Intuitively, it's probably not a big deal (what are the odds that a single router will carry 60 percent or more of your network traffic?), but the exercise is well worth going through. The next two sections examine DR and WR scalability in a CR full mesh.

DR Midpoint Scalability

A DR needs to care about the number of LSPs that originate, terminate, or both originate and terminate in a POP. Table 9-7 breaks down these numbers separately.

You can combine the last two terms into {[(Cn – Cp) * Cp] * 2}. What you have so far, then, is

This ultimately reduces to Cp (2Cn – Cp – 1).

This is the formula for Lp, the number of LSPs that originate and/or terminate in a given POP. So:

But you're not done yet. Because the assumption is an even distribution of load, you need to divide the results from the preceding calculation by the number of DRs to get the number of LSPs per DR in the POP. The formula then becomes

For demonstration purposes, let's plug some real numbers into this.

This calculation obviously makes sense only when DRs are present in a POP, so you know Dp will always be 3.

Cn is 128.

Cp varies, depending on the size of the POP.

Because only three POPs have DRs, here are the calculations for each POP.

In Seattle, Cp is 14. So the formula P(2N – P – 1) / 3 becomes

So each Seattle DR will be a midpoint for approximately 1125 LSPs. This is no big deal, because the scalability limit we're working under has a ceiling of 10,000.

In Raleigh, Cp is 12. This means that each DR is a midpoint for approximately 972 LSPs.

In New York, Cp is 20. This is the largest POP in the network. Even here, a DR is a midpoint for approximately 1567 LSPs.

So it should be obvious that DR LSP scalability is a nonissue. But what about WRs?

WR Midpoint Scalability

WRs are a little tricky. Why? Because WRs do double duty. Because no physical connections exist between the DRs in this network, all LSPs that originate and/or terminate in a given POP also go through the WRs. This is the same equation as for the DRs in POP, but spread over the number of WRs rather than DRs:

Wp is the number of WRs per POP; in this network, it's 2. So the formula for the number of LSPs for which a WR is a midpoint then becomes

This is the number of LSPs that originate or terminate in a POP (Lp) divided by the number of WRs in that pop (2).

But there's more to the calculations than just that. A significant number of LSPs (called transit LSPs) go through WRs that are not in their own POP (transit WRs and transit POPs). This is because if you build an LSP from a CR in St. Louis to a CR in Seattle, those LSPs have to go through other POPs to get there. And because the WRs are the ones with the WAN links, LSPs have to transit WRs that are not in the headend or tailend POP.

But how many LSPs will a given WR see? The real answer to this question is difficult to calculate. It has to account for bandwidth availability at every hop, because a major function of full-mesh TE LSPs is to consume bandwidth, thereby finding the best path through the network. And because LSPs make path determination based on bandwidth, they might build paths across some arbitrary path in the network. For example, you can have an LSP from StLouisCR1 to MiamiCR1 via Raleigh→Atlanta→Miami and an LSP from StLouisCR2 to MiamiCR2 via Dallas→Houston→Miami. And at every transit POP, an LSP might go in one WR and out the same WR, or in one WR and out the other WR. And things get even more complex, as you will see in the section “The Packing Problem.”

Because the path an LSP can take depends on available bandwidth, and because available bandwidth depends on what other LSPs are already in the network, this leads to rather complex calculations. See the section “Offline Strategic TE Design” for more details.

Even if you assume that there are no bandwidth issues, this is still a difficult problem. Assume for a second that all LSPs follow the IGP shortest path from head to tail. Of course, if this were really the case, this obviates the need for MPLS TE, but make the assumption nonetheless. Where does that leave you?

You'd have to run SPF for every POP to every other POP, to see which POPs' traffic would transit other POPs on the way to their destination. Because you have 14 POPs, that's 14 SPFs. And then you'd have to figure out how many LSPs from a headend POP to a tailend POP there would actually be, because this varies depending on the number of CRs in both POPs. This is a much simpler problem to solve, but it's still too big to tackle here.

Let's simplify the problem some more, so we can get an approximate answer. You know that there are Ln LSPs in the network. This means that any one WR will never transit for more than Ln LSPs. So, in other words:

Remember, Lt is the number of LSPs that transit a POP. So you can make this number smaller. You've already accounted for LSPs that originate or terminate in WR's own POP (Lp). So the term can be refined to

You can take this further by realizing that a WR in a given POP will never be transit not only for intra-POP LSPs (LSPs in its own POP), but also for intra-POP LSPs in all other POPs.

So take the sum of Lp for all POPs—in other words, (Cp * (Cp – 1)) across all POPs. If you let X be the set of Cp for each POP, you're now looking at this:

To make this data easier to visualize, check out Table 9-8.

Furthermore, because there are Wn WRs in the network, the formula becomes

Why are you dividing by Wn? Because you're assuming that the number of LSPs is divided equally across all WRs. This is not a terribly valid assumption, because it assumes that all LSPs cross only one transit WR (not including WRs in the originating or terminating POP). In real life, an LSP crosses anywhere from one to 24 WRs.

24? Yes. Because you're not counting WRs in the originating or terminating POPs, that leaves you with 28 – 2 – 2 = 24 WRs that an LSP can possibly pass through. 24 is the absolute worst case, though. 24 WRs would be an LSP that went from St. Louis to Denver along the path St. Louis→Dallas→Houston→Miami→Atlanta→Raleigh→Chicago→ New York→D.C.→Boston→Seattle→San Jose→San Diego→Denver and hits both WRs in every POP along the way.

This, you will agree, is astoundingly suboptimal. But it is possible.

So you should really multiply Wn by some constant, which is the average number of transit WRs that an LSP will touch.

The problem with finding this number is that it requires a fair bit of work to find, and if you could find that number, you could probably also solve some of the other hard problems we've deliberately tried to avoid.

Assume that the average LSP transits through three WRs. Call this number T, the number of transit routers an LSP goes through. T could be through three POPs (WR1 to WR2 to WR1) or two POPs (WR1 to WR2). This number is based on real-world experience with LSPs in ISP networks; feel free to substitute your own number if you like. The calculations here, however, use T = 3.

This means that

Lt is the number of LSPs that cross a WR but that neither terminate nor originate in that POP. But to fully account for the number of LSPs a WR will be a midpoint for, you need to account for Lp again. So the final equation becomes

Because Cp varies per POP, you should do this calculation once per POP. You could probably come up with the average Cp value across all POPs and just use that for each POP, but it's not that much work to calculate this number for each POP. In fact, for this example, you don't have to do any work, because it's already done for you, in Table 9-9.

Table 9-9 shows the estimated number of midpoint LSPs per WR, per POP.

So, as you can see (if you've stayed awake this far!), you're comfortably under the scalability limits for WR midpoints, which is 10,000 LSPs.

This might be clearer with a summarization of all the data in one big table (see Table 9-10). For each POP, there are three columns—WR, DR, and CR. The number in each column is the number of LSPs each router in that POP has to deal with:

• For WRs, this number is the number of LSPs for which the WR is a midpoint.

• For DRs, this number is the number of LSPs for which the DR is a midpoint.

• For CRs, this number is the number of LSPs for which a router is a headend and also the number of LSPs for which the router is a tail.

• At the top of each column is the number that is the scalability limit for that column—10,000 LSPs as a midpoint for WRs and DRs, and 600 headend LSPs at the CRs.

As you can see, you are well under the scalability limits in all cases. Now, some assumptions were made that might not be true (especially in the case of T in the WR calculations), but they are accurate enough to make our point, which is that MPLS TE is not the gating factor in scaling your network.

Other Growth Factors

In the tactical architecture, you read about periodically cleaning up after yourself. The strategic architecture doesn't really have housecleaning per se, but it does have a few things you need to be concerned about. These things are mostly a concern if you're going to do online path calculation; if you do offline path calculation, the path calculation tool itself should help take care of these details.

You should consider three principal things when talking about online tactical housecleaning:

• Tunnel reoptimization

• Tunnel resizing

• Multiple parallel LSPs

These topics are discussed in the following sections.

Tunnel Reoptimization

The first thing you need to consider is tunnel reoptimization. Reoptimization in this context is the recalculation of CSPF for all LSPs, to see if there is a more optimal path across the network they can take. Why is this important?

Consider the network shown in Figure 9-8.

At time 0, Router A has a 400-Mbps LSP to Router D that goes across the OC-48 link between Router B and Router C.

Sometime after this LSP is up (call it time 1), the OC-48 link goes down for some reason. No problem. The 400-Mbps LSP just reroutes over the OC-12, and all is well.

At time 2, the OC-48 link comes up.

What happens next? Eventually, you'd like the A→D LSP to move back to the OC-48 link, because it has more headroom and can better handle a burst of traffic. But without reoptimization of some kind, the A→D LSP will stay across the OC-12 link until either the OC-12 link or the A→D LSP goes down for some reason.

So you need some kind of reoptimization, to tell the headend to try to find new paths for the LSPs, if there are better ones than those it is currently using. There are three kinds of reoptimization events:

• Periodic reoptimization

• Event-driven reoptimization

• Manual reoptimization

Periodic Reoptimization

Periodically, an LSP headend tries to calculate new paths for all its LSPs. If a better path is found, the headend signals the new LSP and installs the new outgoing label; this is make-before-break.

The default periodic reoptimization timer is 1 hour (3600 seconds). In a CR full mesh in our sample network, every hour, a router recalculates paths for all 127 of its LSPs. This periodic interval can be tuned using the global command mpls traffic-eng reoptimize timers frequency. You can go quite low with this timer; some networks tune it down to 60 seconds or less. The reoptimization timer can be set to no less than 30 seconds; trying to set a nonzero value less than 30 will result in a configured value of 30.

You can also disable periodic reoptimization. If you set the periodic reoptimization timer to 0, the tunnel's LSPs will never be reoptimized.

You can see how much more time you have to go until the reoptimization timer kicks in with show mpls traffic-eng tunnels summary, as demonstrated in Example 9-3.

Example 9-3. show mpls traffic-eng tunnels summary Output Shows the Time to Reoptimization

gsr3#show mpls traffic-eng tunnels summary
Signalling Summary:
    LSP Tunnels Process:            running
    RSVP Process:                   running
    Forwarding:                     enabled
    Head: 2 interfaces, 1 active signalling attempts, 1 established
          6 activations, 5 deactivations
    Midpoints: 0, Tails: 0
    Periodic reoptimization:        every 3600 seconds, next in 3599 seconds
    Periodic auto-bw collection:    every 300 seconds, next in 89 seconds

Why tune down to 60 seconds or less? Suppose you have strict SLAs that result in financial penalties if they're not met. This means that you have a strong incentive to try to find the shortest path possible for your LSP as quickly as you can. For the network shown in Figure 9-8, you want LSPs to take the OC-12 link only if they have to and revert to the OC-48 link as quickly as possible. So this is an incentive to make the reoptimization timer as low as possible.

But you run into scalability issues here. If you ask the router to reoptimize all its LSPs every 30 seconds, and the time to run CSPF for all that router's LSPs is more than 30 seconds, you're in trouble. Tunnel CSPF run times are typically in the low milliseconds (about 20 to 40 ms, depending on the size of your network), so you can fit quite a lot of LSPs in this time. But still, running that often might suck up too much CPU and cause other problems.

This problem makes for a nice segue into its solution—event-driven reoptimization.

Event-Driven Reoptimization

The problem with periodic reoptimization is that all LSPs are recalculated every time the reoptimization timer goes off. And if you tune the reoptimization timer pretty low, almost all the time your per-LSP CSPF is run, no new paths will be found, because the odds of a link flapping somewhere in your TE network (usually your core) in the last 30 seconds are quite low.

What you'd rather do is recalculate LSP paths only if something changes in the network. From this perspective, there are two kinds of changes—links going up and links going down. The network considers one node going down to be the same thing as all the links on that node going down.

Links going down don't need to trigger reoptimization. Any LSPs that cross those links will be rerouted when they receive failure notification via RSVP or the IGP, and any LSPs not crossing a failed link don't need to care about a link going down.

But what about when links come up? When a link comes up, this represents new resources that existing LSPs can take advantage of. LSPs that are currently down (because they can't find the resources they want) try to find new paths every 30 seconds, so no worries there.

What about LSPs that are up but that don't follow the most optimal path? You'd like to run reoptimization checks for these LSPs only when a link (or multiple links) somewhere in the network comes up, because only then would these LSPs be able to find a better path.

Event-driven link-up reoptimization can be enabled with mpls traffic-eng reoptimize events link-up. If you turn this on, whenever a link comes up somewhere in the network, CSPF for all existing LSPs is run to see if they can get a more optimal path through the network using the link that's just come up.

This can be dangerous if you have a link that flaps (goes up or down) more often than you'd run the reoptimize timer. Suppose you want to turn the reoptimize timer down to 60 seconds, but you use events link-up instead. If you have a link that flaps every 5 seconds (down for 5 seconds, up for 5 seconds, etc.), you'll run six times as many reoptimizations as you would without this command. You might also end up with more RSVP signalling, depending on whether the reoptimization on linkup moves LSPs onto the link that just came up.

Not to worry, though. Because link availability information is flooded in the IGP, you can fall back on the IGP's safety mechanisms to prevent this. IS-IS and OSPF both have timers you can tweak to control constant link advertisement and withdrawal in this type of situation. These knobs apply to the TE information for that link as well. Relying on the IGP only makes things a little worse than they would be without MPLS TE (you run an IGP SPF and many tunnel CSPFs), but not much worse, and then only in pathological failure conditions.

How do you know if a tunnel is following a suboptimal path? Use the command show mpls traffic-eng tunnels suboptimal constraints current. Like many other show mpls traffic-eng tunnels commands, this can be used with or without the brief keyword, as demonstrated in Example 9-4.

Example 9-4. Determining Whether the Tunnel Is Following a Suboptimal Path Using show mpls traffic-eng tunnels suboptimal constraints current brief

gsr3#show mpls traffic-eng tunnels suboptimal constraints current brief
Signalling Summary:
    LSP Tunnels Process:            running
    RSVP Process:                   running
    Forwarding:                     enabled
    Periodic reoptimization:        every 3600 seconds, next in 967 seconds
    Periodic auto-bw collection:    every 300 seconds, next in 157 seconds
TUNNEL NAME                      DESTINATION      UP IF     DOWN IF   STATE/PROT
Tunnel1 from GSR3 to GSR7        192.168.1.7     -         PO2/1     up/up
Displayed 1 (of 2) heads, 0 (of 0) midpoints, 0 (of 0) tails

Table 9-11 describes the three types of constraints you can display.

Manual Reoptimization

Assume that after careful study, you decide to leave the periodic reoptimization timer at its default of 1 hour and to not enable link-up reoptimization. Most of the time, you're fine with that. But occasionally, you find a particular LSP that needs reoptimization urgently and that can't wait for the timer to expire. What do you do then? You could shut down the tunnel and bring it back up again, but that disrupts traffic.

A better way to do this is via the EXEC command mpls traffic-eng reoptimize mpls traffic-eng reoptimize, which by itself reoptimizes all tunnels for which a router is a headend. You can also reoptimize a single tunnel with mpls traffic-eng reoptimize tunnel-interface.

Tunnel Resizing

Tunnel resizing is slightly different from the tunnel reoptimization discussed in the previous sections. Reoptimization involves finding a new path for an existing tunnel. Resizing involves changing the bandwidth requirement on tunnels, based on demand. Of course, changing the bandwidth requirements might also change the path a tunnel takes.

There are two ways bandwidth requirements can change—they can go up or down. Obviously, you need to care about both types of events.

You still have a few questions to be answered. The first is “How often do I resize my tunnels?”, and the second is “What size do I make my new tunnels?”

When determining how often to resize new tunnels, you want to be careful not to resize too often, because this makes TE look more like a microscale thing than a macroscale thing. Resizing every 10 minutes, for example, is too often.

On the other hand, you don't want to resize too infrequently. If you don't resize in a timely fashion, you'll have a TE LSP topology that's out of sync with the real world.

There is no clear-cut answer to how often you resize your tunnels. Like so many things, the answer is “It depends.” Here are some questions to ask:

• What do your traffic patterns look like?

• Do you have clearly defined busy periods in which your traffic patterns obviously increase and decrease?

• Do you have daily, twice-daily, or weekly traffic patterns?

You'll have to investigate. In the absence of any sense of where to start, you might want to start by resizing every 8 hours to see if that fits your traffic demands. Getting this resize interval right takes a fair bit of work to get started, but as soon as you have a better handle on your traffic, it gets easier.

You might not resize TE tunnels on a fixed interval. Maybe you want to resize every 2 hours from 6 a.m. to 6 p.m. and every 4 hours from 6 p.m. to 6 a.m. The choice is up to you.

So now that you know you need to resize your LSPs, how do you actually do this? There are two ways. One is to use an offline tool, and the other is to have the router itself make the resize decision. Using an offline tool is covered in the next section, so we won't deal with it here.

How do you have the router resize tunnels automatically? By using auto bandwidth. See the section “Automatic Bandwidth Adjustment” in Chapter 5 for a discussion of how auto bandwidth works. Auto bandwidth has reasonable defaults for how often and how accurately it resizes tunnels, but like everything else, it's a good idea to keep an eye on things and tune your network as you see fit.

Multiple Parallel LSPs

As mentioned in the discussion of tunnel numbering, sometimes you might want parallel tunnels between two routers. You might find parallel tunnels effective if you want to send more traffic between those routers than you can fit on any single path between CRs. Or if the only path you can find between two routers goes quite out of the way, splitting that traffic across two tunnels can help alleviate the problem.

The basic idea behind multiple tunnels was covered in the tactical discussion earlier. But how do you apply it to a full-mesh (strategic) architecture?

This is actually pretty tricky in an online strategic model. You need some way to decide that traffic heading toward a particular destination is more than the available physical bandwidth. Auto bandwidth doesn't have such a facility.

At first glance, you might think that all you need to do is monitor both the reserved bandwidth for an LSP and the actual traffic going down an LSP, and change the reserved bandwidth when the two values get too far out of sync. The problem with this approach, though, is that it can't always help you. TCP, which is the majority of Internet traffic, adjusts itself to fit the available bandwidth. This means that if you build a TE LSP across a link that is a bottleneck (either it's a lower-speed link or it's close to maximum utilization), the traffic down the TE LSP decreases to fit in the congested link. To do this right, you need to monitor your network for links that are close to maximum utilization and move TE LSPs off those links, to allow the TCP traffic inside those LSPs to open up to as much bandwidth as it can take.

Doing this monitoring and readjusting is nontrivial. This problem was touched on earlier, in the section “Determining When TE LSPs Are Causing Problems.” It's something you need to be aware of if you plan to use MPLS TE on a large scale.