Normally, ARP changes are not logged on a switch. Even if they were, scrolling through pages of log entries is not my idea of fun, so the idea of storing these events in a database is a bit appealing to me. It’s been a long time since I was a database guy, so let’s see how rusty I am. The way to retrieve the ARP events from the database is with the show event-monitor arp command.

Arista#sho event-monitor arp
2012-05-23 02:34:12|192.168.1.200|Management1|00:02:55:b7:da:9d|0|added|3
2012-05-23 02:34:12|192.168.1.200|Management1|||removed|4
2012-05-23 06:38:49|192.168.1.1|Management1|00:14:6a:a2:d4:38|0|added|26
2012-05-23 18:11:27|192.168.1.200|Management1|00:02:55:b7:da:9d|0|added|27

My main problem with this tool is that the output is, shall we say, ugly. Well, it’s ugly from my “all user interfaces should be elegant and beautiful” point of view. From my “it’s SQL, and that’s what SQL output looks like” point of view, it’s beautiful. I know, I’m a complicated person.

Don’t get me wrong, this is very useful information, but I have no idea what some of the values are. For example, I see some fields with zeros in them, and some with no values, and I have no idea what that last field in every line represents.

To help figure out what this stuff is, I’ll take advantage of a nifty sqlite command that shows me the command used to create the table. This command is called .schema, and can be accessed by using the sqlite keyword. I’ll cover the usage of the sqlite keyword a bit later in this chapter, but for now, bear with me and know that I’m sending the sqlite command .schema arp from EOS with the following command:

Arista#sho event-monitor sqlite .schema arp
CREATE TABLE arp( time text, prefix text, intf text, ethAddr text,
static integer, delta text,counter integer UNIQUE );

This output tells me that there was a table created named arp that contains the following fields, listed in order. I’ve also added what they mean for any nonprogrammer types out there:

Let’s see this table in action. First, here is the list of existing ARP entries:

Arista(config)#sho arp
Address         Age (min)  Hardware Addr   Interface
192.168.1.1             0  0014.6aa2.d438  Management1
192.168.1.200           0  0002.55b7.da9d  Management1

Now I’ll add a static ARP entry to my switch by mapping the IP address 192.168.1.2 to the MAC address learned for 192.168.1.1:

Arista(config)#arp 192.168.1.2 0014.6aa2.d438 arpa

Let’s see what the Event Monitor database shows now that I’ve added the static ARP:

Arista(config)#sho event-monitor arp
2012-05-23 02:34:12|192.168.1.200|Management1|00:02:55:b7:da:9d|0|
added|3
2012-05-23 02:34:12|192.168.1.200|Management1|||removed|4
2012-05-23 06:38:49|192.168.1.1|Management1|00:14:6a:a2:d4:38|0|
added|26
2012-05-23 18:11:27|192.168.1.200|Management1|00:02:55:b7:da:9d|0|
added|27
2012-05-25 16:05:37|192.168.1.2|Management1|00:14:6a:a2:d4:38|1|
added|108
% Writing 1 Arp, 0 Route, 1 Mac events to the database

Take a look at the line in bold, and let’s apply what we know. At 4:05 p.m. on May 25th, 2012, a static ARP was added that mapped the IP address 192.168.1.2 to the MAC address 00:14:6a:a2:d4:38. The ARP entry became active on interface Management1.

Now I’ll go in and delete the static ARP entry:

Arista(config)#no arp 192.168.1.2 0014.6aa2.d438 arpa

And here’s the output from the show event-monitor arp command after the change:

Arista(config)#sho event-monitor arp
2012-05-23 02:34:12|192.168.1.200|Management1|00:02:55:b7:da:9d|0|added|3
2012-05-23 02:34:12|192.168.1.200|Management1|||removed|4
2012-05-23 06:38:49|192.168.1.1|Management1|00:14:6a:a2:d4:38|0|added|26
2012-05-23 18:11:27|192.168.1.200|Management1|00:02:55:b7:da:9d|0|added|27
2012-05-25 16:05:37|192.168.1.2|Management1|00:14:6a:a2:d4:38|1|added|108
2012-05-25 16:06:56|192.168.1.2|Management1|||removed|109
% Writing 1 Arp, 0 Route, 0 Mac events to the database

Reading the line in bold, we can see that at 4:06 p.m. on May 25th, 2012, the ARP entry for the IP address 192.168.1.2 was removed. The ARP entry was previously active on interface Management1.

Remember that ARP entries are only made on the switch when the switch communicates directly using IP, or if the ARP table is manually manipulated. If you have no IP addresses active on your switch, this table will be empty. For example, if you manage your switches solely though console servers, and don’t even use the Management interface, you may not need to have IP addresses active on your switch at all. In this case, there would be no ARP activity to record.

MAC changes are logged any time a MAC address is learned, added, or deleted. This can happen a lot on a busy switch, but it might not happen much at all on a smaller, stable network. This can be a pretty useful tool, and I’ll show you how, so let’s start by taking a look at the table the same way we did for the ARP table. First, let’s look at the schema with the .schema mac sqlite command:

Arista#sho event-monitor sqlite .schema mac
CREATE TABLE mac( time text, fid integer, ethAddr text, intf text,
type text, delta text, counter integer UNIQUE);
% Writing 0 Arp, 0 Route, 2 Mac events to the database

Here’s the breakdown of the fields:

So let’s look at an example of how I like to use this feature. Generic Bob comes up to you at 4:30 p.m. on a Friday and says, “Hey, the network sucks.”

“What is it this time, Bob?” you ask, trying really hard not to roll your eyes since you read that chapter in Network Warrior about how not to be that guy.

“My system doesn’t work, and it used to. Nothing changed, and I don’t feel like thinking, so it has to be the network.”

After a heavy sigh, you stop the important work you were doing and focus on Generic Bob’s dilemma. “Where are you connected, Bob?”

“How should I know? You’re the network guru, and it’s your crappy network, so you figure it out.”

Repressing the thoughts that are better left to mystery writers and psychopaths, you ask, “Well, do you know your IP address?”

“No, I think it’s dynamic or something. Will this take long?”

“Look Bob, I need either the IP address or the MAC address to find your system. Get me either of those, or even better, both, and I’ll see what I can find, OK?”

Generic Bob storms off to do his generic thing while you pull up your résumé. Just as you start to include all your Arista switch experience, Bob comes back and thrusts a piece of paper at your face. “Here’s your Mick address.”

You smile the false, hopeless smile of the damned, gently take the paper, and log in to your Arista switch. Generic Bob’s coffee-stained paper has 3c:07:54:43:88:d4 scribbled on it in what appears to be red crayon, or maybe lipstick.

With event-monitor, you have a record of every MAC event for quite some time. Using the match-mac keyword, you pull up the event-monitor mac table including only events regarding the MAC address you care about:

Arista#sho event-monitor mac match-mac 3c:07:54:43:88:d4
2012-11-03 09:18:08|1006|3c:07:54:43:88:d4|Ethernet24|
learnedDynamicMac|added|549
2012-05-26 16:19:51|1|3c:07:54:43:88:d4|Ethernet6|learnedDynamicMac|
added|553

Looking at this data, you can see that Generic Bob’s MAC address was first learned on January 11th, 2012, and this switch saw it on interface e24. Since you know that e24 is an uplink to another switch, you know that Bob’s system was originally on another switch.

But the next line shows that the MAC address was learned today (hey, it was today when I wrote it), at 4:19 p.m. on the switch’s local interface, e6. That seems odd, so you look at the running-config for e6 on this switch:

Arista#sho run int e6
interface Ethernet6
   description [ Unused ]
   switchport access vlan 999

Since you’ve configured the switch with a dead VLAN (999) that you put on all unused ports, you’re sure that Generic Bob glommed a free interface on a different switch than the one to which he was assigned. With a couple of short commands, you’ve shown that Generic Bob was lying because someone had clearly moved his system from one switch to another, and that’s why it stopped working. You can’t blame the guy for thinking the new Arista switches would give him better performance, but plugging into network ports without prior authorization should not be tolerated.

I’ll leave it to your imagination as to how you might deal with Generic Bob now that you have the tools to prove him wrong. Just try not to be that guy…much. I’d probably just smile and tell him to put his system back into the network port where it belonged, and then make sure his access to the data center was revoked, but not until I was done adding Event Monitor to my resume.

If you’ve got your switches configured for IP routing, the show event-monitor route command can be a great tool to see historical information regarding route changes. As with the others, let’s take a look at the schema, and then work up an example to see it in action:

Arista#sho event-monitor sqlite .schema route
CREATE TABLE route( time text, prefix text, protocol text, metric integer, 
    preference integer, delta text,counter integer UNIQUE );

Here’s the breakdown of the fields from the route table:

Let’s start with a simple example. Here, I’ll add a static route to the switch:

Arista(config)#ip route 20.20.20.0/24 192.168.1.1

Using the match-ip ip-prefix option, here’s what shows up in the Event Monitor:

Arista(config)#sho event-monitor route match-ip 20.20.20.0/24
2012-05-27 00:54:40|20.20.20.0/24|invalid|0|1|added|1310

Now I’ll delete that route:

Arista(config)#no ip route 20.20.20.0/24 192.168.1.1

And an appropriate remove message shows up in the Event Monitor:

Arista(config)#sho event-monitor route match-ip 20.20.20.0/24
2012-05-27 00:54:40|20.20.20.0/24|invalid|0|1|added|1310
2012-05-27 00:56:29|20.20.20.0/24||||removed|1312

When the switch learns routes through other means, the results are pretty much the same, but since I went through the trouble to prove that to myself, I figure it’s only fair to share.

First, I’ll show what the routing table looks like:

Arista#sho ip route | begin Gateway
Gateway of last resort:
 S      0.0.0.0/0 [1/0] via 192.168.1.1

 C      30.30.30.0/24 is directly connected, Loopback0
 C      192.168.1.0/24 is directly connected, Vlan901

Next, I’ll add a simple OSPF configuration:

Arista(config)#router ospf 100
Arista(config-router-ospf)#network 192.168.1.0/24 area 0
Arista(config-router-ospf)#redistribute connected

Make sure we have a working neighbor (I’m not showing the neighbor’s configuration because it’s not an Arista switch, and really, who wants to see that?):

Arista(config-router-ospf)#sho ip ospf neighbor
Neighbor ID     Pri   State            Dead Time   Address         Interface
70.91.46.85       1   FULL/DR          00:00:34    192.168.1.1     Vlan901

So now that OSPF is up and working, let’s see what routes we’ve learned from our neighbor:

Arista#sho ip route | begin Gateway
Gateway of last resort:
 S      0.0.0.0/0 [1/0] via 192.168.1.1

 C      30.30.30.0/24 is directly connected, Loopback0
 O E2   192.168.0.0/24 [110/10] via 192.168.1.1
 C      192.168.1.0/24 is directly connected, Vlan901
 O E2   192.168.2.0/24 [110/10] via 192.168.1.1

Now that we’ve got two OSPF routes learned, each of which has a metric of 10, let’s take a look at Event Monitor. Before I do that, I’m going to clear out the buffer using the event-monitor clear command, because there’s too much output to wade through, and I’m tired of filtering the output:

Arista(config)#event-monitor clear
Clearing buffer and forever logs

Since I cleared the buffer with the event-monitor clear command, the counters at the end of the lines have restarted. Notice also that the OSPF routes shown previously show a metric of 10, but the route entries in the Event Monitor show protocol values of invalid, and metric values of 0:

Arista(config-router-ospf)#sho event-monitor route
2012-05-27 01:35:07|192.168.0.0/24|invalid|0|1|added|7
2012-05-27 01:35:07|192.168.2.0/24|invalid|0|1|added|8
% Writing 0 Arp, 4 Route, 4 Mac events to the database

Now let’s see what happens when I remove OSPF from the switch. First, I’ll remove the protocol from active duty:

Arista(config)#no router ospf 100

Then take a quick look to make sure all my OSPF routes are gone:

Arista(config)#sho ip route | beg Gateway
Gateway of last resort:
 S      0.0.0.0/0 [1/0] via 192.168.1.1

 S      20.20.20.0/24 [155/0] via 192.168.1.1
 C      30.30.30.0/24 is directly connected, Loopback0
 C      192.168.1.0/24 is directly connected, Vlan901

And finally, a look into the Event Monitor to see how it responded:

Arista(config)#sho event-monitor route
2012-05-27 01:35:07|192.168.0.0/24|invalid|0|1|added|7
2012-05-27 01:35:07|192.168.2.0/24|invalid|0|1|added|8
2012-05-27 01:39:02|192.168.0.0/24||||removed|11
2012-05-27 01:39:02|192.168.2.0/24||||removed|12
% Writing 0 Arp, 2 Route, 2 Mac events to the database

As you can see, it reacts pretty much the same way as when I added and removed static routes. Convinced that I could get the protocol field to show something other than invalid, I fired up a similar test using BGP. Again, I’m only showing the Arista side of things because this isn’t a lesson on routing, but rather a view into the workings of Event Monitor.

Here’s my quick and dirty BGP config, just to show that the protocol, metric, and preferences never change:

Arista(config)#router bgp 100
Arista(config-router-bgp)#neighbor 192.168.1.1 remote-as 100

And here’s the newly learned BGP route sitting snug in the routing table:

Arista(config-router-bgp)#sho ip route | begin Gateway
Gateway of last resort:
 S      0.0.0.0/0 [1/0] via 192.168.1.1

 S      20.20.20.0/24 [155/0] via 192.168.1.1
 C      30.30.30.0/24 is directly connected, Loopback0
 C      192.168.1.0/24 is directly connected, Vlan901
 B I    192.168.2.0/24 [200/0] via 192.168.1.1

And here’s the new route as seen in event-monitor:

Arista#sho event-monitor route
2012-05-27 02:00:51|192.168.2.0/24|invalid|0|1|added|18

Now I’ll remove BGP from the switch:

Arista(config)#no router bgp 100

And show that the BGP route is gone:

Arista(config)#sho ip route | beg Gateway
Gateway of last resort:
 S      0.0.0.0/0 [1/0] via 192.168.1.1

 S      20.20.20.0/24 [155/0] via 192.168.1.1
 C      30.30.30.0/24 is directly connected, Loopback0
 C      192.168.1.0/24 is directly connected, Vlan901

And here’s the final output from event-monitor:

Arista(config)#sho event-monitor route
2012-05-27 02:00:51|192.168.2.0/24|invalid|0|1|added|18
2012-05-27 02:01:44|192.168.2.0/24||||removed|19