A Note About Versions

In EOS 4.16 the internal behavior of this tool changed a bit, which made using SQL commands via the sqlite and interact commands problematic, and by problematic, I mean they no longer work. To show what I mean, on EOS 4.15 I could issue the following command to get information about the fields within the ARP table:

Arista#sho event-monitor sqlite .schema arp
CREATE TABLE arp( time text, prefix text, intf text, ethAddr text,
static integer, delta text,counter integer UNIQUE );

Issuing this command post 4.16 yields the following output:

Arista(config)#sho event-monitor sqlite .schema arp
% Cannot read from EventMon DB table

In fact, issuing any sort of more complicated sqlite command yields a similar output where it used to work on earlier code. If you really want to get that sort of information from Event Monitor, you can use Bash and interact using SQLite3 directly:

Arista#bash sqlite3   /var/log/eventMon.db '.schema arp'
CREATE TABLE arp (time text,prefix text,intf text,ethAddr text,static
 integer,delta text,counter integer UNIQUE);Arista#

Although that may seem unfortunate, especially if you’re used to using SQL commands to view your data, the good news is that (with the exception of the .schema command) there are better ways to get the information using command-line interface (CLI) commands that don’t require knowledge of SQL. It’s technically not as flexible, but in my experience, network engineers tend to favor using CLI commands anyway, so this isn’t really something to get riled up about.

With that out of the way, let’s take a look at each one of these tables.

ARP

Normally, ARP changes are not logged on a switch. Even if they were, scrolling through pages of log entries is not my idea of fun, so the idea of storing these events in a database is a bit appealing to me. It’s been a long time since I was a database guy, so let’s see how rusty I am. The way to retrieve the ARP events from the database is by using the show event-monitor arp command.

Arista#sho event-monitor arp
2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0
2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1
2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2
2019-01-09 20:24:36.011052|10.10.18.5|Ethernet32|28:99:3a:be:9d:42|0|added|3
2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4
2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5

My main problem with this tool is that the output is, shall we say, ugly. Well, it’s ugly from my “all user interfaces should be elegant and beautiful” point of view. From my “it’s SQL, and that’s what SQL output looks like” point of view, it’s beautiful. I know, I’m a complicated person.

Don’t get me wrong, this is very useful information, but I have no idea what some of the values are. For example, I see some fields with zeros in them and some with no values, and I have no idea what that last field in every line represents.

To help figure out what this stuff is, I’ll take advantage of a nifty SQLite command that shows me the command used to create the table. This command is called .schema, and we can access it by using the bash sqlite3 command. The database resides in var/log with the fairly logical name of eventMon.db. Each of the supported Event Monitor types are found in similarly named tables within this database. Here, I’m getting the schema for the arp table:

Arista#bash sqlite3 /var/log/eventMon.db ".schema arp"    
CREATE TABLE arp (time text,prefix text,intf text,ethAddr text,static integer,
 delta text,counter integer UNIQUE);

This output tells me that there was a table created named arp that contains the following fields, listed in order. I’ve also added what they mean for any nonprogrammer types out there:

Time (text string)

The time in which this log entry was added.

Prefix (text string)

The IP address related to the ARP entry.

Interface (text string)

What interface the ARP event occurred on.

Ethernet Address (text string)

The MAC address tied to the IP address (prefix).

Static (integer)

Value is 0 if ARP entry was dynamically learned, and 1 if statically assigned.

Delta (text string)

Typical entries are added and removed.

Counter (unique integer)

Every time an entry is made, it is assigned a counter value as a unique identifier for this record.

Let’s see this table in action. First, here is the list of existing ARP entries from one of my switches (which happens to be the one I used for the LANZ lab):

Arista-2#sho arp
Address         Age (min)  Hardware Addr   Interface
88.0.0.1              N/A  2899.3abe.a026  Ethernet48
10.0.0.95             N/A  001c.7328.2f6c  Management1
10.0.0.100            N/A  0cc4.7aa8.87ad  Management1

For this exercise, I add a static ARP entry to my switch by mapping the IP address 192.168.1.2 to the MAC address learned for 192.168.1.1:

Arista-2#conf
Arista-2(config)#arp 10.0.0.222 0014.6aa2.d438 arpa

Let’s see what the Event Monitor database shows now that I’ve added the static ARP:

Arista-2(config)#sho event-monitor arp
2019-01-16 22:42:42.091148|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0
2019-01-16 22:44:19.899883|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|1
2019-01-17 01:45:24.089620|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|2
2019-01-19 20:04:58.469700|88.0.0.1|Ethernet48|||removed|3
2019-01-19 20:05:05.205365|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|4
2019-01-19 20:06:30.330914|10.0.0.222|Management1|00:14:6a:a2:d4:38|1|added|5

Take a look at the line in bold, and let’s apply what we know. At 8:06 p.m. on January 19, 2019, a static ARP was added that mapped the IP address 10.0.0.222 to the MAC address 00:14:6a:a2:d4:38. The ARP entry became active on interface Management1.

Now, let’s delete the static ARP entry:

Arista-2(config)#no arp 10.0.0.222 0014.6aa2.d438 arpa

And here’s the output from the show event-monitor arp command after the change:

Arista-2(config)#sho event-monitor arp
2019-01-16 22:42:42.091148|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0
2019-01-16 22:44:19.899883|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|1
2019-01-17 01:45:24.089620|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|2
2019-01-19 20:04:58.469700|88.0.0.1|Ethernet48|||removed|3
2019-01-19 20:05:05.205365|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|4
2019-01-19 20:06:30.330914|10.0.0.222|Management1|00:14:6a:a2:d4:38|1|added|5
2019-01-19 20:08:09.111484|10.0.0.222|Management1|||removed|6

Reading the line in bold, we can see that at 8:08 p.m. on January 19, 2019, the ARP entry for the IP address 10.0.0.222 was removed. The ARP entry was previously active on interface Management1.

Remember that ARP entries are made on the switch only when the switch communicates directly using IP or if the ARP table is manually manipulated. If you have no IP addresses active on your switch, this table will be empty. For example, if you manage your switches solely through console servers and don’t even use the Management interface, you might not need to have IP addresses active on your switch at all. In this case, there would be no ARP activity to record.

Event Monitor includes CLI commands to help you organize this output in various ways without having to resort to grep or arcane Bash commands (which I recommend you learn, anyway). For example, you can change how the output is grouped instead of showing the default order, which is time based. This is done by using the group-by keyword, which as of EOS 4.21.1F has the options interface, mac, and ip. If we group by IP address, our output changes to look like this:

Arista#sho event-monitor arp group-by ip
2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0
2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1
2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5
2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4

Why are there fewer lines? Because this is the functional equivalent of the SQLite GROUP BY statement:

Arista#bash sqlite3 /var/log/eventMon.db 'select * from arp GROUP BY prefix;'
2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0
2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1
2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5
2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4

You can also show only specific matches using various match statements from CLI:

Arista#sho event-monitor arp match-interface e31
2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2
2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5

Using CLI like this is actually a very cool option because it allows abbreviation, which SQLite does not. Notice that I used e31 as the interface name. That doesn’t work using SQLite directly:

Arista#bash sqlite3 /var/log/eventMon.db "select * from arp where intf='e31';"
Arista#

To make that work we need the exact name as referenced in the table:

Arista#bash sqlite3 /var/log/eventMon.db "select * from arp where
 intf='Ethernet31';"
2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2
2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5

Hell, just looking at that SQL statement is probably enough for the typical network engineer to say, “yeah, I’m not doing that,” so having the CLI commands that wrap that up into network engineer–friendly syntax is wonderful. You can also match on IP address, MAC address, and time.

MAC

MAC changes are logged any time a MAC address is learned, added, or deleted. This can happen a lot on a busy switch, but it might not happen much at all on a smaller, stable network. This can be a pretty useful tool, so let’s look into how you can use it by taking a look at the table the same way we did for the ARP table. First, let’s look at the schema via the .schema mac SQLite command:

Arista-2#bash sqlite3 /var/log/eventMon.db ".schema mac"
CREATE TABLE mac (time text,fid integer,ethAddr text,intf text,type text,delta
 text,counter integer UNIQUE);

Here’s the breakdown of the fields:

Time (text string)

The time at which this log entry was added.

FID (integer)

FID stands for Filter ID. Switches keep an internal database of VLANs, and Arista switches assign internal VLANs to certain ports. This number will likely be unimportant to you because the interface on which the MAC address was learned is also present.

Ethernet Address (MAC) (text string)

The MAC address related to this event.

Interface (text string)

The interface on which this MAC event occurred.

Type (text string)

How did the entry change? A typical value would be learnedDynamicMac. When a MAC address is removed from the table, this field is null. If you statically add a MAC address by using the mac address-table static command, this field will read configuredStaticMac.

Delta (text string)

How did the entry change? Typical entries are added and removed.

Counter (unique integer)

Every time an entry is made, it is assigned a counter value as a unique identifier for this record.

So, let’s look at an example of how I like to use this feature. Generic Bob comes up to you at 4:30 p.m. on a Friday and says, “Hey, the network sucks.”

“What is it this time, Bob?” you ask, trying really hard not to roll your eyes, since you read that chapter in Network Warrior about how not to be that guy.

“My system doesn’t work, and it used to. Nothing changed, and I don’t feel like thinking, so it has to be the network.”

After a heavy sigh, you stop the important work you were doing and focus on Generic Bob’s dilemma. “Where are you connected, Bob?”

“How should I know? You’re the network guru, and it’s your crappy network, so you figure it out.”

Repressing the thoughts that are better left to mystery writers and psychopaths, you ask, “Well, do you know your IP address?”

“No, I think it’s dynamic or something. Will this take long?”

“Look Bob, I need either the IP address or the MAC address to find your system. Get me either of those, or even better, both, and I’ll see what I can find, OK?”

Generic Bob storms off to do his generic thing while you pull up your résumé. Just as you begin to include all of your Arista switch experience, Bob comes back and thrusts a piece of paper at your face. “Here’s your Mick address.”

You smile the false, hopeless smile of the damned, gently take the paper, and log in to your Arista switch. Generic Bob’s coffee-stained paper has 3c:07:54:43:88:d4 scribbled on it in what appears to be red crayon, or maybe lipstick.

With Event Monitor, you have a record of every MAC event for quite some time. Using the match-mac keyword, you pull up the event-monitor mac table including only events regarding the MAC address you care about:

Arista#sho event-monitor mac match-mac 3c:07:54:43:88:d4
2018-11-26 09:18:08|1006|3c:07:54:43:88:d4|Ethernet24|learnedDynamicMac|added|549
2019-01-03 16:19:51|1|3c:07:54:43:88:d4|Ethernet6|learnedDynamicMac|added|553

Looking at this data, you can see that Generic Bob’s MAC address was first learned on Ethernet24 on November 26, 2018. Because you know that e24 is an uplink to another switch, you know that Bob’s system was originally on another switch.

But the next line shows that the MAC address was learned today (the first day back from work after the new year), at 4:19 p.m. on the switch’s local interface, e6. That seems odd, so you look at the running-config for e6 on this switch:

Arista#sho run int e6
interface Ethernet6
   description [ Unused ]
   switchport access vlan 999

Because you’ve configured the switch with a dead VLAN (999) that you put on all unused ports, you’re sure that Generic Bob glommed a free interface on a different switch than the one to which he was assigned, likely due to either a massive hangover or a natural tendency for Generic Bob to be a pain in the ass. Or both. Yeah, it was probably both.

With a couple of short commands, you’ve shown that Generic Bob was lying because someone had clearly moved his system from one switch to another, and that’s why it stopped working. You can’t blame the guy for thinking the new Arista switches would give him better performance, but plugging into network ports without prior authorization should not be tolerated.

I’ll leave it to your imagination as to how you might deal with Generic Bob now that you have the tools to prove him wrong. Just try not to be that guy...much. I’d probably just smile and tell him to put his system back into the network port where it belonged, and then make sure his access to the data center was revoked, but not until I was done adding Event Monitor to my résumé.

As a fun aside, I asked the Arista customer engineers if they saw this feature being used in the wild and got a pretty significant response saying that it was. I also got a reply that basically reinforced that there are Generic Bobs out there and that this feature managed to save a network engineer’s job after said engineer was blamed for an outage that wasn’t his fault. Now that’s a useful feature, and I bet that engineer has been praising Arista ever since.

Route

If you have your switches configured for IP routing, the show event-monitor route command can be a great tool to see historical information regarding route changes. As with the others, let’s take a look at the schema and then work up an example to see it in action:

Arista#bash sqlite3 /var/log/eventMon.db '.schema route'
CREATE TABLE route (time text,prefix text,protocol text,metric integer,preference
 integer,delta text,counter integer UNIQUE);

Here’s the breakdown of the fields from the route table:

Time (text string)

The time in which this log entry was added.

Prefix (text string)

The IP prefix, including the Classless Internet Domain Routing (CIDR) mask related to the route entry.

Protocol (text string)

I would imagine that this should read the protocol from which the prefix was learned or removed, but I’ve only ever seen this field read invalid when a route is added, or null when a route is removed.

Metric (integer)

Again, I would assume this field should contain the metric for the route, but I’ve never been able to make it display anything other than 0 when adding a route, and null when removing a route.

Preference (integer)

This field should include the administrative distance for the route, but in my testing, this field reports a 1 regardless of route type learned.

Delta (text string)

How did the entry change? Typical entries are added and removed.

Counter (unique integer)

Every time an entry is made, it is assigned a counter value as a unique identifier for this record.

Here is a sample output from one of my switches:

Arista#sho event-monitor route
2019-01-22 18:48:51.911482|127.0.0.0/8|martian|0|1|added|0
2019-01-22 18:48:51.911524|127.0.0.1/32|martian|0|1|added|1
2019-01-22 18:48:51.911565|0.0.0.0/8|martian|0|1|added|2
2019-01-22 18:48:51.931530|10.0.0.255/32|receiveBcast|0|0|added|3
2019-01-22 18:48:51.931616|10.0.0.0/32|receiveBcast|0|0|added|4
2019-01-22 18:48:51.931685|10.0.0.0/24|connected|1|0|added|5
2019-01-22 18:48:51.931749|10.0.0.14/32|receive|0|0|added|6

Wait, is my switch talking to Martians?

In networking parlance, a Martian Packet is a packet seen on the public internet using a private address that is thus considered to be “not of this Earth,” so the packets must be sourced from Martians. Usually they’re from hackers or poorly configured Border Gateway Protocol (BGP) routers, but calling them Martians is much more fun.

To take that a step further, Bogons are Martian IP addresses and IP addresses that have not been allocated (0.0.0.0/8 and 169.254.0.0/16 being two examples) and as such should also not be seen on the public internet. It has become best practice to filter Bogons on BGP edge routers because they’re usually associated with hacker attacks.

The terms Bogon and Martian are essentially interchangeable today, at least when it comes to internet routing tables. From the point of view of our switch IP routing tables, though, things like 0.0.0.0/8 and 127.0.0.0/8 should only be seen locally, whereas 10.0.0.0/8 might appear from elsewhere on the network. The important distinctions are scale and point of view. The switch might not have anything to do with the internet, so the RFC1918 addresses might very well be legitimate, whereas routes to localhost certainly would not be.

Let’s begin with a simple example. Here, I add a static route to the switch:

Arista(config)#ip route 20.20.20.0/24 10.0.0.1

Using the match-ip ip-address option, here’s what shows up in the Event Monitor:

Arista(config)#sho event-monitor route match-ip 20.20.20.0/24
2019-01-22 20:27:09.835624|20.20.20.0/24|staticConfig|0|1|added|7

Now I delete that route:

Arista(config)#no ip route 20.20.20.0/24 192.168.1.1

And an appropriate remove message shows up in the Event Monitor:

Arista(config)#sho event-monitor route match-ip 20.20.20.0/24
2019-01-22 20:27:09.835624|20.20.20.0/24|staticConfig|0|1|added|7
2019-01-22 20:29:00.837986|20.20.20.0/24||||removed|8

When the switch learns routes through other means, the results are pretty much the same, though the Protocol field will of course change to match how the route was learned.

LACP

LACP (802.3ad) is defined as an IEEE standard that, so far as I can tell, cannot be easily read on the internet due to the IEEE wanting us to pay for the specifications to this open standard. That is infuriating, so I’ll leave it to you to determine whether there are any number of ways to get around that while I sit here and grumble to myself about the IEEE and its supposedly open standards.

The output of this event-monitor is filled with things that might not make much sense if you don’t have a copy of the IEEE specification, and to make matters more complicated, the internal Arista documents state, “The primary purpose of logging LACP events is to allow for post-mortem failure analysis of the Receive and Mux State Machines…” Quick, what is the purpose of the mux State Machine in LACP? No problem; I’ll just check out my copy of the IEEE 802.3ad specification…oh, right.

I have included only a small sample output for the show event-monitor lacp command because the resulting text is so wide that there was really no hope of formatting it well in a book. Here are a couple of lines to show what I mean:

DC4-2(config)#sho event-monitor lacp
2019-01-22 21:16:07.444659|Ethernet33|unknown|rx|rxSmInitialize|12|0|0|0|0|0|
selectedStateUnselected|0|1
2019-01-22 21:16:07.444815|Ethernet33|unknown|rx|rxSmPortDisabled|1|0|0|0|0|0|
selectedStateUnselected|0|2
 [--output truncated--]

This command is a little different from the other event-monitor options in that it also has a verbose option. Note that the output has been significantly altered to fit on the page. I’ve altered the date format and collapsed the width considerably:

DC4-2#sho event-monitor lacp verbose
Time      Port  LAG    State         Reason
-------- ---- ------ ------------ ----------------------------------------------
21:16:07 Et33        Initialize   Initializing state machine from Sysdb state    
21:16:07 Et33        PortDisabled Unconditional transition                        
21:16:07 Et33        LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE
21:16:07 Et33 Po1    Expired      LACP has been re-enabled                        
21:16:07 Et48        Initialize   Initializing state machine from Sysdb state    
21:16:07 Et48        PortDisabled Unconditional transition                        
21:16:07 Et48        LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE
21:16:07 Et48 Po1000 Expired      LACP has been re-enabled                        
21:16:07 Et47        Initialize   Initializing state machine from Sysdb state    
21:16:07 Et47        PortDisabled Unconditional transition                        
21:16:07 Et47        LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE
21:16:07 Et47 Po1000 Expired      LACP has been re-enabled  
[--output truncated--]

Here’s what a couple of lines look like wrapped with their original widths and content:

DC4-2#sho event-monitor lacp verbose
Time                        Port  LAG   State         Reason
--------------------------- ----- ----- ------------- ---------------------------
2019-01-22 21:16:07.444659  Et33        Initialize    Initializing state machine
                                                      from Sysdb state        
2019-01-22 21:16:07.444815  Et33        PortDisabled  Unconditional transition                          
2019-01-22 21:16:07.478546  Et33        LacpDisabled  portEnabled changed to TRUE
                                                      with lacpEnabled=FALSE
2019-01-22 21:16:07.479061  Et33  Po1   Expired       LACP has been re-enabled  
[--output truncated--]

As you can see, the verbose option is well named, but it’s also significantly more useful, at least in my opinion. Here is the table layout for LACP in Event Monitor:

Arista(config)#bash sqlite3 /var/log/eventMon.db '.schema lacp'
CREATE TABLE lacp (time text,intf text,portchannel text,statemachine text,state
 text,reason int16,portenabled int16,lacpenabled int16,currentwhiletimerexpired
 int16,lacpdurx int16,portmoved int16,selected text,partnersync int16,counter
 integer UNIQUE);

Some of these entries are a bit esoteric due to them being flags or states, as specified in the LACP IEEE specification that I don’t have.

Time (text string)

The time at which this log entry was added.

Interface (text string)

The interface involved in this entry.

Portchannel (text string)

The port-channel interface involved in this entry.

Statemachine (text sting)

The section of the state-machine involved in this entry. Examples include rx and mux.

State (text string)

This is the new state after the state change.

Reason (16-bit integer)

Reason for change. What are the reason numbers? Check with the IEEE.

PortEnabled (16-bit integer)

Is the port enabled? 0 = no, 1 = yes.

LacpEnabled (16-bit integer)

Is LACP enabled? 0 = no, 1 = yes.

CurentWhileTimerExpired (16-bit integer)

0 = no, 1 = yes.

LacpDurx (16-bit integer)

Has LACPDU been received? 0 = no, 1 = yes.

PortMoved (16-bit integer)

Has port moved? 0 = no, 1 = yes.

Selected (text string)

SelectedStateUnselected, selectedStateStandby, or selectedStateSelected.

PartnerSync (16-bit integer)

Partner Sync flag 0 = false, 1 = true.

Counter (unique integer)

A unique counter for this entry.

IGMP Snooping

This table is much simpler than the last couple of examples:

Arista#bash sqlite3 /var/log/eventMon.db '.schema igmpsnooping'
CREATE TABLE igmpsnooping (time text,vlan text,ethAddr text,intf text,delta
 text,counter integer UNIQUE);

Here’s the breakdown of the fields from the igmpsnooping table:

Time (text string)

The time in which this log entry was added.

VLAN (text string)

The VLAN number on which this entry occurs.

Ethernet Address (text string)

The MAC address.

Interface (integer)

The interface.

Delta (text string)

How did the entry change? Typical entries are added and removed.

Counter (unique integer)

Every time an entry is made, it is assigned a counter value as a unique identifier for this record.

MRoute

If you have multicast running, this table will show the changes to the multicast route (ip mroute) table similar to the way that the regular route table changes are logged.

Arista#bash sqlite3 /var/log/eventMon.db '.schema mroute'
CREATE TABLE mroute (time text,vrf text,sourceIp text,groupIp text,intf
 text,intfType text,delta text,counter integer UNIQUE);
Arista#

Here’s the breakdown of the fields from the route table:

Time (text string)

The time at which this log entry was added.

Virtual Routing and Forwarding (VRF) instance (text string)

The VRF in which this mroute resides.

SourceIP (text string)

The source IP address for this entry.

GroupIP (text string)

The group IP address for this entry.

Intf (text string)

The name of the interface for this entry.

IntfType (text string)

The type of the interface (iif/oif).

Delta (text string)

How did the entry change? Typical entries are added and removed.

Counter (unique integer)

Every time an entry is made, it is assigned a counter value as a unique identifier for this record.