Chapter 24. Scheduler
I was working in a data center, building out an Arista network, when I decided that I wanted to automate a command that should run every five minutes. Knowing that these super-cool Arista switches run Linux, I dropped into Bash and proceeded to muck around with cron. I couldn’t get it to work, became frustrated, and called my Arista sales engineer, who asked, “Why not just use schedule?”
Schedule is a feature that’s been around since version 4.5 of EOS that allows the regular scheduling of commands. The cool part of this feature is that it’s completely configured from the command-line interface (CLI), so you don’t need to spend any time swearing at cron. In this chapter, we take a look at this powerful tool.
The schedule command is the root of just about everything we’re going to do in this chapter. It’s simple to use, and the question mark (?) along with tab completion will get you most of what you want to know about its function:
Arista(config)#schedule ? WORD Scheduled job name config Set CLI scheduler configuration parameters
The only configuration options as of EOS 4.21.1F are max-concurrent-jobs, which you can set from 1 to 4, with the default being 1, and prepend-hostname-logfile, which does exactly what it sounds like it does and is now the default behavior:
Arista(config)#schedule config max-concurrent-jobs ? <1-4> Maximum number of concurrent jobs
To create a scheduled job, you first must specify a name for the job. Because I’m the writer and I love my daughter, I’ll use the name Colleen for my job. There are some cool new options with schedule that didn’t exist back in the dark ages when I wrote the first edition of Arista Warrior:
Arista(config)#schedule Colleen ? at Set the start time of the schedule interval Set interval for CLI command execution now Set the start time of the schedule to now
The interval is the amount of time in minutes to wait between each iteration of the job. If you want the job to run every five minutes, the interval would be 5. In version 4.21.1F, acceptable values range from 2 to 1440:
Arista(config)#schedule Colleen interval ? <2-1440> Interval in minutes for CLI command execution
Note that this is a significant change from earlier versions in which the range was 1 to 1440. Although that might not seem all that significant, the change is a result of the interval needing to now be at least twice the value of the timeout, which defaults to 30 seconds:
Arista(config)#schedule Colleen interval 5 timeout ? <1-480> Timeout in minutes for CLI command execution
In fact, if you set the interval to a value that is not greater than the timeout, the schedule will not be accepted:
Arista(config)#schedule Colleen interval 5 max-log-files 5 command sho ver ! Schedule a command starting in past % For job colleen, Interval 5 should be greater than the timeout 30
Also, that message, Schedule a command starting in past, is a cosmetic bug that happens when you don’t specify an at, which I cover in a minute. Let’s set a timeout of 4:
Arista(config)#schedule Colleen interval 5 timeout 4 ? max-log-files Set maximum number of logfiles to be stored
The next thing I need to specify is the maximum number of logfiles that will be retained. Every time my job runs, it creates output. That output is saved to a logfile that we take a look at in a bit. If the job runs every minute, for the next year, the job would produce a half-million log entries. The chances are that I’d want to see only the last 100 or so; thus, I’ll specify max-log-files 100, after which the switch will only save the last 100 log entries. The range of acceptable values is inclusive from 1 to 10,000:
Arista(config)#schedule Colleen interval 5 timeout 4 max-log-files ? <1-10000> Number of logfiles to be stored
Finally, we must include the word command, followed by the command to be run:
Arista(config)#schedule Colleen int 5 time 4 max-log-files 100 command sho ver ! Schedule a command starting in past Arista(config)#
Warning
Note that the parser does not check the validity of the commands entered with the schedule command. The command schedule mistake int 1 max 10 command ILikeCake will be accepted without complaint, but the job will never run successfully, and you’ll never see an error message (though there will be indications when you use show commands, as you’ll see), or get any cake. If you see No log files are stored in flash when you show the job, check to see whether your command is entered correctly. There is no indication as to whether the cake is a lie, so don’t bother asking.
With the job now entered, nothing obvious happens. That’s pretty anticlimactic, but rest assured that good things are happening (assuming that you entered a valid command). To see what jobs have been scheduled on the switch, use the show schedule summary command. Note that the word summary must be spelled in its entirety, lest it be confused with the name of a scheduled job:
Arista(config)#show schedule summary
Maximum concurrent jobs 1
Name At time Last Inter Max Logfile Location Status
time val log
(mins) files
------------ ------- ----- ------ ----- ----------------------------- ------
colleen now 04:51 5 100 flash:/schedule/colleen/ Success
tech-support now 04:23 60 100 flash:/schedule/tech-support/ Success
Note
Schedules can be entered with capital letters, but the parser will convert them to all lowercase. For a writer who expects to see his daughter’s name in all caps, this is frustrating, even maddening, but it’s generally harmless.
To see the specifics of a scheduled job, use the show schedule job-name command. This shows the command scheduled, the details of the schedule limitations as configured, and information regarding the logfiles generated as a result of this job. As we can see in this example, my job has run five times:
Arista(config)#sho schedule Colleen The last CLI command execution was successful CLI command "sho ver" is scheduled next at "20:47:13 01/08/2019", interval is 5 minutes Timeout is 4 minutes Maximum of 100 log files will be stored Verbose logging is off 5 log files currently stored in flash:/schedule/colleen Start time Size Filename ----------------------- ----------------- ------------------------------------- Jan 08 2019 20:42 290.0 bytes Arista_colleen_2019-01-08.2042.log.gz Jan 08 2019 20:37 290.0 bytes Arista_colleen_2019-01-08.2037.log.gz Jan 08 2019 20:32 290.0 bytes Arista_colleen_2019-01-08.2032.log.gz Jan 08 2019 20:27 291.0 bytes Arista_colleen_2019-01-08.2027.log.gz Jan 08 2019 20:25 290.0 bytes Arista_colleen_2019-01-08.2025.log.gz
See how I passive-aggressively insisted on using the capital C in the job name? That works because CLI is (more or less) case-insensitive. Colleen is a proper noun and thus has a capital C, dammit. You know what’s worse? The hostname that is injected by default includes the capital A in Arista!
Note
I asked the developers about the whole “forcing lowercase” thing, and the answer they gave me was basically, “I don’t remember why we did this but it’s been that way forever.” That’s not my favorite response, so I dug around in the super-secret internal documentation and found this: Please note that [the job] name is not case sensitive and is converted to lowercase before it is stored. It would appear that the reason for the lowercase job names is lost to history. The only thing I hate more than a mystery is a historical mystery, but even wearing my best fedora, I couldn’t unearth the answer in time for my publishing deadline. Not even considering the three four many deadlines that I missed!
Back in the day, if I decided to delete my schedule job, I would do so using the no schedule job-name command, and being the passive-aggressive type that I am, I would still include the uppercase letter; the command would be taken but nothing would happen. Now, the job name is just crushed…er, converted to lowercase, no matter how I use it, in order to make the command work:
Arista(config)#no schedule Colleen
Arista(config)#sho schedule summary
Maximum concurrent jobs 1
Name At time Last Inter Max Logfile Location Status
time val log
(mins) files
------------ ------- ----- ------ ----- ----------------------------- ------
tech-support now 04:23 60 100 flash:/schedule/tech-support/ Success
Arista(config)#
When you delete a scheduled job, the logfiles remain, and because these logfiles are on flash:, they persist after a reboot.
By the way, if you’re the type who needs to try to create a job titled “summary,” I like the way you think! I did exactly that in the first edition of this book, and EOS took it. Well, they’re on to me, and they fixed that and ruined my fun:
Arista(config)#schedule summary int 5 time 4 max 1 command sho int e1 ! Schedule a command starting in past % Job name cannot match keyword summary
You might have noticed that there is a job in each example named tech-support. This job is installed by Arista and runs on all switches by default. Though you can remove it, there’s generally no reason to do so. I’m sure you’re as curious as I was to see what this job is doing, so let’s take a look, because it’s an excellent example of how we can use schedule.
First, let’s use the show schedule tech-support command:
Arista(config)#sho schedule tech-support The last CLI command execution was successful CLI command "show tech-support" is scheduled next at "20:55:22 01/08/2019", interval is 60 minutes Timeout is 30 minutes Maximum of 100 log files will be stored Verbose logging is off 100 log files currently stored in flash:/schedule/tech-support Start time Size Filename ----------------------- ------------- ------------------------------------------ Jan 08 2019 19:53 74.8 KB Arista_tech-support_2019-01-08.1953.log.gz Jan 08 2019 19:11 80.6 KB Arista_tech-support_2019-01-08.1911.log.gz Jan 08 2019 18:11 80.5 KB Arista_tech-support_2019-01-08.1811.log.gz Jan 08 2019 17:11 80.5 KB Arista_tech-support_2019-01-08.1711.log.gz Jan 08 2019 16:11 80.5 KB Arista_tech-support_2019-01-08.1611.log.gz Jan 08 2019 15:11 80.6 KB Arista_tech-support_2019-01-08.1511.log.gz
As you can see in the second line, this scheduled job performs a show tech-support command once every hour. Take a look at the line in the preceding output that’s highlighted in bold that says there are 100 logfiles currently stored in flash:/schedule/tech-support. It then goes on to list them. How can we see what’s in these logfiles? There are a couple of ways. First, with CLI. By concatenating the path from the line in bold with one of the filenames from the list, we can see the contents of any of the files using the more command:
Arista#more flash:/schedule/tech-support/Arista_tech-support_2019-01-08.1911.log .gz ------------- show version detail ------------- Arista DCS-7280SR-48C6-M-F Hardware version: 21.05 Deviations: D0003570, D0003550 Serial number: SSJ17290598 System MAC address: 2899.3abe.9f92 [-- output massively truncated --]
I’d like to point out a very cool thing about what we just did, and that’s the fact that we used the more command to show the contents of a binary (g-zipped) file. You can’t do that in Bash, and this capability is a benefit of using CLI. If you used the bash more command, it would likely pollute your screen with control characters, after which you’d need to issue the reset command to clear it up. So how might we view the file in Bash? Let’s take a look:
To see the logfiles, drop into Bash and change directories to /mnt/flash/schedule/:
Arista#bash Arista Networks EOS shell [admin@Arista ~]$ cd /mnt/flash/schedule/ [admin@Arista schedule]$ ls colleen tech-support
We can see that there’s a directory for my Colleen (I’ll never give up!) job and one for the tech-support job. Changing into the tech-support directory lets us see the logs for the tech-support job:
[admin@Arista ~]$ cd tech-support [admin@Arista tech-support]$ ls Arista_tech-support_2019-01-04.2220.log.gz Arista_tech-support_2019-01-04.2320.log.gz Arista_tech-support_2019-01-05.0020.log.gz Arista_tech-support_2019-01-05.0120.log.gz Arista_tech-support_2019-01-05.0220.log.gz Arista_tech-support_2019-01-05.0320.log.gz Arista_tech-support_2019-01-05.0420.log.gz [--- output truncated ---]
Because the tech-support job is configured to keep the last 100 log entries, there are 100 files in this directory, each with the date and timestamp in the filename. The files are all in the gzip format. To view them, use the zmore (or zcat) Linux command from Bash. You can also use gunzip, but that will expand the file on disk, which is a waste of space when we just want to look at the contents for no other reason than to see what’s there:
[admin@Arista tech-support]$ zmore Arista_tech-support_2019-01-04.2320.log.gz ------> Arista_tech-support_2019-01-04.2320.log.gz <------ ------------- show version detail ------------- Arista DCS-7280SR-48C6-M-F Hardware version: 21.05 Deviations: D0003570, D0003550 Serial number: SSJ17290598 System MAC address: 2899.3abe.9f92 Software image version: 4.21.1F Architecture: i386 Internal build version: 4.21.1F-9887494.4211F Internal build ID: 1497e24b-a79b-48e7-a876-43061e109b92 Uptime: 0 weeks, 0 days, 1 hours and 5 minutes Total memory: 32458980 kB Free memory: 30481448 kB Installed software packages: Package Version Release -------------------------------------------- Aaa 1.1.2 9854397.4211F Aaa-cli 1.1.2 9854397.4211F Aaa-clientlib 1.1.2 9854397.4211F Aaa-lib 1.1.2 9854397.4211F Aboot-utils 1.0.4 9854397.4211F Acl 1.0.2 9854397.4211F Acl-cli 1.0.2 9854397.4211F Acl-lib 1.0.2 9854397.4211F AclAgent 1.0.3 9854397.4211F AclAgent-cli 1.0.3 9854397.4211F AclAgent-lib 1.0.3 9854397.4211F AclSnmp 1.0.1 9854397.4211F Adt7462 1.0.2 9854397.4211F Adt7462-lib 1.0.2 9854397.4211F --More--
This switch (and all other Arista switches out there) has done a show tech-support command every hour since the switch booted. There are 100 files in this directory, which represents the state of the switch in hourly intervals for the past 100 hours. That’s 4.166 days for the mathematically challenged. Had a problem yesterday and want to see the status of the switch? It was saved in these files. Think that’s cool and would like to see it save the last 1,000 hours? Just go in and resubmit the tech-support job the way you’d like.
Schedule is cool and all, but what if you want to run more than one command? Not only can the schedule command run CLI commands, but it can run bash commands through the use of the bash CLI command. Here I’ve created a script in /mnt/flash called GAD. I used all caps because it makes me feel like I’ve won my earlier battle and to show that caps can be used as a reference in the command, even if they can’t be used in the schedule job name.
The file looks like this:
[admin@Arista flash]$ cat GAD #!/usr/bin/Cli sho ver | grep version | email -s "Show Version" [email protected] sho int e24 | email -s "Show int e24" [email protected]
Note
For details on configuring and using the email command, see Chapter 22.
Next, back in EOS, I create the schedule that calls this script:
Arista(config)#schedule dog int 2 time 1 max 10 command bash /mnt/flash/GAD
And now, every minute, I get two emails showing the requested output. Trust me when I say that this gets old quickly:
+ N 2665 May 16 7:16pm [email protected] (943) Show Version + N 2666 May 16 7:16pm [email protected] (2K) Show int e24 + N 2667 May 16 7:18pm [email protected] (943) Show Version + N 2668 May 16 7:18pm [email protected] (2K) Show int e24 + N 2669 May 16 7:20pm [email protected] (943) Show Version + N 2670 May 16 7:20pm [email protected] (2K) Show int e24
Imagine that you’re having a problem, and you’d like to see what the running processes are on your switch every five minutes. Maybe you’re convinced that you’ve got a runaway process on your switch and you want to see for yourself. The Linux command ps -ef r shows the running processes:
[admin@Arista ~]$ ps -ef r UID PID PPID C STIME TTY STAT TIME CMD root 1454 1453 3 18:02 ? R 2:42 Sysdb root 1827 1453 6 18:03 ? R 4:54 Mdio admin 4058 3923 13 19:24 pts/2 R+ 0:00 ps -ef r
I can schedule a job to send me that information by email every five minutes. The schedule command that I would use would look like this:
Arista(config)#schedule proc int 5 time 3 max-log-files 1 command bashps -ef r | email -s "Running Procs" [email protected]
There in my email, every five minutes, is a list of the running processes on my switch (email and attachment concatenated in the interest of brevity):
Date: Fri, 25 Nov 2018 22:17:25 From: [email protected] To: [email protected] Subject: Running Procs UID PID PPID C STIME TTY STAT TIME CMD root 4143 4142 14 19:27 ? R 0:00 ps -ef r root 4144 4142 15 19:27 ? R 0:00 python /usr/bin/email -s Running Procs [email protected]
Conclusion
Hopefully you can see now that schedule can be pretty darn useful. Though it might be a bit less impactful now that tools like CloudVision and eAPI are available, it’s still a great way for network engineers to schedule commands without having to use another tool and without having to write any code.