10. cron and at
The cron and at packages provide Linux users with a method for scheduling jobs. at is used to run a job once. cron is used to run jobs on a schedule. If a report should be run every Friday at 8 p.m., cron is perfect for the job. If a user wants to run a sweep of the system to find a misplaced file, the job can be scheduled to run that evening using the at command.
This chapter explains how cron and at work. You might not be familiar with the anacron and kcron packages, but they extend the features of cron. We also explain these tools in this chapter. We show how to use cron and the other utilities, but we also show how they work and provide examples of what can go wrong.
This chapter is organized into the following sections:
• cron—The basics of cron are explained. The crontab command is used to submit and edit jobs. The reader will see the various crontab syntaxes and the format of the cron configuration file. The other files cron uses are explained as well. The cron daemon runs the jobs submitted with crontab. This topic details how the daemon gets started, where it logs, and the differences between cron packages. We also discuss a graphical front end to crontab called kcron.
• anacron—anacron is a utility to run jobs cron missed due to system downtime. Learn how it works in this section.
• at—at is a utility similar to cron that runs jobs once. We show examples of submitting, removing, and monitoring jobs with at.
We conclude the chapter with a section on four troubleshooting scenarios that demonstrate good methodologies for fixing problems with cron.
cron
The crontab command is used to manage cron jobs. The -l option displays the current list of jobs for the user. The following example runs the report mentioned in this chapter’s introduction:
[dave@sawnee dave]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.9889 installed on Fri Oct 15 09:42:06 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
0 20 * * fri /home/dave/acct_prod_rpt.sh
The crontab command is the only user command that cron provides. There is a cron daemon to run jobs, but only crontab is run from the command line by users. It is used to create, edit, or display the crontab configuration file, which is called a crontab file for a user. Switches available for the crontab command are shown in Table 10-1.
We recommend using crontab -l rather than crontab -e to make crontab changes. For example:
$ crontab -l >crontab.out # Saves the crontab file
$ cp crontab.out crontab.bkup # Create backup
$ vi crontab.out # Make desired changes
$ crontab crontab.out # Activate the changes
$ crontab –l # Verify changes
It took us five commands to modify the crontab file, but we do have a backup of the original as well as a copy of the new crontab. We could have just run crontab -e to edit the crontab file. When you run crontab -e for the first time, an editor starts with an empty document. Add the job command lines and save the file, and a crontab file is created for the user. If you would rather not use the default editor, set the EDITOR environment variable first. For example, run export EDITOR=vim;crontab -e to edit the crontab file with the vim editor.
The crontab entries can be set using monthly, weekly, daily, and hourly time definitions. The crontab(5) man page shows the format of the crontab lines. The time and date fields from the man page are shown in Table 10-2.
Table 10-2. Time and Date Fields and Allowable Values
Follow the scheduling fields with the command to be run. The following crontab entry runs a backup process every day at 11:00 p.m.:
00 23 * * * /prod_serv/scripts/prod_backup_2
This entry runs a report at 6:15 a.m. on the first of every month:
15 6 1 * * /prod_serv/scripts/monthly_reports
This one runs a report every weekday at 2 p.m.:
0 14 * * 1-5 /home/ted/daily_rpt.sh
The crontab command validates the formatting of the command lines. Look at the output from crontab -e when the minute and hour fields are reversed:
[ted@sawnee ted]$ crontab -e
9 50 * * * /home/ted/sweep.sh
~
~
~
~
no crontab for ted - using an empty one
crontab: installing new crontab
"/tmp/crontab.1279":0: bad hour
errors in crontab file, can't install.
Do you want to retry the same edit?
Consider inserting this snippet from the crontab(5) man page as a comment block in your crontab files so that it is easy to understand the format of the job command lines:
########################################################################
# field allowed values
# ------ --------------
# minute 0–59
# hour 0–23
# day of month 1–31
# month 1–12 (or names, see below)
# day of week 0–7 (0 or 7 is Sun, or use names)
# min hour day month wkday
########################################################################
crontab does not validate the paths of commands, however, as you can see here:
[ted@sawnee ted]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.26370 installed on Mon Aug 30 08:57:28 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
DBHOME="/prod_serv"
03 09 * * * /home/ted/sweeper.sh
crontab accepted the file, but look at the email Ted receives when it is executed by cron:
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <ted@sawnee> /home/ted/sweeper.sh
X-Cron-Env: <DBHOME=/prod_serv>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/ted>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=ted>
/bin/sh: line 1: /home/ted/sweeper.sh: No such file or directory
Environment variables can be added to the crontab file as well as comments. cron sets very few environment variables. The path is set to /usr/bin:/bin, and the shell is /bin/sh. If environment variables from the shell are needed, they must be set manually in the crontab file. For example, the following crontab sets DBHOME, DBINST, and the PATH variable:
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.1828 installed on Mon Sep 13 17:26:58 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
DBHOME=/prod_serv/scripts
DBINST=prod
PATH=/home/susan/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/gam
es:/opt/gnome/bin:/opt/kde3/bin:
30 17 * * * /prod_serv/scripts/prod_daily_report
Output from a cron job is mailed to the owner of the crontab. There is a cron environment variable MAILTO. If set, any cron email goes to the user specified by MAILTO. If MAILTO="", no email is sent by cron. For further information, see the crontab(5) man page.
cron and at are similar but not identical among Linux distributions. We show Red Hat and SUSE examples and provide techniques to understand the differences in other Linux distributions.
The crontab files are stored in the /var/spool/cron directory in Red Hat and /var/spool/cron/tabs in SUSE. The directory contains one file for every user with a cron schedule. The user name is the name of the crontab file. The crontab(1) man page has more detailed information.
The capability to use crontab can be restricted by the system administrator by creating an allow or deny file. In Red Hat, the files are /etc/cron.allow and /etc/cron. deny. In SUSE, the files are /var/spool/cron/allow and deny. The allow file is a list of users who can run the crontab command. The deny file is a list of users who cannot. Both files are just lists of user names. For example:
$ cat /etc/cron.allow
dave
root
If the allow file exists but does not contain dbuser, then the database administrator (DBA) sees the following message when trying to run crontab while logged in as dbuser:
$ crontab -e
You (dbuser) are not allowed to use this program (crontab)
See crontab(1) for more information
If the deny file exists and the allow file does not exist, all users can run crontab except those listed in the deny file. The allow file takes precedence. If both allow and deny exist, a user listed in both files can use crontab. To prevent any user from running crontab, including root, create an empty allow file. The at command uses /etc/at.allow and /etc/at.deny and follows the same rules as crontab.
The directory permissions on /var/spool/cron or /var/spool/cron/tabs limit access to only the root user to prevent users from manually creating or editing a crontab file. This forces users to use only the crontab command to change their crontab files. The system administrator should not manually edit these files either.
# ls -al /var/spool/cron
total 16
drwx------- 2 root root 4096 Aug 11 10:54 .
drwxr-xr-x 16 root root 4096 Nov 6 2003 ..
-rw------- 1 root dave 254 Jul 31 12:53 dave
-rw------- 1 root dbadmin 280 Aug 11 10:51 dbadmin
The crontab command has the setuid bit set so that non-root users can edit their own crontab files:
# ls -l /usr/bin/crontab
-rwsr-xr-x 1 root root 110114 Feb 19 2003
/usr/bin/crontab
Try to lock down the crontab command by changing the permissions to 555, and you receive the following error when running it as a non-root user:
[dave@sawnee dave]$ crontab -e
seteuid: Operation not permitted
Root has a special crontab file, /etc/crontab. The format is slightly different. The job command line has an additional field to specify what user cron should use to run the job. The default Red Hat /etc/crontab is:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
The /etc/crontab supplied with Red Hat Linux makes use of the run-parts script. On a Linux installation, there are probably several processes that should be run daily such as backups and reports. There can be weekly or monthly jobs such as reports, software updates, system scans, and so on. This is the beauty of the run-parts script. The /etc/crontab shown previously runs the daily jobs at 4:02 a.m., the weekly jobs at 4:22 a.m. on Sunday, and the monthly jobs at 4:42 a.m. on the first day of the month.
The /usr/bin/run-parts script expects to be supplied with a directory path as an argument. The run-parts script executes all the scripts contained in that directory except for those ending in rpmsave, rpmorig, rpmnew, and swp scripts. There is no man page for run-parts, but it is a simple script to read and understand.
The functionality provided by /usr/bin/run-parts is very important. A software package that includes a cron job can be written to place a file in the appropriate /etc/cron.hourly, cron.daily, cron.weekly, or cron.monthly directory rather than trying to manipulate root’s crontab file. This capability makes creating packages containing cron jobs manageable. You can see which packages delivered cron jobs to a Linux system with the command rpm -qa --filesbypkg grep '/etc/cron'.
The SUSE /etc/crontab file is similar to Red Hat’s /etc/crontab:
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/
cron/run-crons >/dev/null 2>&1
59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly
14 4 * * * root rm -f /var/spool/cron/lastrun/cron.daily
29 4 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly
44 4 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly
SUSE Linux supplies the run-crons script. It runs the jobs in the /etc/cron.* directories like run-parts does on Red Hat. cron on SUSE is configured to use /var/spool/cron/lastrun to keep track of the last time that the /etc/cron.* scripts were run. cron does not log the run-crons job execution.
Make sure only root can write to the /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, and /etc/cron.monthly directories. Otherwise, any user can save a script in one of these directories and use cron to run his or her process with root privileges. Notice that the command line indicates that root should run these scripts. Even if the script in /etc/cron.daily is owned by a user, it is executed with root privileges.
# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 Feb 19 2003 /etc/cron.d
drwxr-xr-x 2 root root 4096 Aug 10 04:08 /etc/cron.daily
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.hourly
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.monthly
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.weekly
Note
You can confirm that the permissions of a package are correct with the rpm –V cron_package_name command. You can confirm the permissions of a single file or directory with rpm –V -f filename. Troubleshooting using rpm is covered later in this chapter.
Here is a sample /etc/cron.daily:
# ll /etc/cron.daily
total 36
-rwxr-xr-x 1 root root 135 Jan 25 2003 00webalizer
-rwxr-xr-x 1 root root 276 Jan 24 2003 0anacron
-rwxr-xr-x 1 root root 123 Jan 26 2003 inn-cron-expire
-rwxr-xr-x 1 root root 51 Jan 24 2003 logrotate
-rwxr-xr-x 1 root root 418 Feb 10 2003 makewhatis.cron
-rwxr-xr-x 1 root root 135 May 14 15:59 rpm
-rwxr-xr-x 1 root root 132 Jan 21 2004 slocate.cron
-rwxr-xr-x 1 root root 164 May 17 09:18 sophos
-rwxr-xr-x 1 root root 193 Feb 10 2003 tmpwatch
cron Daemon
The cron daemon (/usr/sbin/cron for SUSE and /usr/sbin/crond for Red Hat) is responsible for running the cron jobs. Every minute, the cron daemon looks at the crontabs, loads new or modified crontabs into memory, and checks whether any jobs are scheduled to execute during that minute. The cron daemon is started during the startup scripts at runlevels 2, 3, 4, and 5.
#chkconfig --list crond
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The startup script is /etc/init.d/crond in Red Hat and /etc/init.d/cron in SUSE. For more information on startup scripts, refer to Chapter 1, “System Boot, Startup, and Shutdown Issues.”
A log entry is created when cron jobs are run, crontab commands are used, the cron daemon is started, and so on. cron uses the syslog daemon to log cron events. The syslog facility for cron messages is cron. See the syslogd(8) man page for an explanation of how syslog routes messages. The log file for cron is determined by /etc/syslog. conf. Red Hat has a log just for cron, /var/log/cron, to log cron messages. SUSE uses /var/log/messages. A Red Hat /etc/syslog.conf has the following entry to route cron messages to /var/log/cron:
# Log cron stuff
cron.* /var/log/cron
With Red Hat, cron messages are not duplicated in /var/log/messages because of the cron.none on the following syslog.conf line:
*.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages
It would be a simple matter to change SUSE systems to log cron messages to /var/log/cron. Here is a sample of the entries that are written to Red Hat’s /var/log/cron file:
Aug 15 04:22:00 sawnee CROND[25198]: (root) CMD (run-parts
/etc/cron.weekly)
Aug 15 04:22:04 sawnee anacron[25202]: Updated timestamp for job
'cron.weekly' to 2004-08-15
Aug 15 05:01:00 sawnee CROND[2420]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 06:01:00 sawnee CROND[2519]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 07:01:00 sawnee CROND[2532]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 08:01:00 sawnee CROND[2544]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 09:01:00 sawnee CROND[2556]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 10:01:00 sawnee CROND[2568]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 11:01:00 sawnee CROND[2588]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 12:01:00 sawnee CROND[2600]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 13:01:00 sawnee CROND[2612]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:01:00 sawnee CROND[2910]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:35:46 sawnee crontab[3107]: (dave) LIST (dave)
Aug 15 14:36:56 sawnee crontab[3113]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) REPLACE (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) END EDIT (dave)
Aug 15 14:37:09 sawnee crontab[3116]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) REPLACE (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) END EDIT (dave)
Aug 15 14:38:00 sawnee crond[1792]: (dave) RELOAD (cron/dave)
Aug 15 14:38:00 sawnee CROND[3166]: (dave) CMD (/home/dave/daily_rpt)
The sample cron log shows cron activity on host sawnee. cron events are broken out by crond, crontab, and CROND (which shows jobs being started). The number between the [] is the process ID. Everything that crontab performs is logged. Note that run-parts is listed as one job and is not broken down by each job in /etc/cron.hourly.
As we noted earlier, cron and at are similar but not identical across Linux distributions. To understand the differences, look closely at the man pages, see which packages are installed, and look for additional documentation supplied with the packages. Run rpm -qa to see what packages are installed. This list is from a Red Hat 9.0 system:
# rpm -qa|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25
A typical SUSE 9.0 system might show:
# rpm -qa|grep cron
cron-3.0.1-824
The package name doesn’t indicate it, but this is vixie-cron too, which can be verified in the README supplied with the package.
Look at the files delivered with the packages by running rpm -q --filesbypkg. Take note of man pages, READMEs, and other documentation provided with each package. You can see what commands are available with the package as well as what documentation you can review. Here are the files delivered with Red Hat 9.0:
# rpm -q --filesbypkg vixie-cron-3.0.1-74
vixie-cron /etc/cron.d
vixie-cron /etc/rc.d/init.d/crond
vixie-cron /usr/bin/crontab
vixie-cron /usr/sbin/crond
vixie-cron /usr/share/man/man1/crontab.1.gz
vixie-cron /usr/share/man/man5/crontab.5.gz
vixie-cron /usr/share/man/man8/cron.8.gz
vixie-cron /usr/share/man/man8/crond.8.gz
vixie-cron /var/spool/cron
# rpm -q --filesbypkg crontabs-1.10-5
crontabs /etc/cron.daily
crontabs /etc/cron.hourly
crontabs /etc/cron.monthly
crontabs /etc/cron.weekly
crontabs /etc/crontab
crontabs /usr/bin/run-parts
SUSE 9.0 delivers the following files with the cron package:
# rpm -q --filesbypkg cron-3.0.1-824
cron /etc/crontab
cron /etc/init.d/cron
cron /usr/bin/crontab
cron /usr/lib/cron
cron /usr/lib/cron/run-crons
cron /usr/sbin/cron
cron /usr/sbin/rccron
cron /usr/share/doc/packages/cron
cron /usr/share/doc/packages/cron/CHANGES
cron /usr/share/doc/packages/cron/CONVERSION
cron /usr/share/doc/packages/cron/FEATURES
cron /usr/share/doc/packages/cron/MAIL
cron /usr/share/doc/packages/cron/MANIFEST
cron /usr/share/doc/packages/cron/README
cron /usr/share/doc/packages/cron/THANKS
cron /usr/share/man/man1/crontab.1.gz
cron /usr/share/man/man5/crontab.5.gz
cron /usr/share/man/man8/cron.8.gz
cron /var/spool/cron
cron /var/spool/cron/deny
cron /var/spool/cron/lastrun
cron /var/spool/cron/tabs
Listing the files delivered in the package makes it easy to see the differences between distributions.
kcron
A nice graphical front-end utility is available for the crontab command. The kcron utility is part of KDE and is supplied with the optional kdeadmin package. The kcron utility is a simple, intuitive method for users to manage their cron jobs. The English version of the kcron handbook is available at http://docs.kde.org/en/3.1/kdeadmin/kcron/index.html. To start kcron, choose System Tools, Task Scheduler from the main KDE menu, and you will see the interface shown in Figure 10-1.
Figure 10-1. The kcron interface
No crontab knowledge is needed to schedule jobs when kcron is used. Users don’t have to learn vi just to schedule a job. kcron inserts a header and a footer in the crontab file, so the system administrator knows it was saved from kcron. Here is an example:
$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/kde-susan/kcronzMmGJa.tmp installed on Tue Sep 7 09:47:40 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
#
5 4 * * * /home/susan/screening.sh
# This file was written by KCron. Copyright (c) 1999, Gary Meyer
# Although KCron supports most crontab formats, use care when editing.
# Note: Lines beginning with "#" indicates a disabled task.
There are a couple of wrinkles with kcron. kcron does not seem to be aware of the crontab allow and deny files, meaning users who are restricted from using the crontab command can still create cron tasks.
If kcron is run as root, the save function saves crontab files for every user on the system even if only root’s jobs were modified. If the user already has a crontab file, kcron inserts its header and footer. If the user doesn’t have a crontab file, kcron creates one with the normal crontab header plus the kcron header line. We don’t recommend using kcron as root, but it is fine for other users.
To see the list of known bugs in kcron, visit http://bugs.kde.org/. We submitted new bug reports for those we mentioned here:
Bug 89488:
kcrondoesn’t prohibitcrontabchanges based oncron’s allow and deny files.Bug 89491:
kcroncreatescrontabfiles for all users when the rootcrontabis created.
More information about KDE is available at http://www.kde.org/.
anacron
anacron is similar to cron, and it works much the same way. anacron is intended to run jobs in /etc/crontab that cron missed due to system downtime. If a Linux box is left on continuously, anacron won’t run any jobs because cron will run them. anacron has its own crontab file, /etc/anacrontab. Changes made to /etc/crontab need to be made in /etc/anacrontab as well. anacron is a utility for root only. There is no anacrontab command to edit the file for users. The system administrator should edit the file manually. Here is the anacrontab supplied with anacron on Red Hat systems:
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
1 65 cron.daily run-parts /etc/cron.daily
7 70 cron.weekly run-parts /etc/cron.weekly
30 75 cron.monthly run-parts /etc/cron.monthly
The format is similar to the crontab files, but there are some differences. The first few lines set the shell and path. The remaining lines contain the scripts to execute; anacron calls these job description lines.
The first field of the job description line indicates how many days should occur between runs of the process. The second field indicates how many minutes to wait before starting the process. The third field is the job identifier. The last field and remainder of the line is the shell command to execute.
See the anacrontab(8) man page for further details.
The anacrontab file doesn’t execute run-parts /etc/cron.hourly because the smallest time unit for anacron is one day.
There is a potential problem with the previous anacrontab file. The full path to the run-parts script is not given. The run-parts script is in /usr/bin. The /usr/local/sbin and /usr/local/bin directories are searched before /usr/bin according to the PATH variable. If users have write permission to /usr/local/sbin or /usr/local/bin, a script called run-parts could be placed in those directories and would run instead of the intended /usr/bin/run-parts. This issue can be prevented by always using the full path for scripts in anacrontab. For example:
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
1 65 cron.daily /usr/bin/run-parts /etc/cron.daily
7 70 cron.weekly /usr/bin/run-parts /etc/cron.weekly
30 75 cron.monthly /usr/bin/run-parts /etc/cron.monthly
anacron is started by a startup script.
#chkconfig --list anacron
anacron 0:off 1:off 2:on 3:on 4:on 5:on 6:off
anacron checks the timestamp files in /var/spool/anacron and runs any jobs that were missed.
# ls -al /var/spool/anacron
total 20
drwxr-xr-x 2 root root 4096 Nov 6 2003 .
drwxr-xr-x 16 root root 4096 Nov 6 2003 ..
-rw------- 1 root root 9 Aug 14 12:53 cron.daily
-rw------- 1 root root 9 Aug 1 04:42 cron.monthly
-rw------- 1 root root 9 Aug 8 04:22 cron.weekly
anacron records whether cron is running normally so that jobs are not run twice. This is done with an anacron script in /etc/cron.daily, /etc/cron.weekly, and /etc/cron.monthly, which is executed by cron. The anacron -u command is run by this anacron script to update the timestamps and indicate that all jobs were run for that day by cron.
anacron messages route to the same log file as cron messages. anacron logs messages using syslogd with the facility value set to cron. Here are some sample anacron log entries:
Aug 18 10:43:58 sawnee anacron[2489]: Job 'cron.daily' started
Aug 18 10:43:58 sawnee anacron[2687]: Updated timestamp for job
'cron.daily' to 2004-08-18
The following files are delivered with anacron:
# rpm -q --filesbypkg anacron-2.3-25
anacron /etc/anacrontab
anacron /etc/cron.daily/0anacron
anacron /etc/cron.monthly/0anacron
anacron /etc/cron.weekly/0anacron
anacron /etc/rc.d/init.d/anacron
anacron /usr/sbin/anacron
anacron /usr/share/doc/anacron-2.3
anacron /usr/share/doc/anacron-2.3/COPYING
anacron /usr/share/doc/anacron-2.3/README
anacron /usr/share/man/man5/anacrontab.5.gz
anacron /usr/share/man/man8/anacron.8.gz
anacron /var/spool/anacron
at
The /usr/bin/at command is used to submit jobs for later execution. Jobs submitted with at are run once at the specified time.
Specify the script or binary to run along with the time specification. Here are a few examples showing how a database administrator could submit a job to run a daily report:
at -f /prod_serv/scripts/prod_daily_report now + 1 minute
at -f /prod_serv/scripts/prod_daily_report now + 1 hour
at -f /prod_serv/scripts/prod_daily_report 13:55
at -f /prod_serv/scripts/prod_daily_report 4am + 3 days
The time specification format is very flexible. See the at(1) man page and /usr/share/doc/at-3.1.8/timespec for more details. Jobs submitted with at inherit the environment variables of the shell from which they were submitted.
Several more commands are provided with the at package. See the man page for a complete list and description. Here are a few useful examples.
To list jobs that are queued, use this command:
# atq
5 2004-08-18 04:00 a root
4 2004-08-15 14:54 a root
To remove a job with an ID number of 4, use this command:
# atrm 4
# atq
5 2004-08-18 04:00 a root
Linux provides a batch command that works similarly to at, except that the command runs when the load average of the system is less than .8 or the value specified with the atrun command. See the next topic for details on atrun. An execution time is not specified when submitting a job with the batch command. For example:
# batch -f /prod_serv/scripts/prod_daily_report
The /etc/at.allow and /etc/at.deny files can be used to control which users can run at and batch. Their format and use are the same as cron’s allow and deny files.
at jobs are queued as files in /var/spool/at in Red Hat and /var/spool/atjobs in SUSE. The job files are plain text files.
at Daemon and atrun
Jobs submitted using at are started either by a daemon atd process or by the atrun script. The newer method is to have atd run as a daemon. You can determine whether atd is started as a daemon by using chkconfig:
#chkconfig --list atd
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
Here we see atd is started at runlevels 3, 4, and 5. If atd is not started as a daemon at startup, the atrun script should be started by cron periodically to run the at jobs. atrun is just a simple script to run atd:
#! /bin/sh
prefix=/usr
exec_prefix=/usr
exec /usr/sbin/atd -s "$@"
The at README (/usr/share/doc/at-3.1.8/README) suggests adding a line to root’s crontab file to start atrun every five minutes:
* * * * 0,5,10,15,20,25,30,35,40,45,50,55 /usr/sbin/atrun
If atd runs as a daemon, the at jobs run instantly. The cost is one additional process as overhead, but that is a small price to pay. If atrun is started with the previous crontab entry, the at jobs are checked only every five minutes. The atrun command can be executed with the -l load average parameter to specify the load average at which jobs submitted with batch will run.
Here are the files delivered with the Red Hat 9.0 at package:
# rpm -q --filesbypkg at
at /etc/at.deny
at /etc/rc.d/init.d/atd
at /usr/bin/at
at /usr/bin/atq
at /usr/bin/atrm
at /usr/bin/batch
at /usr/sbin/atd
at /usr/sbin/atrun
at /usr/share/doc/at-3.1.8
at /usr/share/doc/at-3.1.8/ChangeLog
at /usr/share/doc/at-3.1.8/Copyright
at /usr/share/doc/at-3.1.8/Problems
at /usr/share/doc/at-3.1.8/README
at /usr/share/doc/at-3.1.8/timespec
at /usr/share/man/man1/at.1.gz
at /usr/share/man/man1/atq.1.gz
at /usr/share/man/man1/atrm.1.gz
at /usr/share/man/man1/batch.1.gz
at /usr/share/man/man5/at.allow.5.gz
at /usr/share/man/man5/at.deny.5.gz
at /usr/share/man/man8/atd.8.gz
at /usr/share/man/man8/atrun.8.gz
at /var/spool/at
at /var/spool/at/.SEQ
at /var/spool/at/spool
Troubleshooting cron
We present four troubleshooting scenarios to demonstrate different strategies for resolving some common problems with cron. The examples are limited to cron because cron is by far the most popular of the utilities in this chapter. Also, these examples should provide the information needed to troubleshoot problems in the other utilities covered.
The first scenario shows how to use the information cron provides. Many issues can be resolved just by understanding what the emails from cron and cron’s syslog messages mean. The second scenario shows that cron can start a process only to have it hang. The third scenario shows a problem with the cron daemon. Finally, we cover a problem common to many log files.
Scenario 10-1: Using cron Output for Debugging
The database administrator complains that his report doesn’t run. Not that our DBA would mislead us intentionally, but let’s check the crontab to verify the job entry.
We can confirm the job entry with the crontab command. We use the -u dbprod option to look at another user’s crontab. Only root can do this.
# crontab -u dbprod -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports
We could just cat the crontab file as well:
# cat /var/spool/cron/dbprod
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports
The crontab entry looks good. The next step is to check the cron log to see if we can determine whether cron started the job. There should be a line in /var/log/cron similar to the following:
Aug 15 15:06:00 sawnee CROND[3400]: (dbprod) CMD
(/prod_serv/scripts/prod_daily_reports)
The cron log shows that cron did start the job at 15:06, and the process ID is 3400. We know the job ran. But did it finish? The crontab command line doesn’t redirect standard output (stdout) or standard error (stderr), so any output should go to dbprod’s mail.
From [email protected] Sun Aug 15 15:06:00 2004
Date: Sun, 15 Aug 2004 15:06:00 -0400
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <dbprod@sawnee> /prod_serv/scripts/prod_daily_reports
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/dbprod>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=dbprod>
/bin/sh: line 1: /prod_serv/scripts/prod_daily_reports: No such file or
directory
We can see from the message that the DBA probably made a typo. We should confirm this idea:
# ll /prod_serv/scripts/
total 12
-rwx------ 1 dbprod root 73 Aug 5 17:32 prod_backup_1
-rwx------ 1 dbprod root 42 Jul 27 18:15 prod_backup_2
-rwx------ 1 dbprod root 79 Aug 15 13:51 prod_daily_report
Looks like the DBA meant to run prod_daily_report rather than prod_daily_reports.
If dbprod’s crontab redirects stdout (standard out) and stderr (standard error) to /dev/null, the output is thrown away. 1 is always the file number of stdout, and 2 is the file number for stderr. The >/dev/null 2>&1 on the following crontab entry means make stdout /dev/null and make stderr the same file as stdout, which throws away the script output and any error messages:
06 15 * * * /prod_serv/scripts/prod_daily_reports >/dev/null 2>&1
The /dev/null file is not a real file. The device file throws away anything written to it:
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Oct 2 2003 /dev/null
cron jobs often redirect output to /dev/null to avoid having to trim a log file or delete lots of emails from cron. If the cron job stops working, the crontab line should be modified to send stdout and stderr to a log file so that any output from the script can be examined:
06 15 * * * /prod_serv/scripts/prod_daily_reports >/tmp/dbprod.out 2>&1
The crontab could be changed to email the user with any output:
06 15 * * * /prod_serv/scripts/prod_daily_reports
Set the time fields so that the job runs again and see whether any output is generated.
Scenario 10-2: Using pstree to Find a Hanging cron Job
Ted, a software engineer, complains that his code sweep didn’t run. Using the troubleshooting steps from the first scenario, we verify that Ted’s job should have been run at 7:47 a.m.:
# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5691 installed on Mon Aug 16 07:45:59 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
47 07 * * * /home/ted/sweep.sh
The job was started according to the cron log:
Aug 16 07:47:00 sawnee CROND[5697]: (ted) CMD (/home/ted/sweep.sh)
The user ted has not received email from cron. We know the job was started but never finished. Thus, it is probably still running. The pstree command is a good way to see what cron is doing. pstree shows process parent-child relationships in a tree format. Here is the entry for ted’s job:
If necessary, we could look at the output from ps -ef to see more detail. We can see from the cron log that the process ID for the job is 5697. The process might be working normally or could be hanging, but cron is doing its job.
Scenario 10-3: Jobs Not Started Because cron Daemon Is Not Running
Ted’s back. He says his code sweep didn’t run. Running through the steps, we check the crontab and cron log.
The crontab has a job that should have run at 9:15 a.m.
[root@sawnee root]# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5962 installed on Mon Aug 16 09:13:24 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
15 09 * * * /home/ted/sweep.sh
Ted modified his crontab this morning but was finished before 9:15. cron should have run the job but apparently did not because there is no line in the cron log indicating that the job was started.
Aug 16 09:12:48 sawnee crontab[5962]: (ted) BEGIN EDIT (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) REPLACE (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) END EDIT (ted)
Next, we check whether crond is running:
# ps -ef|grep cron
root 5977 5832 0 09:23 pts/7 00:00:00 grep cron
The crond binary is not running. Let’s restart it and then try to figure out what happened.
# /etc/init.d/crond start
Starting crond: [ OK ]
We can see crond is now running.
# ps -ef|grep cron
root 6001 1 0 09:31 ? 00:00:00 crond
root 6005 5832 0 09:31 pts/7 00:00:00 grep cron
We check /var/log/cron, and it seems crond is working:
Aug 16 09:31:14 sawnee crond[6001]: (CRON) STARTUP (fork ok)
We probably need to run a test cron job just to confirm that cron is working. If so, we move on and try to determine what happened. We check /var/log/messages for system messages that may show a problem. We verify that the cron packages are installed properly.
What cron packages can we find?
# rpm -q -a|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25
The crond binary is probably delivered with vixie-cron. We confirm this suspicion:
# rpm -q -filesbypkg vixie-cron-3.0.1-74
vixie-cron /etc/cron.d
vixie-cron /etc/rc.d/init.d/crond
vixie-cron /usr/bin/crontab
vixie-cron /usr/sbin/crond
vixie-cron /usr/share/man/man1/crontab.1.gz
vixie-cron /usr/share/man/man5/crontab.5.gz
vixie-cron /usr/share/man/man8/cron.8.gz
vixie-cron /usr/share/man/man8/crond.8.gz
vixie-cron /var/spool/cron
We see /usr/sbin/crond is delivered with vixie-cron. Now, we just make sure vixie-cron-3.0.1-74 is installed properly:
# rpm -V vixie-cron-3.0.1-74
No output means that the package is installed properly. Maybe someone with root access killed crond by accident. We could also check for updates to the package that might correct a new bug.
See Chapter 8, “Linux Processes: Structure, Hangs, and Core Dumps,” for further troubleshooting guidance.
Scenario 10-4: cron Log Not Updating
Users say cron is working fine, but a system administrator notices that the cron log isn’t being updated. The cron log is zero bytes in size.
# ls -l /var/log/cron
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
Let’s run a command to generate a cron log update.
# crontab -l
no crontab for root
That should have generated an entry in the cron log even though root has no crontab file, but it didn’t.
# ls -l /var/log/cron
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
We know cron uses syslogd to do its logging. Let’s make sure syslogd is running and working:
# ps -ef|grep syslog
root 1566 1 0 Aug23 ? 00:00:00 syslogd -m 0
root 28889 28848 0 20:21 pts/3 00:00:00 grep syslog
We can see syslogd is running. Is it still logging messages?
# tail -3 /var/log/messages
Aug 31 17:36:59 sawnee sshd(pam_unix)[28511]: session closed for user
root
Aug 31 20:19:36 sawnee sshd(pam_unix)[28846]: session opened for user
root by (uid=0)
From the previous output, we can see syslogd was working as of Aug 31 8:19 p.m. If there were no recent entries in the messages log, we could use the logger command to generate a test message:
# logger -p kern.info "test from sysadmin"
# tail -1 /var/log/messages
Aug 31 20:30:47 sawnee root: test from sysadmin
Where do cron messages go?
# grep cron /etc/syslog.conf
*.info;mail.none;news.none;authpriv.none;cron.none
/var/log/messages
# Log cron stuff
cron.* /var/log/cron
Messages should be routed to /var/log/cron. Because syslogd is doing its job, something must be wrong with the cron file itself.
Someone with root access might have tried to trim the file incorrectly by removing or moving it and then touching cron to start a new log. If we look for cron files, we can see a cron.bkup that seems out of place.
# ls -l /var/log/cron*
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
-rw------- 1 root root 19795 Aug 29 04:02 /var/log/cron.1
-rw------- 1 root root 13506 Aug 22 04:02 /var/log/cron.2
-rw------- 1 root root 19512 Aug 15 04:02 /var/log/cron.3
-rw------- 1 root root 12288 Jul 28 22:01 /var/log/cron.4
-rw------- 1 root root 10254 Aug 31 20:27 /var/log/cron.bkup
Sure enough, our crontab entry went to the old file.
# til -1 /var/log/cron.bkup
Aug 31 20:27:19 sawnee crontab[28906]: (root) LIST (root)
Moving a file using the mv command within the same filesystem while a file is open for writing generally causes problems. The program attached to the open file descriptor continues writing to the old file until the file descriptor is closed. This happens even though the filename has changed. The file descriptor points to the open inodes on the disk. In this instance, even though the file has a new name, syslogd keeps writing to it. The solution is to restart syslogd or put the file back. The syslogd(8) man page tells us to restart syslogd with:
kill -SIGHUP 'cat /var/run/syslogd.pid'
This command tells syslog to close and open the files again and fixes the logging issue. More syslog information can be found in the syslogd(8) and syslog.conf(5) man pages.
# crontab -l
no crontab for root
# tail -1 /var/log/cron
Aug 31 20:39:49 sawnee crontab[28949]: (root) LIST (root)
The cron log is updating properly.
Methodology
The flow chart in Figure 10-2 shows the troubleshooting methodology that was used in the examples.
Figure 10-2. Troubleshooting methodology flow chart
Summary
It is necessary to understand how applications work before they can be effectively troubleshot. Our goal for this chapter was to explain how cron, at, anacron, and kcron work so that you will be ready when problems occur. The scenarios build on the foundation provided by the earlier sections. They present step-by-step resolutions for some common problems.