10. cron and at

The cron and at packages provide Linux users with a method for scheduling jobs. at is used to run a job once. cron is used to run jobs on a schedule. If a report should be run every Friday at 8 p.m., cron is perfect for the job. If a user wants to run a sweep of the system to find a misplaced file, the job can be scheduled to run that evening using the at command.

This chapter explains how cron and at work. You might not be familiar with the anacron and kcron packages, but they extend the features of cron. We also explain these tools in this chapter. We show how to use cron and the other utilities, but we also show how they work and provide examples of what can go wrong.

This chapter is organized into the following sections:

cronThe basics of cron are explained. The crontab command is used to submit and edit jobs. The reader will see the various crontab syntaxes and the format of the cron configuration file. The other files cron uses are explained as well. The cron daemon runs the jobs submitted with crontab. This topic details how the daemon gets started, where it logs, and the differences between cron packages. We also discuss a graphical front end to crontab called kcron.

anacronanacron is a utility to run jobs cron missed due to system downtime. Learn how it works in this section.

atat is a utility similar to cron that runs jobs once. We show examples of submitting, removing, and monitoring jobs with at.

We conclude the chapter with a section on four troubleshooting scenarios that demonstrate good methodologies for fixing problems with cron.

cron

The crontab command is used to manage cron jobs. The -l option displays the current list of jobs for the user. The following example runs the report mentioned in this chapter’s introduction:

[dave@sawnee dave]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.9889 installed on Fri Oct 15 09:42:06 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
0 20 * * fri /home/dave/acct_prod_rpt.sh

The crontab command is the only user command that cron provides. There is a cron daemon to run jobs, but only crontab is run from the command line by users. It is used to create, edit, or display the crontab configuration file, which is called a crontab file for a user. Switches available for the crontab command are shown in Table 10-1.

Table 10-1. crontab Switches

image

We recommend using crontab -l rather than crontab -e to make crontab changes. For example:

$ crontab -l >crontab.out              # Saves the crontab file
$ cp crontab.out crontab.bkup          # Create backup

$ vi crontab.out                       # Make desired changes
$ crontab crontab.out                  # Activate the changes
$ crontab –l                           # Verify changes

It took us five commands to modify the crontab file, but we do have a backup of the original as well as a copy of the new crontab. We could have just run crontab -e to edit the crontab file. When you run crontab -e for the first time, an editor starts with an empty document. Add the job command lines and save the file, and a crontab file is created for the user. If you would rather not use the default editor, set the EDITOR environment variable first. For example, run export EDITOR=vim;crontab -e to edit the crontab file with the vim editor.

The crontab entries can be set using monthly, weekly, daily, and hourly time definitions. The crontab(5) man page shows the format of the crontab lines. The time and date fields from the man page are shown in Table 10-2.

Table 10-2. Time and Date Fields and Allowable Values

image

Follow the scheduling fields with the command to be run. The following crontab entry runs a backup process every day at 11:00 p.m.:

00 23 * * * /prod_serv/scripts/prod_backup_2

This entry runs a report at 6:15 a.m. on the first of every month:

15 6 1 * * /prod_serv/scripts/monthly_reports

This one runs a report every weekday at 2 p.m.:

0 14 * * 1-5 /home/ted/daily_rpt.sh

The crontab command validates the formatting of the command lines. Look at the output from crontab -e when the minute and hour fields are reversed:

[ted@sawnee ted]$ crontab -e

9 50 * * * /home/ted/sweep.sh
~
~
~
~


no crontab for ted - using an empty one
crontab: installing new crontab
"/tmp/crontab.1279":0: bad hour
errors in crontab file, can't install.
Do you want to retry the same edit?

Consider inserting this snippet from the crontab(5) man page as a comment block in your crontab files so that it is easy to understand the format of the job command lines:

########################################################################
#              field         allowed values
#              ------        --------------
#              minute        0–59
#              hour          0–23
#              day of month  1–31
#              month         1–12 (or names, see below)
#              day of week   0–7 (0 or 7 is Sun, or use names)
# min  hour  day  month  wkday
########################################################################

crontab does not validate the paths of commands, however, as you can see here:

[ted@sawnee ted]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.26370 installed on Mon Aug 30 08:57:28 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)

DBHOME="/prod_serv"
03 09 * * * /home/ted/sweeper.sh

crontab accepted the file, but look at the email Ted receives when it is executed by cron:

From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <ted@sawnee> /home/ted/sweeper.sh
X-Cron-Env: <DBHOME=/prod_serv>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/ted>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=ted>


/bin/sh: line 1: /home/ted/sweeper.sh: No such file or directory

Environment variables can be added to the crontab file as well as comments. cron sets very few environment variables. The path is set to /usr/bin:/bin, and the shell is /bin/sh. If environment variables from the shell are needed, they must be set manually in the crontab file. For example, the following crontab sets DBHOME, DBINST, and the PATH variable:

# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.1828 installed on Mon Sep 13 17:26:58 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
DBHOME=/prod_serv/scripts
DBINST=prod
PATH=/home/susan/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/gam
es:/opt/gnome/bin:/opt/kde3/bin:

30 17 * * * /prod_serv/scripts/prod_daily_report

Output from a cron job is mailed to the owner of the crontab. There is a cron environment variable MAILTO. If set, any cron email goes to the user specified by MAILTO. If MAILTO="", no email is sent by cron. For further information, see the crontab(5) man page.

cron and at are similar but not identical among Linux distributions. We show Red Hat and SUSE examples and provide techniques to understand the differences in other Linux distributions.

The crontab files are stored in the /var/spool/cron directory in Red Hat and /var/spool/cron/tabs in SUSE. The directory contains one file for every user with a cron schedule. The user name is the name of the crontab file. The crontab(1) man page has more detailed information.

The capability to use crontab can be restricted by the system administrator by creating an allow or deny file. In Red Hat, the files are /etc/cron.allow and /etc/cron. deny. In SUSE, the files are /var/spool/cron/allow and deny. The allow file is a list of users who can run the crontab command. The deny file is a list of users who cannot. Both files are just lists of user names. For example:

$ cat /etc/cron.allow
dave
root

If the allow file exists but does not contain dbuser, then the database administrator (DBA) sees the following message when trying to run crontab while logged in as dbuser:

$ crontab -e
You (dbuser) are not allowed to use this program (crontab)
See crontab(1) for more information

If the deny file exists and the allow file does not exist, all users can run crontab except those listed in the deny file. The allow file takes precedence. If both allow and deny exist, a user listed in both files can use crontab. To prevent any user from running crontab, including root, create an empty allow file. The at command uses /etc/at.allow and /etc/at.deny and follows the same rules as crontab.

The directory permissions on /var/spool/cron or /var/spool/cron/tabs limit access to only the root user to prevent users from manually creating or editing a crontab file. This forces users to use only the crontab command to change their crontab files. The system administrator should not manually edit these files either.

# ls -al /var/spool/cron
total 16
drwx-------    2 root     root         4096 Aug 11 10:54 .
drwxr-xr-x    16 root     root         4096 Nov  6 2003 ..
-rw-------     1 root     dave          254 Jul 31 12:53 dave
-rw-------     1 root     dbadmin       280 Aug 11 10:51 dbadmin

The crontab command has the setuid bit set so that non-root users can edit their own crontab files:

# ls -l /usr/bin/crontab
-rwsr-xr-x    1 root      root       110114 Feb 19 2003
/usr/bin/crontab

Try to lock down the crontab command by changing the permissions to 555, and you receive the following error when running it as a non-root user:

[dave@sawnee dave]$ crontab -e
seteuid: Operation not permitted

Root has a special crontab file, /etc/crontab. The format is slightly different. The job command line has an additional field to specify what user cron should use to run the job. The default Red Hat /etc/crontab is:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

The /etc/crontab supplied with Red Hat Linux makes use of the run-parts script. On a Linux installation, there are probably several processes that should be run daily such as backups and reports. There can be weekly or monthly jobs such as reports, software updates, system scans, and so on. This is the beauty of the run-parts script. The /etc/crontab shown previously runs the daily jobs at 4:02 a.m., the weekly jobs at 4:22 a.m. on Sunday, and the monthly jobs at 4:42 a.m. on the first day of the month.

The /usr/bin/run-parts script expects to be supplied with a directory path as an argument. The run-parts script executes all the scripts contained in that directory except for those ending in rpmsave, rpmorig, rpmnew, and swp scripts. There is no man page for run-parts, but it is a simple script to read and understand.

The functionality provided by /usr/bin/run-parts is very important. A software package that includes a cron job can be written to place a file in the appropriate /etc/cron.hourly, cron.daily, cron.weekly, or cron.monthly directory rather than trying to manipulate root’s crontab file. This capability makes creating packages containing cron jobs manageable. You can see which packages delivered cron jobs to a Linux system with the command rpm -qa --filesbypkg grep '/etc/cron'.

The SUSE /etc/crontab file is similar to Red Hat’s /etc/crontab:

SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * *   root  test -x /usr/lib/cron/run-crons && /usr/lib/
cron/run-crons >/dev/null 2>&1
59 * * * *      root  rm -f /var/spool/cron/lastrun/cron.hourly
14 4 * * *      root  rm -f /var/spool/cron/lastrun/cron.daily
29 4 * * 6      root  rm -f /var/spool/cron/lastrun/cron.weekly
44 4 1 * *      root  rm -f /var/spool/cron/lastrun/cron.monthly

SUSE Linux supplies the run-crons script. It runs the jobs in the /etc/cron.* directories like run-parts does on Red Hat. cron on SUSE is configured to use /var/spool/cron/lastrun to keep track of the last time that the /etc/cron.* scripts were run. cron does not log the run-crons job execution.

Make sure only root can write to the /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, and /etc/cron.monthly directories. Otherwise, any user can save a script in one of these directories and use cron to run his or her process with root privileges. Notice that the command line indicates that root should run these scripts. Even if the script in /etc/cron.daily is owned by a user, it is executed with root privileges.

# ls -ld /etc/cron.*
drwxr-xr-x    2 root     root           4096 Feb 19  2003 /etc/cron.d
drwxr-xr-x    2 root     root           4096 Aug 10 04:08 /etc/cron.daily
drwxr-xr-x    2 root     root           4096 Nov  6  2003 /etc/cron.hourly
drwxr-xr-x    2 root     root           4096 Nov  6  2003 /etc/cron.monthly
drwxr-xr-x    2 root     root           4096 Nov  6  2003 /etc/cron.weekly


Note

You can confirm that the permissions of a package are correct with the rpm –V cron_package_name command. You can confirm the permissions of a single file or directory with rpm –V -f filename. Troubleshooting using rpm is covered later in this chapter.


Here is a sample /etc/cron.daily:

# ll /etc/cron.daily
total 36
-rwxr-xr-x    1 root     root          135 Jan 25  2003 00webalizer
-rwxr-xr-x    1 root     root          276 Jan 24  2003 0anacron
-rwxr-xr-x    1 root     root          123 Jan 26  2003 inn-cron-expire
-rwxr-xr-x    1 root     root           51 Jan 24  2003 logrotate
-rwxr-xr-x    1 root     root          418 Feb 10  2003 makewhatis.cron
-rwxr-xr-x    1 root     root          135 May 14 15:59 rpm
-rwxr-xr-x    1 root     root          132 Jan 21  2004 slocate.cron
-rwxr-xr-x    1 root     root          164 May 17 09:18 sophos
-rwxr-xr-x    1 root     root          193 Feb 10  2003 tmpwatch

cron Daemon

The cron daemon (/usr/sbin/cron for SUSE and /usr/sbin/crond for Red Hat) is responsible for running the cron jobs. Every minute, the cron daemon looks at the crontabs, loads new or modified crontabs into memory, and checks whether any jobs are scheduled to execute during that minute. The cron daemon is started during the startup scripts at runlevels 2, 3, 4, and 5.

#chkconfig --list crond
crond           0:off    1:off    2:on    3:on    4:on    5:on    6:off

The startup script is /etc/init.d/crond in Red Hat and /etc/init.d/cron in SUSE. For more information on startup scripts, refer to Chapter 1, “System Boot, Startup, and Shutdown Issues.”

A log entry is created when cron jobs are run, crontab commands are used, the cron daemon is started, and so on. cron uses the syslog daemon to log cron events. The syslog facility for cron messages is cron. See the syslogd(8) man page for an explanation of how syslog routes messages. The log file for cron is determined by /etc/syslog. conf. Red Hat has a log just for cron, /var/log/cron, to log cron messages. SUSE uses /var/log/messages. A Red Hat /etc/syslog.conf has the following entry to route cron messages to /var/log/cron:

# Log cron stuff
cron.*                                /var/log/cron

With Red Hat, cron messages are not duplicated in /var/log/messages because of the cron.none on the following syslog.conf line:

*.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages

It would be a simple matter to change SUSE systems to log cron messages to /var/log/cron. Here is a sample of the entries that are written to Red Hat’s /var/log/cron file:

Aug 15 04:22:00 sawnee CROND[25198]: (root) CMD (run-parts
/etc/cron.weekly)
Aug 15 04:22:04 sawnee anacron[25202]: Updated timestamp for job
'cron.weekly' to 2004-08-15

Aug 15 05:01:00 sawnee CROND[2420]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 06:01:00 sawnee CROND[2519]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 07:01:00 sawnee CROND[2532]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 08:01:00 sawnee CROND[2544]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 09:01:00 sawnee CROND[2556]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 10:01:00 sawnee CROND[2568]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 11:01:00 sawnee CROND[2588]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 12:01:00 sawnee CROND[2600]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 13:01:00 sawnee CROND[2612]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:01:00 sawnee CROND[2910]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:35:46 sawnee crontab[3107]: (dave) LIST (dave)
Aug 15 14:36:56 sawnee crontab[3113]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) REPLACE (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) END EDIT (dave)
Aug 15 14:37:09 sawnee crontab[3116]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) REPLACE (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) END EDIT (dave)
Aug 15 14:38:00 sawnee crond[1792]: (dave) RELOAD (cron/dave)
Aug 15 14:38:00 sawnee CROND[3166]: (dave) CMD (/home/dave/daily_rpt)

The sample cron log shows cron activity on host sawnee. cron events are broken out by crond, crontab, and CROND (which shows jobs being started). The number between the [] is the process ID. Everything that crontab performs is logged. Note that run-parts is listed as one job and is not broken down by each job in /etc/cron.hourly.

As we noted earlier, cron and at are similar but not identical across Linux distributions. To understand the differences, look closely at the man pages, see which packages are installed, and look for additional documentation supplied with the packages. Run rpm -qa to see what packages are installed. This list is from a Red Hat 9.0 system:

# rpm -qa|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25

A typical SUSE 9.0 system might show:

# rpm -qa|grep cron
cron-3.0.1-824

The package name doesn’t indicate it, but this is vixie-cron too, which can be verified in the README supplied with the package.

Look at the files delivered with the packages by running rpm -q --filesbypkg. Take note of man pages, READMEs, and other documentation provided with each package. You can see what commands are available with the package as well as what documentation you can review. Here are the files delivered with Red Hat 9.0:

# rpm -q --filesbypkg vixie-cron-3.0.1-74
vixie-cron                /etc/cron.d
vixie-cron                /etc/rc.d/init.d/crond
vixie-cron                /usr/bin/crontab
vixie-cron                /usr/sbin/crond
vixie-cron                /usr/share/man/man1/crontab.1.gz
vixie-cron                /usr/share/man/man5/crontab.5.gz
vixie-cron                /usr/share/man/man8/cron.8.gz
vixie-cron                /usr/share/man/man8/crond.8.gz
vixie-cron                /var/spool/cron

# rpm -q --filesbypkg crontabs-1.10-5
crontabs                  /etc/cron.daily
crontabs                  /etc/cron.hourly
crontabs                  /etc/cron.monthly
crontabs                  /etc/cron.weekly
crontabs                  /etc/crontab
crontabs                  /usr/bin/run-parts

SUSE 9.0 delivers the following files with the cron package:

# rpm -q --filesbypkg cron-3.0.1-824
cron                      /etc/crontab
cron                      /etc/init.d/cron
cron                      /usr/bin/crontab
cron                      /usr/lib/cron
cron                      /usr/lib/cron/run-crons
cron                      /usr/sbin/cron
cron                      /usr/sbin/rccron
cron                      /usr/share/doc/packages/cron
cron                      /usr/share/doc/packages/cron/CHANGES
cron                      /usr/share/doc/packages/cron/CONVERSION
cron                      /usr/share/doc/packages/cron/FEATURES
cron                      /usr/share/doc/packages/cron/MAIL
cron                      /usr/share/doc/packages/cron/MANIFEST
cron                      /usr/share/doc/packages/cron/README
cron                      /usr/share/doc/packages/cron/THANKS
cron                      /usr/share/man/man1/crontab.1.gz
cron                      /usr/share/man/man5/crontab.5.gz
cron                      /usr/share/man/man8/cron.8.gz
cron                      /var/spool/cron
cron                      /var/spool/cron/deny
cron                      /var/spool/cron/lastrun
cron                      /var/spool/cron/tabs

Listing the files delivered in the package makes it easy to see the differences between distributions.

kcron

A nice graphical front-end utility is available for the crontab command. The kcron utility is part of KDE and is supplied with the optional kdeadmin package. The kcron utility is a simple, intuitive method for users to manage their cron jobs. The English version of the kcron handbook is available at http://docs.kde.org/en/3.1/kdeadmin/kcron/index.html. To start kcron, choose System Tools, Task Scheduler from the main KDE menu, and you will see the interface shown in Figure 10-1.

Figure 10-1. The kcron interface

image

No crontab knowledge is needed to schedule jobs when kcron is used. Users don’t have to learn vi just to schedule a job. kcron inserts a header and a footer in the crontab file, so the system administrator knows it was saved from kcron. Here is an example:

$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/kde-susan/kcronzMmGJa.tmp installed on Tue Sep  7 09:47:40 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
#
5 4 * * *       /home/susan/screening.sh
# This file was written by KCron. Copyright (c) 1999, Gary Meyer
# Although KCron supports most crontab formats, use care when editing.
# Note: Lines beginning with "#" indicates a disabled task.

There are a couple of wrinkles with kcron. kcron does not seem to be aware of the crontab allow and deny files, meaning users who are restricted from using the crontab command can still create cron tasks.

If kcron is run as root, the save function saves crontab files for every user on the system even if only root’s jobs were modified. If the user already has a crontab file, kcron inserts its header and footer. If the user doesn’t have a crontab file, kcron creates one with the normal crontab header plus the kcron header line. We don’t recommend using kcron as root, but it is fine for other users.

To see the list of known bugs in kcron, visit http://bugs.kde.org/. We submitted new bug reports for those we mentioned here:

Bug 89488: kcron doesn’t prohibit crontab changes based on cron’s allow and deny files.

Bug 89491: kcron creates crontab files for all users when the root crontab is created.

More information about KDE is available at http://www.kde.org/.

anacron

anacron is similar to cron, and it works much the same way. anacron is intended to run jobs in /etc/crontab that cron missed due to system downtime. If a Linux box is left on continuously, anacron won’t run any jobs because cron will run them. anacron has its own crontab file, /etc/anacrontab. Changes made to /etc/crontab need to be made in /etc/anacrontab as well. anacron is a utility for root only. There is no anacrontab command to edit the file for users. The system administrator should edit the file manually. Here is the anacrontab supplied with anacron on Red Hat systems:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

1       65     cron.daily          run-parts /etc/cron.daily
7       70     cron.weekly         run-parts /etc/cron.weekly
30      75    cron.monthly         run-parts /etc/cron.monthly

The format is similar to the crontab files, but there are some differences. The first few lines set the shell and path. The remaining lines contain the scripts to execute; anacron calls these job description lines.

The first field of the job description line indicates how many days should occur between runs of the process. The second field indicates how many minutes to wait before starting the process. The third field is the job identifier. The last field and remainder of the line is the shell command to execute.

See the anacrontab(8) man page for further details.

The anacrontab file doesn’t execute run-parts /etc/cron.hourly because the smallest time unit for anacron is one day.

There is a potential problem with the previous anacrontab file. The full path to the run-parts script is not given. The run-parts script is in /usr/bin. The /usr/local/sbin and /usr/local/bin directories are searched before /usr/bin according to the PATH variable. If users have write permission to /usr/local/sbin or /usr/local/bin, a script called run-parts could be placed in those directories and would run instead of the intended /usr/bin/run-parts. This issue can be prevented by always using the full path for scripts in anacrontab. For example:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

1       65      cron.daily          /usr/bin/run-parts /etc/cron.daily
7       70      cron.weekly         /usr/bin/run-parts /etc/cron.weekly
30      75      cron.monthly       /usr/bin/run-parts /etc/cron.monthly

anacron is started by a startup script.

#chkconfig --list anacron
anacron         0:off   1:off   2:on   3:on   4:on   5:on   6:off

anacron checks the timestamp files in /var/spool/anacron and runs any jobs that were missed.

# ls -al /var/spool/anacron
total 20
drwxr-xr-x    2 root     root         4096 Nov  6  2003 .
drwxr-xr-x   16 root     root         4096 Nov  6  2003 ..
-rw-------    1 root     root            9 Aug 14 12:53 cron.daily
-rw-------    1 root     root            9 Aug  1 04:42 cron.monthly
-rw-------    1 root     root            9 Aug  8 04:22 cron.weekly

anacron records whether cron is running normally so that jobs are not run twice. This is done with an anacron script in /etc/cron.daily, /etc/cron.weekly, and /etc/cron.monthly, which is executed by cron. The anacron -u command is run by this anacron script to update the timestamps and indicate that all jobs were run for that day by cron.

anacron messages route to the same log file as cron messages. anacron logs messages using syslogd with the facility value set to cron. Here are some sample anacron log entries:

Aug 18 10:43:58 sawnee anacron[2489]: Job 'cron.daily' started
Aug 18 10:43:58 sawnee anacron[2687]: Updated timestamp for job
'cron.daily' to 2004-08-18

The following files are delivered with anacron:

# rpm -q --filesbypkg anacron-2.3-25
anacron                   /etc/anacrontab
anacron                   /etc/cron.daily/0anacron
anacron                   /etc/cron.monthly/0anacron

anacron                   /etc/cron.weekly/0anacron
anacron                   /etc/rc.d/init.d/anacron
anacron                   /usr/sbin/anacron
anacron                   /usr/share/doc/anacron-2.3
anacron                   /usr/share/doc/anacron-2.3/COPYING
anacron                   /usr/share/doc/anacron-2.3/README
anacron                   /usr/share/man/man5/anacrontab.5.gz
anacron                   /usr/share/man/man8/anacron.8.gz
anacron                   /var/spool/anacron

at

The /usr/bin/at command is used to submit jobs for later execution. Jobs submitted with at are run once at the specified time.

Specify the script or binary to run along with the time specification. Here are a few examples showing how a database administrator could submit a job to run a daily report:

at -f /prod_serv/scripts/prod_daily_report now + 1 minute
at -f /prod_serv/scripts/prod_daily_report now + 1 hour
at -f /prod_serv/scripts/prod_daily_report 13:55
at -f /prod_serv/scripts/prod_daily_report 4am + 3 days

The time specification format is very flexible. See the at(1) man page and /usr/share/doc/at-3.1.8/timespec for more details. Jobs submitted with at inherit the environment variables of the shell from which they were submitted.

Several more commands are provided with the at package. See the man page for a complete list and description. Here are a few useful examples.

To list jobs that are queued, use this command:

# atq
5        2004-08-18 04:00 a root
4        2004-08-15 14:54 a root

To remove a job with an ID number of 4, use this command:

# atrm 4
# atq
5       2004-08-18 04:00 a root

Linux provides a batch command that works similarly to at, except that the command runs when the load average of the system is less than .8 or the value specified with the atrun command. See the next topic for details on atrun. An execution time is not specified when submitting a job with the batch command. For example:

# batch -f /prod_serv/scripts/prod_daily_report

The /etc/at.allow and /etc/at.deny files can be used to control which users can run at and batch. Their format and use are the same as cron’s allow and deny files.

at jobs are queued as files in /var/spool/at in Red Hat and /var/spool/atjobs in SUSE. The job files are plain text files.

at Daemon and atrun

Jobs submitted using at are started either by a daemon atd process or by the atrun script. The newer method is to have atd run as a daemon. You can determine whether atd is started as a daemon by using chkconfig:

#chkconfig --list atd
atd             0:off    1:off    2:off    3:on    4:on    5:on    6:off

Here we see atd is started at runlevels 3, 4, and 5. If atd is not started as a daemon at startup, the atrun script should be started by cron periodically to run the at jobs. atrun is just a simple script to run atd:

#! /bin/sh
prefix=/usr
exec_prefix=/usr
exec /usr/sbin/atd -s "$@"

The at README (/usr/share/doc/at-3.1.8/README) suggests adding a line to root’s crontab file to start atrun every five minutes:

* * * * 0,5,10,15,20,25,30,35,40,45,50,55 /usr/sbin/atrun

If atd runs as a daemon, the at jobs run instantly. The cost is one additional process as overhead, but that is a small price to pay. If atrun is started with the previous crontab entry, the at jobs are checked only every five minutes. The atrun command can be executed with the -l load average parameter to specify the load average at which jobs submitted with batch will run.

Here are the files delivered with the Red Hat 9.0 at package:

# rpm -q --filesbypkg at
at                       /etc/at.deny
at                       /etc/rc.d/init.d/atd
at                       /usr/bin/at
at                       /usr/bin/atq
at                       /usr/bin/atrm
at                       /usr/bin/batch
at                       /usr/sbin/atd
at                       /usr/sbin/atrun
at                       /usr/share/doc/at-3.1.8
at                       /usr/share/doc/at-3.1.8/ChangeLog
at                       /usr/share/doc/at-3.1.8/Copyright
at                       /usr/share/doc/at-3.1.8/Problems
at                       /usr/share/doc/at-3.1.8/README
at                       /usr/share/doc/at-3.1.8/timespec
at                       /usr/share/man/man1/at.1.gz
at                       /usr/share/man/man1/atq.1.gz
at                       /usr/share/man/man1/atrm.1.gz
at                       /usr/share/man/man1/batch.1.gz
at                       /usr/share/man/man5/at.allow.5.gz
at                       /usr/share/man/man5/at.deny.5.gz
at                       /usr/share/man/man8/atd.8.gz
at                       /usr/share/man/man8/atrun.8.gz
at                       /var/spool/at
at                       /var/spool/at/.SEQ
at                       /var/spool/at/spool

Troubleshooting cron

We present four troubleshooting scenarios to demonstrate different strategies for resolving some common problems with cron. The examples are limited to cron because cron is by far the most popular of the utilities in this chapter. Also, these examples should provide the information needed to troubleshoot problems in the other utilities covered.

The first scenario shows how to use the information cron provides. Many issues can be resolved just by understanding what the emails from cron and cron’s syslog messages mean. The second scenario shows that cron can start a process only to have it hang. The third scenario shows a problem with the cron daemon. Finally, we cover a problem common to many log files.

Scenario 10-1: Using cron Output for Debugging

The database administrator complains that his report doesn’t run. Not that our DBA would mislead us intentionally, but let’s check the crontab to verify the job entry.

We can confirm the job entry with the crontab command. We use the -u dbprod option to look at another user’s crontab. Only root can do this.

# crontab -u dbprod -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports

We could just cat the crontab file as well:

# cat /var/spool/cron/dbprod
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports

The crontab entry looks good. The next step is to check the cron log to see if we can determine whether cron started the job. There should be a line in /var/log/cron similar to the following:

Aug 15 15:06:00 sawnee CROND[3400]: (dbprod) CMD
(/prod_serv/scripts/prod_daily_reports)

The cron log shows that cron did start the job at 15:06, and the process ID is 3400. We know the job ran. But did it finish? The crontab command line doesn’t redirect standard output (stdout) or standard error (stderr), so any output should go to dbprod’s mail.

From [email protected] Sun Aug 15 15:06:00 2004
Date: Sun, 15 Aug 2004 15:06:00 -0400
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <dbprod@sawnee> /prod_serv/scripts/prod_daily_reports
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/dbprod>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=dbprod>

/bin/sh: line 1: /prod_serv/scripts/prod_daily_reports: No such file or
directory

We can see from the message that the DBA probably made a typo. We should confirm this idea:

# ll /prod_serv/scripts/
total 12
-rwx------    1 dbprod   root            73 Aug 5 17:32 prod_backup_1
-rwx------    1 dbprod   root            42 Jul 27 18:15 prod_backup_2
-rwx------    1 dbprod   root            79 Aug 15 13:51 prod_daily_report

Looks like the DBA meant to run prod_daily_report rather than prod_daily_reports.

If dbprod’s crontab redirects stdout (standard out) and stderr (standard error) to /dev/null, the output is thrown away. 1 is always the file number of stdout, and 2 is the file number for stderr. The >/dev/null 2>&1 on the following crontab entry means make stdout /dev/null and make stderr the same file as stdout, which throws away the script output and any error messages:

06 15 * * * /prod_serv/scripts/prod_daily_reports >/dev/null 2>&1

The /dev/null file is not a real file. The device file throws away anything written to it:

# ls -l /dev/null
crw-rw-rw-    1 root     root        1, 3 Oct 2 2003 /dev/null

cron jobs often redirect output to /dev/null to avoid having to trim a log file or delete lots of emails from cron. If the cron job stops working, the crontab line should be modified to send stdout and stderr to a log file so that any output from the script can be examined:

06 15 * * * /prod_serv/scripts/prod_daily_reports >/tmp/dbprod.out 2>&1

The crontab could be changed to email the user with any output:

06 15 * * * /prod_serv/scripts/prod_daily_reports

Set the time fields so that the job runs again and see whether any output is generated.

Scenario 10-2: Using pstree to Find a Hanging cron Job

Ted, a software engineer, complains that his code sweep didn’t run. Using the troubleshooting steps from the first scenario, we verify that Ted’s job should have been run at 7:47 a.m.:

# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5691 installed on Mon Aug 16 07:45:59 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
47 07 * * * /home/ted/sweep.sh

The job was started according to the cron log:

Aug 16 07:47:00 sawnee CROND[5697]: (ted) CMD (/home/ted/sweep.sh)

The user ted has not received email from cron. We know the job was started but never finished. Thus, it is probably still running. The pstree command is a good way to see what cron is doing. pstree shows process parent-child relationships in a tree format. Here is the entry for ted’s job:

image

If necessary, we could look at the output from ps -ef to see more detail. We can see from the cron log that the process ID for the job is 5697. The process might be working normally or could be hanging, but cron is doing its job.

Scenario 10-3: Jobs Not Started Because cron Daemon Is Not Running

Ted’s back. He says his code sweep didn’t run. Running through the steps, we check the crontab and cron log.

The crontab has a job that should have run at 9:15 a.m.

[root@sawnee root]# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5962 installed on Mon Aug 16 09:13:24 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
15 09 * * * /home/ted/sweep.sh

Ted modified his crontab this morning but was finished before 9:15. cron should have run the job but apparently did not because there is no line in the cron log indicating that the job was started.

Aug 16 09:12:48 sawnee crontab[5962]: (ted) BEGIN EDIT (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) REPLACE (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) END EDIT (ted)

Next, we check whether crond is running:

# ps -ef|grep cron
root      5977  5832   0 09:23 pts/7     00:00:00 grep cron

The crond binary is not running. Let’s restart it and then try to figure out what happened.

# /etc/init.d/crond start
Starting crond:                                        [ OK ]

We can see crond is now running.

# ps -ef|grep cron
root      6001      1  0 09:31 ?         00:00:00 crond
root      6005   5832  0 09:31 pts/7     00:00:00 grep cron

We check /var/log/cron, and it seems crond is working:

Aug 16 09:31:14 sawnee crond[6001]: (CRON) STARTUP (fork ok)

We probably need to run a test cron job just to confirm that cron is working. If so, we move on and try to determine what happened. We check /var/log/messages for system messages that may show a problem. We verify that the cron packages are installed properly.

What cron packages can we find?

# rpm -q -a|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25

The crond binary is probably delivered with vixie-cron. We confirm this suspicion:

# rpm -q -filesbypkg vixie-cron-3.0.1-74
vixie-cron                /etc/cron.d
vixie-cron                /etc/rc.d/init.d/crond
vixie-cron                /usr/bin/crontab

vixie-cron                /usr/sbin/crond
vixie-cron                /usr/share/man/man1/crontab.1.gz
vixie-cron                /usr/share/man/man5/crontab.5.gz
vixie-cron                /usr/share/man/man8/cron.8.gz
vixie-cron                /usr/share/man/man8/crond.8.gz
vixie-cron                /var/spool/cron

We see /usr/sbin/crond is delivered with vixie-cron. Now, we just make sure vixie-cron-3.0.1-74 is installed properly:

# rpm -V vixie-cron-3.0.1-74

No output means that the package is installed properly. Maybe someone with root access killed crond by accident. We could also check for updates to the package that might correct a new bug.

See Chapter 8, “Linux Processes: Structure, Hangs, and Core Dumps,” for further troubleshooting guidance.

Scenario 10-4: cron Log Not Updating

Users say cron is working fine, but a system administrator notices that the cron log isn’t being updated. The cron log is zero bytes in size.

# ls -l /var/log/cron
-rw-------    1 root      root             0 Aug 31 20:26 /var/log/cron

Let’s run a command to generate a cron log update.

# crontab -l
no crontab for root

That should have generated an entry in the cron log even though root has no crontab file, but it didn’t.

# ls -l /var/log/cron
-rw-------    1 root      root             0 Aug 31 20:26 /var/log/cron

We know cron uses syslogd to do its logging. Let’s make sure syslogd is running and working:

# ps -ef|grep syslog
root      1566     1  0 Aug23 ?        00:00:00 syslogd -m 0
root     28889 28848  0 20:21 pts/3    00:00:00 grep syslog

We can see syslogd is running. Is it still logging messages?

# tail -3 /var/log/messages

Aug 31 17:36:59 sawnee sshd(pam_unix)[28511]: session closed for user
root

Aug 31 20:19:36 sawnee sshd(pam_unix)[28846]: session opened for user
root by (uid=0)

From the previous output, we can see syslogd was working as of Aug 31 8:19 p.m. If there were no recent entries in the messages log, we could use the logger command to generate a test message:

# logger -p kern.info "test from sysadmin"
# tail -1 /var/log/messages
Aug 31 20:30:47 sawnee root: test from sysadmin

Where do cron messages go?

# grep cron /etc/syslog.conf
*.info;mail.none;news.none;authpriv.none;cron.none
/var/log/messages
# Log cron stuff
cron.*                                                  /var/log/cron

Messages should be routed to /var/log/cron. Because syslogd is doing its job, something must be wrong with the cron file itself.

Someone with root access might have tried to trim the file incorrectly by removing or moving it and then touching cron to start a new log. If we look for cron files, we can see a cron.bkup that seems out of place.

# ls -l /var/log/cron*
-rw-------    1 root      root           0 Aug 31 20:26 /var/log/cron
-rw-------    1 root      root       19795 Aug 29 04:02 /var/log/cron.1
-rw-------    1 root      root       13506 Aug 22 04:02 /var/log/cron.2
-rw-------    1 root      root       19512 Aug 15 04:02 /var/log/cron.3
-rw-------    1 root      root       12288 Jul 28 22:01 /var/log/cron.4
-rw-------    1 root      root       10254 Aug 31 20:27 /var/log/cron.bkup

Sure enough, our crontab entry went to the old file.

# til -1 /var/log/cron.bkup
Aug 31 20:27:19 sawnee crontab[28906]: (root) LIST (root)

Moving a file using the mv command within the same filesystem while a file is open for writing generally causes problems. The program attached to the open file descriptor continues writing to the old file until the file descriptor is closed. This happens even though the filename has changed. The file descriptor points to the open inodes on the disk. In this instance, even though the file has a new name, syslogd keeps writing to it. The solution is to restart syslogd or put the file back. The syslogd(8) man page tells us to restart syslogd with:

kill -SIGHUP 'cat /var/run/syslogd.pid'

This command tells syslog to close and open the files again and fixes the logging issue. More syslog information can be found in the syslogd(8) and syslog.conf(5) man pages.

# crontab -l
no crontab for root

# tail -1 /var/log/cron
Aug 31 20:39:49 sawnee crontab[28949]: (root) LIST (root)

The cron log is updating properly.

Methodology

The flow chart in Figure 10-2 shows the troubleshooting methodology that was used in the examples.

Figure 10-2. Troubleshooting methodology flow chart

image

Summary

It is necessary to understand how applications work before they can be effectively troubleshot. Our goal for this chapter was to explain how cron, at, anacron, and kcron work so that you will be ready when problems occur. The scenarios build on the foundation provided by the earlier sections. They present step-by-step resolutions for some common problems.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.224.62.160