The cron
and at
packages provide Linux users with a method for scheduling jobs. at
is used to run a job once. cron
is used to run jobs on a schedule. If a report should be run every Friday at 8 p.m., cron
is perfect for the job. If a user wants to run a sweep of the system to find a misplaced file, the job can be scheduled to run that evening using the at
command.
This chapter explains how cron
and at
work. You might not be familiar with the anacron
and kcron
packages, but they extend the features of cron
. We also explain these tools in this chapter. We show how to use cron
and the other utilities, but we also show how they work and provide examples of what can go wrong.
This chapter is organized into the following sections:
• cron
—The basics of cron
are explained. The crontab
command is used to submit and edit jobs. The reader will see the various crontab
syntaxes and the format of the cron
configuration file. The other files cron
uses are explained as well. The cron
daemon runs the jobs submitted with crontab
. This topic details how the daemon gets started, where it logs, and the differences between cron
packages. We also discuss a graphical front end to crontab
called kcron
.
• anacron
—anacron
is a utility to run jobs cron
missed due to system downtime. Learn how it works in this section.
• at
—at
is a utility similar to cron
that runs jobs once. We show examples of submitting, removing, and monitoring jobs with at
.
We conclude the chapter with a section on four troubleshooting scenarios that demonstrate good methodologies for fixing problems with cron
.
The crontab
command is used to manage cron
jobs. The -l
option displays the current list of jobs for the user. The following example runs the report mentioned in this chapter’s introduction:
[dave@sawnee dave]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.9889 installed on Fri Oct 15 09:42:06 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
0 20 * * fri /home/dave/acct_prod_rpt.sh
The crontab
command is the only user command that cron
provides. There is a cron
daemon to run jobs, but only crontab
is run from the command line by users. It is used to create, edit, or display the crontab
configuration file, which is called a crontab
file for a user. Switches available for the crontab
command are shown in Table 10-1.
We recommend using crontab -l
rather than crontab -e
to make crontab
changes. For example:
$ crontab -l >crontab.out # Saves the crontab file
$ cp crontab.out crontab.bkup # Create backup
$ vi crontab.out # Make desired changes
$ crontab crontab.out # Activate the changes
$ crontab –l # Verify changes
It took us five commands to modify the crontab
file, but we do have a backup of the original as well as a copy of the new crontab
. We could have just run crontab -e
to edit the crontab
file. When you run crontab -e
for the first time, an editor starts with an empty document. Add the job command lines and save the file, and a crontab
file is created for the user. If you would rather not use the default editor, set the EDITOR
environment variable first. For example, run export EDITOR=vim;crontab -e
to edit the crontab
file with the vim
editor.
The crontab
entries can be set using monthly, weekly, daily, and hourly time definitions. The crontab(5)
man page shows the format of the crontab
lines. The time and date fields from the man page are shown in Table 10-2.
Table 10-2. Time and Date Fields and Allowable Values
Follow the scheduling fields with the command to be run. The following crontab
entry runs a backup process every day at 11:00 p.m.:
00 23 * * * /prod_serv/scripts/prod_backup_2
This entry runs a report at 6:15 a.m. on the first of every month:
15 6 1 * * /prod_serv/scripts/monthly_reports
This one runs a report every weekday at 2 p.m.:
0 14 * * 1-5 /home/ted/daily_rpt.sh
The crontab
command validates the formatting of the command lines. Look at the output from crontab -e
when the minute and hour fields are reversed:
[ted@sawnee ted]$ crontab -e
9 50 * * * /home/ted/sweep.sh
~
~
~
~
no crontab for ted - using an empty one
crontab: installing new crontab
"/tmp/crontab.1279":0: bad hour
errors in crontab file, can't install.
Do you want to retry the same edit?
Consider inserting this snippet from the crontab(5)
man page as a comment block in your crontab
files so that it is easy to understand the format of the job command lines:
########################################################################
# field allowed values
# ------ --------------
# minute 0–59
# hour 0–23
# day of month 1–31
# month 1–12 (or names, see below)
# day of week 0–7 (0 or 7 is Sun, or use names)
# min hour day month wkday
########################################################################
crontab
does not validate the paths of commands, however, as you can see here:
[ted@sawnee ted]$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.26370 installed on Mon Aug 30 08:57:28 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
DBHOME="/prod_serv"
03 09 * * * /home/ted/sweeper.sh
crontab
accepted the file, but look at the email Ted receives when it is executed by cron
:
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <ted@sawnee> /home/ted/sweeper.sh
X-Cron-Env: <DBHOME=/prod_serv>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/ted>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=ted>
/bin/sh: line 1: /home/ted/sweeper.sh: No such file or directory
Environment variables can be added to the crontab
file as well as comments. cron
sets very few environment variables. The path is set to /usr/bin:/bin
, and the shell is /bin/sh
. If environment variables from the shell are needed, they must be set manually in the crontab
file. For example, the following crontab
sets DBHOME
, DBINST
, and the PATH
variable:
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.1828 installed on Mon Sep 13 17:26:58 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
DBHOME=/prod_serv/scripts
DBINST=prod
PATH=/home/susan/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/gam
es:/opt/gnome/bin:/opt/kde3/bin:
30 17 * * * /prod_serv/scripts/prod_daily_report
Output from a cron
job is mailed to the owner of the crontab
. There is a cron
environment variable MAILTO
. If set, any cron
email goes to the user specified by MAILTO
. If MAILTO=""
, no email is sent by cron
. For further information, see the crontab(5)
man page.
cron
and at
are similar but not identical among Linux distributions. We show Red Hat and SUSE examples and provide techniques to understand the differences in other Linux distributions.
The crontab
files are stored in the /var/spool/cron
directory in Red Hat and /var/spool/cron/tabs
in SUSE. The directory contains one file for every user with a cron
schedule. The user name is the name of the crontab
file. The crontab(1)
man page has more detailed information.
The capability to use crontab
can be restricted by the system administrator by creating an allow or deny file. In Red Hat, the files are /etc/cron.allow
and /etc/cron. deny
. In SUSE, the files are /var/spool/cron/allow
and deny
. The allow
file is a list of users who can run the crontab
command. The deny
file is a list of users who cannot. Both files are just lists of user names. For example:
$ cat /etc/cron.allow
dave
root
If the allow file exists but does not contain dbuser
, then the database administrator (DBA) sees the following message when trying to run crontab
while logged in as dbuser
:
$ crontab -e
You (dbuser) are not allowed to use this program (crontab)
See crontab(1) for more information
If the deny file exists and the allow file does not exist, all users can run crontab
except those listed in the deny file. The allow file takes precedence. If both allow and deny exist, a user listed in both files can use crontab
. To prevent any user from running crontab
, including root, create an empty allow file. The at
command uses /etc/at.allow
and /etc/at.deny
and follows the same rules as crontab
.
The directory permissions on /var/spool/cron
or /var/spool/cron/tabs
limit access to only the root user to prevent users from manually creating or editing a crontab
file. This forces users to use only the crontab
command to change their crontab
files. The system administrator should not manually edit these files either.
# ls -al /var/spool/cron
total 16
drwx------- 2 root root 4096 Aug 11 10:54 .
drwxr-xr-x 16 root root 4096 Nov 6 2003 ..
-rw------- 1 root dave 254 Jul 31 12:53 dave
-rw------- 1 root dbadmin 280 Aug 11 10:51 dbadmin
The crontab
command has the setuid
bit set so that non-root users can edit their own crontab
files:
# ls -l /usr/bin/crontab
-rwsr-xr-x 1 root root 110114 Feb 19 2003
/usr/bin/crontab
Try to lock down the crontab
command by changing the permissions to 555, and you receive the following error when running it as a non-root user:
[dave@sawnee dave]$ crontab -e
seteuid: Operation not permitted
Root has a special crontab
file, /etc/crontab
. The format is slightly different. The job command line has an additional field to specify what user cron
should use to run the job. The default Red Hat /etc/crontab
is:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
The /etc/crontab
supplied with Red Hat Linux makes use of the run-parts
script. On a Linux installation, there are probably several processes that should be run daily such as backups and reports. There can be weekly or monthly jobs such as reports, software updates, system scans, and so on. This is the beauty of the run-parts
script. The /etc/crontab
shown previously runs the daily jobs at 4:02 a.m., the weekly jobs at 4:22 a.m. on Sunday, and the monthly jobs at 4:42 a.m. on the first day of the month.
The /usr/bin/run-parts
script expects to be supplied with a directory path as an argument. The run-parts
script executes all the scripts contained in that directory except for those ending in rpmsave
, rpmorig
, rpmnew
, and swp
scripts. There is no man page for run-parts
, but it is a simple script to read and understand.
The functionality provided by /usr/bin/run-parts
is very important. A software package that includes a cron
job can be written to place a file in the appropriate /etc/cron.hourly
, cron.daily
, cron.weekly
, or cron.monthly
directory rather than trying to manipulate root’s crontab
file. This capability makes creating packages containing cron
jobs manageable. You can see which packages delivered cron
jobs to a Linux system with the command rpm -qa --filesbypkg grep '/etc/cron'
.
The SUSE /etc/crontab
file is similar to Red Hat’s /etc/crontab
:
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/
cron/run-crons >/dev/null 2>&1
59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly
14 4 * * * root rm -f /var/spool/cron/lastrun/cron.daily
29 4 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly
44 4 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly
SUSE Linux supplies the run-crons
script. It runs the jobs in the /etc/cron
.* directories like run-parts
does on Red Hat. cron
on SUSE is configured to use /var/spool/cron/lastrun
to keep track of the last time that the /etc/cron
.* scripts were run. cron
does not log the run-crons
job execution.
Make sure only root can write to the /etc/cron.hourly
, /etc/cron.daily
, /etc/cron.weekly
, and /etc/cron.monthly
directories. Otherwise, any user can save a script in one of these directories and use cron
to run his or her process with root privileges. Notice that the command line indicates that root should run these scripts. Even if the script in /etc/cron.daily
is owned by a user, it is executed with root privileges.
# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 Feb 19 2003 /etc/cron.d
drwxr-xr-x 2 root root 4096 Aug 10 04:08 /etc/cron.daily
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.hourly
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.monthly
drwxr-xr-x 2 root root 4096 Nov 6 2003 /etc/cron.weekly
You can confirm that the permissions of a package are correct with the rpm –V cron_package_name
command. You can confirm the permissions of a single file or directory with rpm –V -f filename
. Troubleshooting using rpm
is covered later in this chapter.
Here is a sample /etc/cron.daily
:
# ll /etc/cron.daily
total 36
-rwxr-xr-x 1 root root 135 Jan 25 2003 00webalizer
-rwxr-xr-x 1 root root 276 Jan 24 2003 0anacron
-rwxr-xr-x 1 root root 123 Jan 26 2003 inn-cron-expire
-rwxr-xr-x 1 root root 51 Jan 24 2003 logrotate
-rwxr-xr-x 1 root root 418 Feb 10 2003 makewhatis.cron
-rwxr-xr-x 1 root root 135 May 14 15:59 rpm
-rwxr-xr-x 1 root root 132 Jan 21 2004 slocate.cron
-rwxr-xr-x 1 root root 164 May 17 09:18 sophos
-rwxr-xr-x 1 root root 193 Feb 10 2003 tmpwatch
The cron
daemon (/usr/sbin/cron
for SUSE and /usr/sbin/crond
for Red Hat) is responsible for running the cron
jobs. Every minute, the cron
daemon looks at the crontabs
, loads new or modified crontabs
into memory, and checks whether any jobs are scheduled to execute during that minute. The cron
daemon is started during the startup scripts at runlevels 2, 3, 4, and 5.
#chkconfig --list crond
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The startup script is /etc/init.d/crond
in Red Hat and /etc/init.d/cron
in SUSE. For more information on startup scripts, refer to Chapter 1, “System Boot, Startup, and Shutdown Issues.”
A log entry is created when cron
jobs are run, crontab
commands are used, the cron
daemon is started, and so on. cron
uses the syslog
daemon to log cron
events. The syslog
facility for cron
messages is cron
. See the syslogd(8)
man page for an explanation of how syslog
routes messages. The log file for cron
is determined by /etc/syslog. conf
. Red Hat has a log just for cron
, /var/log/cron
, to log cron
messages. SUSE uses /var/log/messages
. A Red Hat /etc/syslog.conf
has the following entry to route cron
messages to /var/log/cron
:
# Log cron stuff
cron.* /var/log/cron
With Red Hat, cron
messages are not duplicated in /var/log/messages
because of the cron.none
on the following syslog.conf
line:
*.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages
It would be a simple matter to change SUSE systems to log cron
messages to /var/log/cron
. Here is a sample of the entries that are written to Red Hat’s /var/log/cron
file:
Aug 15 04:22:00 sawnee CROND[25198]: (root) CMD (run-parts
/etc/cron.weekly)
Aug 15 04:22:04 sawnee anacron[25202]: Updated timestamp for job
'cron.weekly' to 2004-08-15
Aug 15 05:01:00 sawnee CROND[2420]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 06:01:00 sawnee CROND[2519]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 07:01:00 sawnee CROND[2532]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 08:01:00 sawnee CROND[2544]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 09:01:00 sawnee CROND[2556]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 10:01:00 sawnee CROND[2568]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 11:01:00 sawnee CROND[2588]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 12:01:00 sawnee CROND[2600]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 13:01:00 sawnee CROND[2612]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:01:00 sawnee CROND[2910]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 15 14:35:46 sawnee crontab[3107]: (dave) LIST (dave)
Aug 15 14:36:56 sawnee crontab[3113]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) REPLACE (dave)
Aug 15 14:37:03 sawnee crontab[3113]: (dave) END EDIT (dave)
Aug 15 14:37:09 sawnee crontab[3116]: (dave) BEGIN EDIT (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) REPLACE (dave)
Aug 15 14:37:13 sawnee crontab[3116]: (dave) END EDIT (dave)
Aug 15 14:38:00 sawnee crond[1792]: (dave) RELOAD (cron/dave)
Aug 15 14:38:00 sawnee CROND[3166]: (dave) CMD (/home/dave/daily_rpt)
The sample cron
log shows cron
activity on host sawnee
. cron
events are broken out by crond
, crontab
, and CROND
(which shows jobs being started). The number between the []
is the process ID. Everything that crontab
performs is logged. Note that run-parts
is listed as one job and is not broken down by each job in /etc/cron.hourly
.
As we noted earlier, cron
and at
are similar but not identical across Linux distributions. To understand the differences, look closely at the man pages, see which packages are installed, and look for additional documentation supplied with the packages. Run rpm -qa
to see what packages are installed. This list is from a Red Hat 9.0 system:
# rpm -qa|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25
A typical SUSE 9.0 system might show:
# rpm -qa|grep cron
cron-3.0.1-824
The package name doesn’t indicate it, but this is vixie-cron
too, which can be verified in the README supplied with the package.
Look at the files delivered with the packages by running rpm -q --filesbypkg
. Take note of man pages, READMEs, and other documentation provided with each package. You can see what commands are available with the package as well as what documentation you can review. Here are the files delivered with Red Hat 9.0:
# rpm -q --filesbypkg vixie-cron-3.0.1-74
vixie-cron /etc/cron.d
vixie-cron /etc/rc.d/init.d/crond
vixie-cron /usr/bin/crontab
vixie-cron /usr/sbin/crond
vixie-cron /usr/share/man/man1/crontab.1.gz
vixie-cron /usr/share/man/man5/crontab.5.gz
vixie-cron /usr/share/man/man8/cron.8.gz
vixie-cron /usr/share/man/man8/crond.8.gz
vixie-cron /var/spool/cron
# rpm -q --filesbypkg crontabs-1.10-5
crontabs /etc/cron.daily
crontabs /etc/cron.hourly
crontabs /etc/cron.monthly
crontabs /etc/cron.weekly
crontabs /etc/crontab
crontabs /usr/bin/run-parts
SUSE 9.0 delivers the following files with the cron
package:
# rpm -q --filesbypkg cron-3.0.1-824
cron /etc/crontab
cron /etc/init.d/cron
cron /usr/bin/crontab
cron /usr/lib/cron
cron /usr/lib/cron/run-crons
cron /usr/sbin/cron
cron /usr/sbin/rccron
cron /usr/share/doc/packages/cron
cron /usr/share/doc/packages/cron/CHANGES
cron /usr/share/doc/packages/cron/CONVERSION
cron /usr/share/doc/packages/cron/FEATURES
cron /usr/share/doc/packages/cron/MAIL
cron /usr/share/doc/packages/cron/MANIFEST
cron /usr/share/doc/packages/cron/README
cron /usr/share/doc/packages/cron/THANKS
cron /usr/share/man/man1/crontab.1.gz
cron /usr/share/man/man5/crontab.5.gz
cron /usr/share/man/man8/cron.8.gz
cron /var/spool/cron
cron /var/spool/cron/deny
cron /var/spool/cron/lastrun
cron /var/spool/cron/tabs
Listing the files delivered in the package makes it easy to see the differences between distributions.
A nice graphical front-end utility is available for the crontab
command. The kcron
utility is part of KDE and is supplied with the optional kdeadmin
package. The kcron
utility is a simple, intuitive method for users to manage their cron
jobs. The English version of the kcron
handbook is available at http://docs.kde.org/en/3.1/kdeadmin/kcron/index.html. To start kcron
, choose System Tools, Task Scheduler from the main KDE menu, and you will see the interface shown in Figure 10-1.
Figure 10-1. The kcron interface
No crontab
knowledge is needed to schedule jobs when kcron
is used. Users don’t have to learn vi
just to schedule a job. kcron
inserts a header and a footer in the crontab
file, so the system administrator knows it was saved from kcron
. Here is an example:
$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/kde-susan/kcronzMmGJa.tmp installed on Tue Sep 7 09:47:40 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
#
5 4 * * * /home/susan/screening.sh
# This file was written by KCron. Copyright (c) 1999, Gary Meyer
# Although KCron supports most crontab formats, use care when editing.
# Note: Lines beginning with "#" indicates a disabled task.
There are a couple of wrinkles with kcron
. kcron
does not seem to be aware of the crontab
allow and deny files, meaning users who are restricted from using the crontab
command can still create cron
tasks.
If kcron
is run as root, the save function saves crontab
files for every user on the system even if only root’s jobs were modified. If the user already has a crontab
file, kcron
inserts its header and footer. If the user doesn’t have a crontab
file, kcron
creates one with the normal crontab
header plus the kcron
header line. We don’t recommend using kcron
as root, but it is fine for other users.
To see the list of known bugs in kcron
, visit http://bugs.kde.org/. We submitted new bug reports for those we mentioned here:
Bug 89488:
kcron
doesn’t prohibitcrontab
changes based oncron
’s allow and deny files.Bug 89491:
kcron
createscrontab
files for all users when the rootcrontab
is created.
More information about KDE is available at http://www.kde.org/.
anacron
is similar to cron
, and it works much the same way. anacron
is intended to run jobs in /etc/crontab
that cron
missed due to system downtime. If a Linux box is left on continuously, anacron
won’t run any jobs because cron
will run them. anacron
has its own crontab
file, /etc/anacrontab
. Changes made to /etc/crontab
need to be made in /etc/anacrontab
as well. anacron
is a utility for root only. There is no anacrontab
command to edit the file for users. The system administrator should edit the file manually. Here is the anacrontab
supplied with anacron
on Red Hat systems:
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
1 65 cron.daily run-parts /etc/cron.daily
7 70 cron.weekly run-parts /etc/cron.weekly
30 75 cron.monthly run-parts /etc/cron.monthly
The format is similar to the crontab
files, but there are some differences. The first few lines set the shell and path. The remaining lines contain the scripts to execute; anacron
calls these job description lines.
The first field of the job description line indicates how many days should occur between runs of the process. The second field indicates how many minutes to wait before starting the process. The third field is the job identifier. The last field and remainder of the line is the shell command to execute.
See the anacrontab(8)
man page for further details.
The anacrontab
file doesn’t execute run-parts /etc/cron.hourly
because the smallest time unit for anacron
is one day.
There is a potential problem with the previous anacrontab
file. The full path to the run-parts
script is not given. The run-parts
script is in /usr/bin
. The /usr/local/sbin
and /usr/local/bin
directories are searched before /usr/bin
according to the PATH
variable. If users have write permission to /usr/local/sbin
or /usr/local/bin
, a script called run-parts
could be placed in those directories and would run instead of the intended /usr/bin/run-parts
. This issue can be prevented by always using the full path for scripts in anacrontab
. For example:
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
1 65 cron.daily /usr/bin/run-parts /etc/cron.daily
7 70 cron.weekly /usr/bin/run-parts /etc/cron.weekly
30 75 cron.monthly /usr/bin/run-parts /etc/cron.monthly
anacron
is started by a startup script.
#chkconfig --list anacron
anacron 0:off 1:off 2:on 3:on 4:on 5:on 6:off
anacron
checks the timestamp files in /var/spool/anacron
and runs any jobs that were missed.
# ls -al /var/spool/anacron
total 20
drwxr-xr-x 2 root root 4096 Nov 6 2003 .
drwxr-xr-x 16 root root 4096 Nov 6 2003 ..
-rw------- 1 root root 9 Aug 14 12:53 cron.daily
-rw------- 1 root root 9 Aug 1 04:42 cron.monthly
-rw------- 1 root root 9 Aug 8 04:22 cron.weekly
anacron
records whether cron
is running normally so that jobs are not run twice. This is done with an anacron script in /etc/cron.daily
, /etc/cron.weekly
, and /etc/cron.monthly
, which is executed by cron
. The anacron -u
command is run by this anacron script to update the timestamps and indicate that all jobs were run for that day by cron
.
anacron
messages route to the same log file as cron
messages. anacron
logs messages using syslogd
with the facility value set to cron
. Here are some sample anacron
log entries:
Aug 18 10:43:58 sawnee anacron[2489]: Job 'cron.daily' started
Aug 18 10:43:58 sawnee anacron[2687]: Updated timestamp for job
'cron.daily' to 2004-08-18
The following files are delivered with anacron
:
# rpm -q --filesbypkg anacron-2.3-25
anacron /etc/anacrontab
anacron /etc/cron.daily/0anacron
anacron /etc/cron.monthly/0anacron
anacron /etc/cron.weekly/0anacron
anacron /etc/rc.d/init.d/anacron
anacron /usr/sbin/anacron
anacron /usr/share/doc/anacron-2.3
anacron /usr/share/doc/anacron-2.3/COPYING
anacron /usr/share/doc/anacron-2.3/README
anacron /usr/share/man/man5/anacrontab.5.gz
anacron /usr/share/man/man8/anacron.8.gz
anacron /var/spool/anacron
The /usr/bin/at
command is used to submit jobs for later execution. Jobs submitted with at
are run once at the specified time.
Specify the script or binary to run along with the time specification. Here are a few examples showing how a database administrator could submit a job to run a daily report:
at -f /prod_serv/scripts/prod_daily_report now + 1 minute
at -f /prod_serv/scripts/prod_daily_report now + 1 hour
at -f /prod_serv/scripts/prod_daily_report 13:55
at -f /prod_serv/scripts/prod_daily_report 4am + 3 days
The time specification format is very flexible. See the at(1)
man page and /usr/share/doc/at-3.1.8/timespec
for more details. Jobs submitted with at
inherit the environment variables of the shell from which they were submitted.
Several more commands are provided with the at
package. See the man page for a complete list and description. Here are a few useful examples.
To list jobs that are queued, use this command:
# atq
5 2004-08-18 04:00 a root
4 2004-08-15 14:54 a root
To remove a job with an ID number of 4, use this command:
# atrm 4
# atq
5 2004-08-18 04:00 a root
Linux provides a batch
command that works similarly to at
, except that the command runs when the load average of the system is less than .8 or the value specified with the atrun
command. See the next topic for details on atrun
. An execution time is not specified when submitting a job with the batch
command. For example:
# batch -f /prod_serv/scripts/prod_daily_report
The /etc/at.allow
and /etc/at.deny
files can be used to control which users can run at
and batch
. Their format and use are the same as cron
’s allow and deny files.
at
jobs are queued as files in /var/spool/at
in Red Hat and /var/spool/atjobs
in SUSE. The job files are plain text files.
Jobs submitted using at
are started either by a daemon atd
process or by the atrun
script. The newer method is to have atd
run as a daemon. You can determine whether atd
is started as a daemon by using chkconfig
:
#chkconfig --list atd
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
Here we see atd
is started at runlevels 3, 4, and 5. If atd
is not started as a daemon at startup, the atrun
script should be started by cron
periodically to run the at
jobs. atrun
is just a simple script to run atd
:
#! /bin/sh
prefix=/usr
exec_prefix=/usr
exec /usr/sbin/atd -s "$@"
The at
README (/usr/share/doc/at-3.1.8/README
) suggests adding a line to root’s crontab
file to start atrun
every five minutes:
* * * * 0,5,10,15,20,25,30,35,40,45,50,55 /usr/sbin/atrun
If atd
runs as a daemon, the at
jobs run instantly. The cost is one additional process as overhead, but that is a small price to pay. If atrun
is started with the previous crontab
entry, the at
jobs are checked only every five minutes. The atrun
command can be executed with the -l
load average
parameter to specify the load average at which jobs submitted with batch
will run.
Here are the files delivered with the Red Hat 9.0 at
package:
# rpm -q --filesbypkg at
at /etc/at.deny
at /etc/rc.d/init.d/atd
at /usr/bin/at
at /usr/bin/atq
at /usr/bin/atrm
at /usr/bin/batch
at /usr/sbin/atd
at /usr/sbin/atrun
at /usr/share/doc/at-3.1.8
at /usr/share/doc/at-3.1.8/ChangeLog
at /usr/share/doc/at-3.1.8/Copyright
at /usr/share/doc/at-3.1.8/Problems
at /usr/share/doc/at-3.1.8/README
at /usr/share/doc/at-3.1.8/timespec
at /usr/share/man/man1/at.1.gz
at /usr/share/man/man1/atq.1.gz
at /usr/share/man/man1/atrm.1.gz
at /usr/share/man/man1/batch.1.gz
at /usr/share/man/man5/at.allow.5.gz
at /usr/share/man/man5/at.deny.5.gz
at /usr/share/man/man8/atd.8.gz
at /usr/share/man/man8/atrun.8.gz
at /var/spool/at
at /var/spool/at/.SEQ
at /var/spool/at/spool
We present four troubleshooting scenarios to demonstrate different strategies for resolving some common problems with cron
. The examples are limited to cron
because cron
is by far the most popular of the utilities in this chapter. Also, these examples should provide the information needed to troubleshoot problems in the other utilities covered.
The first scenario shows how to use the information cron
provides. Many issues can be resolved just by understanding what the emails from cron
and cron
’s syslog
messages mean. The second scenario shows that cron
can start a process only to have it hang. The third scenario shows a problem with the cron
daemon. Finally, we cover a problem common to many log files.
The database administrator complains that his report doesn’t run. Not that our DBA would mislead us intentionally, but let’s check the crontab
to verify the job entry.
We can confirm the job entry with the crontab
command. We use the -u dbprod
option to look at another user’s crontab
. Only root can do this.
# crontab -u dbprod -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports
We could just cat
the crontab
file as well:
# cat /var/spool/cron/dbprod
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.3389 installed on Sun Aug 15 15:04:14 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
06 15 * * * /prod_serv/scripts/prod_daily_reports
The crontab
entry looks good. The next step is to check the cron
log to see if we can determine whether cron
started the job. There should be a line in /var/log/cron
similar to the following:
Aug 15 15:06:00 sawnee CROND[3400]: (dbprod) CMD
(/prod_serv/scripts/prod_daily_reports)
The cron
log shows that cron
did start the job at 15:06, and the process ID is 3400. We know the job ran. But did it finish? The crontab
command line doesn’t redirect standard output (stdout
) or standard error (stderr
), so any output should go to dbprod
’s mail.
From [email protected] Sun Aug 15 15:06:00 2004
Date: Sun, 15 Aug 2004 15:06:00 -0400
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <dbprod@sawnee> /prod_serv/scripts/prod_daily_reports
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/dbprod>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=dbprod>
/bin/sh: line 1: /prod_serv/scripts/prod_daily_reports: No such file or
directory
We can see from the message that the DBA probably made a typo. We should confirm this idea:
# ll /prod_serv/scripts/
total 12
-rwx------ 1 dbprod root 73 Aug 5 17:32 prod_backup_1
-rwx------ 1 dbprod root 42 Jul 27 18:15 prod_backup_2
-rwx------ 1 dbprod root 79 Aug 15 13:51 prod_daily_report
Looks like the DBA meant to run prod_daily_report
rather than prod_daily_reports
.
If dbprod
’s crontab
redirects stdout (standard out)
and stderr (standard error)
to /dev/null
, the output is thrown away. 1
is always the file number of stdout
, and 2
is the file number for stderr
. The >/dev/null 2>&1
on the following crontab
entry means make stdout /dev/null
and make stderr
the same file as stdout
, which throws away the script output and any error messages:
06 15 * * * /prod_serv/scripts/prod_daily_reports >/dev/null 2>&1
The /dev/null
file is not a real file. The device file throws away anything written to it:
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Oct 2 2003 /dev/null
cron
jobs often redirect output to /dev/null
to avoid having to trim a log file or delete lots of emails from cron
. If the cron
job stops working, the crontab
line should be modified to send stdout
and stderr
to a log file so that any output from the script can be examined:
06 15 * * * /prod_serv/scripts/prod_daily_reports >/tmp/dbprod.out 2>&1
The crontab
could be changed to email the user with any output:
06 15 * * * /prod_serv/scripts/prod_daily_reports
Set the time fields so that the job runs again and see whether any output is generated.
Ted, a software engineer, complains that his code sweep didn’t run. Using the troubleshooting steps from the first scenario, we verify that Ted’s job should have been run at 7:47 a.m.:
# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5691 installed on Mon Aug 16 07:45:59 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
47 07 * * * /home/ted/sweep.sh
The job was started according to the cron
log:
Aug 16 07:47:00 sawnee CROND[5697]: (ted) CMD (/home/ted/sweep.sh)
The user ted
has not received email from cron
. We know the job was started but never finished. Thus, it is probably still running. The pstree
command is a good way to see what cron
is doing. pstree
shows process parent-child relationships in a tree format. Here is the entry for ted
’s job:
If necessary, we could look at the output from ps -ef
to see more detail. We can see from the cron
log that the process ID for the job is 5697. The process might be working normally or could be hanging, but cron
is doing its job.
Ted’s back. He says his code sweep didn’t run. Running through the steps, we check the crontab
and cron
log.
The crontab
has a job that should have run at 9:15 a.m.
[root@sawnee root]# crontab -u ted -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.5962 installed on Mon Aug 16 09:13:24 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
15 09 * * * /home/ted/sweep.sh
Ted modified his crontab
this morning but was finished before 9:15. cron
should have run the job but apparently did not because there is no line in the cron
log indicating that the job was started.
Aug 16 09:12:48 sawnee crontab[5962]: (ted) BEGIN EDIT (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) REPLACE (ted)
Aug 16 09:13:24 sawnee crontab[5962]: (ted) END EDIT (ted)
Next, we check whether crond
is running:
# ps -ef|grep cron
root 5977 5832 0 09:23 pts/7 00:00:00 grep cron
The crond
binary is not running. Let’s restart it and then try to figure out what happened.
# /etc/init.d/crond start
Starting crond: [ OK ]
We can see crond
is now running.
# ps -ef|grep cron
root 6001 1 0 09:31 ? 00:00:00 crond
root 6005 5832 0 09:31 pts/7 00:00:00 grep cron
We check /var/log/cron
, and it seems crond
is working:
Aug 16 09:31:14 sawnee crond[6001]: (CRON) STARTUP (fork ok)
We probably need to run a test cron
job just to confirm that cron
is working. If so, we move on and try to determine what happened. We check /var/log/messages
for system messages that may show a problem. We verify that the cron
packages are installed properly.
What cron
packages can we find?
# rpm -q -a|grep cron
crontabs-1.10-5
vixie-cron-3.0.1-74
anacron-2.3-25
The crond
binary is probably delivered with vixie-cron
. We confirm this suspicion:
# rpm -q -filesbypkg vixie-cron-3.0.1-74
vixie-cron /etc/cron.d
vixie-cron /etc/rc.d/init.d/crond
vixie-cron /usr/bin/crontab
vixie-cron /usr/sbin/crond
vixie-cron /usr/share/man/man1/crontab.1.gz
vixie-cron /usr/share/man/man5/crontab.5.gz
vixie-cron /usr/share/man/man8/cron.8.gz
vixie-cron /usr/share/man/man8/crond.8.gz
vixie-cron /var/spool/cron
We see /usr/sbin/crond
is delivered with vixie-cron
. Now, we just make sure vixie-cron-3.0.1-74
is installed properly:
# rpm -V vixie-cron-3.0.1-74
No output means that the package is installed properly. Maybe someone with root access killed crond
by accident. We could also check for updates to the package that might correct a new bug.
See Chapter 8, “Linux Processes: Structure, Hangs, and Core Dumps,” for further troubleshooting guidance.
Users say cron
is working fine, but a system administrator notices that the cron
log isn’t being updated. The cron
log is zero bytes in size.
# ls -l /var/log/cron
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
Let’s run a command to generate a cron
log update.
# crontab -l
no crontab for root
That should have generated an entry in the cron
log even though root has no crontab
file, but it didn’t.
# ls -l /var/log/cron
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
We know cron
uses syslogd
to do its logging. Let’s make sure syslogd
is running and working:
# ps -ef|grep syslog
root 1566 1 0 Aug23 ? 00:00:00 syslogd -m 0
root 28889 28848 0 20:21 pts/3 00:00:00 grep syslog
We can see syslogd
is running. Is it still logging messages?
# tail -3 /var/log/messages
Aug 31 17:36:59 sawnee sshd(pam_unix)[28511]: session closed for user
root
Aug 31 20:19:36 sawnee sshd(pam_unix)[28846]: session opened for user
root by (uid=0)
From the previous output, we can see syslogd
was working as of Aug 31 8:19 p.m. If there were no recent entries in the messages log, we could use the logger
command to generate a test message:
# logger -p kern.info "test from sysadmin"
# tail -1 /var/log/messages
Aug 31 20:30:47 sawnee root: test from sysadmin
Where do cron
messages go?
# grep cron /etc/syslog.conf
*.info;mail.none;news.none;authpriv.none;cron.none
/var/log/messages
# Log cron stuff
cron.* /var/log/cron
Messages should be routed to /var/log/cron
. Because syslogd
is doing its job, something must be wrong with the cron
file itself.
Someone with root access might have tried to trim the file incorrectly by removing or moving it and then touching cron
to start a new log. If we look for cron
files, we can see a cron.bkup
that seems out of place.
# ls -l /var/log/cron*
-rw------- 1 root root 0 Aug 31 20:26 /var/log/cron
-rw------- 1 root root 19795 Aug 29 04:02 /var/log/cron.1
-rw------- 1 root root 13506 Aug 22 04:02 /var/log/cron.2
-rw------- 1 root root 19512 Aug 15 04:02 /var/log/cron.3
-rw------- 1 root root 12288 Jul 28 22:01 /var/log/cron.4
-rw------- 1 root root 10254 Aug 31 20:27 /var/log/cron.bkup
Sure enough, our crontab
entry went to the old file.
# til -1 /var/log/cron.bkup
Aug 31 20:27:19 sawnee crontab[28906]: (root) LIST (root)
Moving a file using the mv
command within the same filesystem while a file is open for writing generally causes problems. The program attached to the open file descriptor continues writing to the old file until the file descriptor is closed. This happens even though the filename has changed. The file descriptor points to the open inodes on the disk. In this instance, even though the file has a new name, syslogd
keeps writing to it. The solution is to restart syslogd
or put the file back. The syslogd(8)
man page tells us to restart syslogd
with:
kill -SIGHUP 'cat /var/run/syslogd.pid'
This command tells syslog
to close and open the files again and fixes the logging issue. More syslog
information can be found in the syslogd(8)
and syslog.conf(5)
man pages.
# crontab -l
no crontab for root
# tail -1 /var/log/cron
Aug 31 20:39:49 sawnee crontab[28949]: (root) LIST (root)
The cron
log is updating properly.
The flow chart in Figure 10-2 shows the troubleshooting methodology that was used in the examples.
Figure 10-2. Troubleshooting methodology flow chart
It is necessary to understand how applications work before they can be effectively troubleshot. Our goal for this chapter was to explain how cron
, at
, anacron
, and kcron
work so that you will be ready when problems occur. The scenarios build on the foundation provided by the earlier sections. They present step-by-step resolutions for some common problems.
3.148.104.124