Sometimes, we need to fire a job at a specific hour without any need to repeat the action, so just a one off. What we can use in this case is a simple utility called at with its companion batch. What does it do? It simply reads from the input or a file on what to execute and when, and it will use /bin/sh to invoke whatever we want. There is a little twist though: batch will do it but not at a specific time. It will be done when the system load drops below 1.5 or any level specified at the atd runtime.

So, we introduced atd; what is this? This is the daemon that executes the one shot jobs defined and put in its queue by the at utility, and so, it is a daemon that usually runs under a dedicated daemon user:

root:# ps -fC atd
UID PID PPID C STIME TTY TIME CMD
daemon 722 1 0 Apr25 ? 00:00:00 /usr/sbin/atd -f

So, this is a daemon that is fired up as a system service, but we have some other options; we can pass it to modify how it deals with the scheduled jobs. Let's see what we can make use of:

• -l: Defines the load factor over which the scheduled jobs will not be run. If no limit is imposed, it defaults to 1.5; but, in systems with x CPUs, a good limit value could be higher than x-1.
• -b: This sets the minimum interval in seconds between two consecutive jobs being fired. This defaults to 60.
• -d: Having issues here? This will enable a debug feature that will divert the error messages from syslog to stderr, usually the terminal. This option goes along with the -f option.
• -f: Good for debugging, this options forces atd to run in the foreground.
• -s: Forces the processing of the at and batch queue to only a single run. This is used for backward compatibility with older versions of at; and it is run as we ran the old atrun command.

Few files are involved in the atd running process:

• /var/spool/cron/atjobs: This is the directory where the jobs created by at are stored and atd will read the queue from. It must be owned by the same owner of the process (in our case, daemon) and have strict access rights of 700.
• /var/spool/cron/atspool: This is the directory that temporarily holds the output of the jobs. It is owned by a daemon user with an access mode of 700.
• /etc/at.allow, /etc/at.deny: In this file, we can set which account can submit jobs to atd and which are forbidden.

There is one limit in using atd; if it's spool directory is mounted with NFS, then whichever option you use for mounting, it will simply not work.

We saw a couple of interesting files:

/etc/at.allow
/etc/at.deny

This administrator who can submit jobs to the atd daemon can have a really simple syntax: it is a simple list of account names, one per line, with no whitespaces. There is a precedence order in which the files are parsed:/etc/at.allow is the first one to be read. If any account names are found in it, these will be the only accounts to be allowed to submit the jobs.

If /etc/at.allow does not exist, then /etc/at.deny is parsed and every account name found in it will be forbidden to send jobs to atd. If the file exists but it is empty, it is interpreted as that every account can submit jobs to atd.

If neither /etc/at.allow or /etc/at.deny exists, this means that only the superuser can submit jobs to atd.

In the systems, we use a sandbox. We have /etc/at.deny and its contents here:

root:# cat /etc/at.deny 
alias
backup
bin
daemon
ftp
games
gnats
guest
irc
lp
mail
man
nobody
operator
proxy
qmaild
qmaill
qmailp
qmailq
qmailr
qmails
sync
sys
www-data

So for the rules we just saw, if /etc/at. allow is absent, all the accounts listed in /etc/at.deny are forbidden to submit jobs to atd. This makes sense as we can see that these are accounts related to services or a nobody user, and these are not supposed to have any need to submit a job.

This is all we need to know about the service part of the at facility; let's see what we can rely on to schedule a job. On the client side, we have the following utilities that we can use to submit jobs to atd.

The at and batch read commands from a standard input or a specified file, which are to be executed at a later time, using /bin/sh.

• at: This is the main utility we deal with, and its function is to submit jobs that are to be executed at a specific time. Time specification can be really smart and flexible.
• HH:MM: This specifies the time of the current day to run a command called by at. If the time has already passed, the command is intended to be run on the next day at the same hour.
• midnight: The job is meant to be run at midnight:

root:# at midnight
warning: commands will be executed using /bin/sh
at> ls
at> <EOT>
job 6 at Fri Apr 28 00:00:00 2017

To submit a job, simply press Ctrl+D on a newline:

• noon: This job is meant to be run at noon
• teatime: This job will be run at 4 PM
• AM-PM: We can add a trailing AM or PM to have the job be executed at a certain hour in the morning or in the afternoon:

root:# echo "systemctl restart ssh" | at 04:43 AM
warning: commands will be executed using /bin/sh
job 7 at Thu Apr 27 04:43:00 2017
root:# echo "systemctl restart ssh" | at 04:43 PM
warning: commands will be executed using /bin/sh
job 8 at Thu Apr 27 16:43:00 2017

• date month_name [year]: We can also have a job running at a specific time on a precise day, month, and optionally, a year too; but, we have to keep in mind that whatever format we choose, the date must follow the time specification and not precede it:

root:# echo "systemctl restart ssh" | at 11:45 Aug 28
warning: commands will be executed using /bin/sh
job 12 at Mon Aug 28 11:45:00 2017
root:# echo "systemctl restart ssh" | at 11:45 Aug 28 2018
warning: commands will be executed using /bin/sh
job 13 at Tue Aug 28 11:45:00 2018

• MMDD[CC]YY: Date specification as month, day, optional century, and year without spaces:

root:# echo "systemctl restart ssh" | at 11:45 070527
warning: commands will be executed using /bin/sh
job 14 at Mon Jul 5 11:45:00 2027

• MM/DD/[CC]YY: Date specification as month/day/optional century/year separated by a slash:

root:# echo "systemctl restart ssh" | at 07:23 PM 08222017
warning: commands will be executed using /bin/sh
job 15 at Tue Aug 22 19:23:00 2017

• DD.MM.[CC]YY: Date specification as day.month.optional century.year separated by a dot:

root:# echo "systemctl restart ssh" | at 18:05 15.09.29
warning: commands will be executed using /bin/sh
job 17 at Sat Sep 15 18:05:00 2029

• [CC]YY-MM-DD: Date specification as optional century year-month-day, separated by a dash:

root:# echo "systemctl restart ssh" | at 18:15 17-09-15
warning: commands will be executed using /bin/sh
job 18 at Fri Sep 15 18:15:00 2017

• now + minutes | hours | days | weeks: We can also set the time in minutes, hours, days, or weeks from the time/date on the system at the moment of creating the job:

root:# date ; echo "systemctl restart ssh" | at now + 1 minutes 
Thu Apr 27 05:22:11 EDT 2017
warning: commands will be executed using /bin/sh
job 20 at Thu Apr 27 05:23:00 2017
root:# date ; echo "systemctl restart ssh" | at now + 1 days
Thu Apr 27 05:22:59 EDT 2017
warning: commands will be executed using /bin/sh
job 21 at Fri Apr 28 05:22:00 2017
root:# date ; echo "systemctl restart ssh" | at now + 2 weeks
Thu Apr 27 05:23:05 EDT 2017
warning: commands will be executed using /bin/sh
job 22 at Thu May 11 05:23:00 2017

• today: We can set a job to be run at a specific hour relative to today; if we do not define an hour, it will be simply executed immediately:

root:# date ; echo "systemctl restart ssh" | at today
Thu Apr 27 05:25:06 EDT 2017
warning: commands will be executed using /bin/sh
job 23 at Thu Apr 27 05:25:00 2017
root:# date ; echo "systemctl restart ssh" | at 14:28 today
Thu Apr 27 05:25:19 EDT 2017
warning: commands will be executed using /bin/sh
job 24 at Thu Apr 27 14:28:00 2017

• tomorrow: We can set a job to be run at a specific hour the next day; if we do not define an hour, it will be simply executed at the same hour that the job has been created, but the next day:

root:# date ; echo "systemctl restart ssh" | at tomorrow
Thu Apr 27 05:27:34 EDT 2017
warning: commands will be executed using /bin/sh
job 25 at Fri Apr 28 05:27:00 2017
root:# date ; echo "systemctl restart ssh" | at 14:28 tomorrow
Thu Apr 27 05:27:41 EDT 2017
warning: commands will be executed using /bin/sh
job 26 at Fri Apr 28 14:28:00 2017

The complete reference for time specification is available at /usr/share/doc/at/timespec.

As we can see from the examples, the commands are read from stdin or from a file, if using the option -f filename:

root:# echo "systemctl restart ssh" > /root/atjob1
root:# at -f /root/atjob1 14:28 tomorrow
warning: commands will be executed using /bin/sh
job 27 at Fri Apr 28 14:28:00 2017 

Once the job is put in the queue, it retains some bits from the moment it was created:

The environment variables except for BASH_VERSINFO, DISPLAY, EUID, GROUPS, SHELLOPTS, TERM, UID, _ , and umask are retained from the time of invocation.

the working directory.
the umask.

What kind of environment libraries are exported to the job depends on the future developments; but, for instance, if we want to schedule a compile job for some program sources, we will have to set such libraries as LD_LIBRARY_PATH or LD_PRELOAD from inside the job itself since they are not inherited.

Once the job is executed, its results will be displayed to stdout and stderr and mailed to the user using /usr/sbin/sendmail, but, if it is executed from su and at and retains its original user id, the results will be mailed to the user who originally logged in to su.

Let's run a simple command that will generate some output, and then let's make sure an email is sent:

root:# echo "df" | at -m now

And now, let's check the mailbox for the default alias user for root (check /etc/aliases to know whom the emails to root will be delivered to):

From root@debian Thu Apr 27 07:56:28 2017
Return-path: <root@debian>
Envelope-to: root@debian
Delivery-date: Thu, 27 Apr 2017 07:56:28 -0400
Received: from root by spoton with local (Exim 4.84_2)
 (envelope-from <root@debian>)
 id 1d3i28-0002vS-Ep
 for root@debian; Thu, 27 Apr 2017 07:56:28 -0400
Subject: Output from your job 37
To: root@debian
Message-Id: <E1d3i28-0002vS-Ep@spoton>
From: root <root@debian>
Date: Thu, 27 Apr 2017 07:56:28 -0400
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 117913932 12476420 99424792 12% /
udev 10240 0 10240 0% /dev
tmpfs 779256 9192 770064 2% /run
tmpfs 1948140 0 1948140 0% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1948140 0 1948140 0% /sys/fs/cgroup
tmpfs 389628 0 389628 0% /run/user/0

Now that we saw how to schedule a job, let's see what options it supports:

• -q: Force at to use the specified queue to place a job. Queues are designated with a single character, from a to z and from A to Z; the default queue is named after a for at and b for batch. Queues with higher letters have an increased niceness while there is a special queue named = , which is reserved for jobs that are actually running. If a job is submitted to a queue with a capital letter, it is treated as if it was submitted to batch; so, once the time specification is hit, the job is executed only if the load average of the system is below the threshold.
• -V: Just prints the version number of the utility to stderr and exit successfully.
• -m: Sends an email to the user once the job has been completed, even if the job itself has no output.
• -M: Never sends any emails to the user.
• -f filename: We already saw this option; it forces at to read from the specified file the commands to run within the job, rather than from the stdin.
• -t [[CC]YY]MMDDhhmm[.ss]: Defines the time to run the job named at/.
• -l: This is actually an alias for atq.
• -r: This is actually an alias for atrm.
• -d: This is actually an alias for atrm.
• -b: This is actually an alias for batch.
• -v: Shows when a job will be executed before reading it:

root:# at -vf /root/atjob1 14:28 tomorrow
Fri Apr 28 14:28:00 2017
warning: commands will be executed using /bin/sh
job 31 at Fri Apr 28 14:28:00 2017

• -c: Shows on the stdout the specified job:

root:# at -c 21
#!/bin/sh
# atrun uid=0 gid=0
# mail root 0
umask 22
XDG_SESSION_ID=68; export XDG_SESSION_ID
SSH_CLIENT=192.168.0.10 32994 9999; export SSH_CLIENT
SSH_TTY=/dev/pts/0; export SSH_TTY
USER=root; export USER
MAIL=/var/mail/root; export MAIL
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin; export PATH
PWD=/root; export PWD
LANG=en_US.UTF-8; export LANG
SHLVL=1; export SHLVL
HOME=/root; export HOME
LOGNAME=root; export LOGNAME
SSH_CONNECTION=192.168.0.10 32994 192.168.0.5 9999; export SSH_CONNECTION
XDG_RUNTIME_DIR=/run/user/0; export XDG_RUNTIME_DIR
cd /root || {
 echo 'Execution directory inaccessible' >&2
 exit 1
}
systemctl restart ssh

• batch: Runs a job when the average system load is under a specific threshold, the default threshold defaulting to 1.5. There is a major caveat though: in order to work correctly, batch depends on Linux on a proc filesystem mounted on /proc:

root:# echo "systemctl restart ssh" | batch 
warning: commands will be executed using /bin/sh
job 33 at Thu Apr 27 07:08:00 2017

• atq: Shows a list of pending job, for the user. If run by superuser, it shows the list of all the scheduled jobs for all the accounts. The format of the list is here:

job_id, date, hour, queue, account name

Here, what the unprivileged users see is as follows:

root:# atq
3 Thu Apr 27 08:00:00 2017 a zarrelli

And this is what root sees at the same time on the same system:

root:# atq
2 Wed Apr 26 14:00:00 2017 a root
3 Thu Apr 27 08:00:00 2017 a zarrelli

As we can see from the list, the queue with the name of a, and indeed queues, are designated with a single character, from a to z and from A to Z, the default queue being named after a for at and b for batch. Queues with higher letters have an increased niceness, while there is a special queue named = , which is reserved for jobs actually running. If atq is given a specific queue as an argument, it will show only the jobs in that queue. Let's see what atq can show without arguments:

root:# atq 
5 Wed Apr 26 05:46:00 2017 b root
2 Wed Apr 26 14:00:00 2017 a root
3 Thu Apr 27 08:00:00 2017 a zarrelli

Now, let's restrict the list to just the batch queue:

root:# atq -q b
5 Wed Apr 26 05:46:00 2017 b root

So, atq supports the following options:

• -q: It is the only option, along with V, accepted by atq and restricts its output to the specified queue content
• -V: Just prints the version number of the utility to the stderr and exit successfully
• atrm: Deletes jobs identified by the job id:

root:# atq
9 Wed Jan 31 11:45:00 2018 a root
3 Thu Apr 27 08:00:00 2017 a zarrelli
13 Tue Aug 28 11:45:00 2018 a root
29 Fri Apr 28 14:28:00 2017 a root

So, we have four jobs whose ids are 3, 9, 13, and 29; let's remove them:

root:# atrm 3 9 13 29
And check the queue again:
root:# atq
root:#

Nothing left, we are done. On a final note, atrm accepts only one option, that is -V.

What we saw so far is good for a one shot job, since at will not allow us to set some recurring task; so, if we want to execute some recurring jobs over time, we need to resort to a different kind of facility: the well-known cron service.