Globbing and the for loop

Now, let's look at a few more practical examples. Most things you will do on Linux will deal with files (remember why?). Imagine that you have a bunch of log files sitting on the server and you'd like to perform some actions on them. If it is just a single action with a single command, you can most probably use a globbing pattern with that command (such as with grep -i 'error' *.log). However, imagine a situation where you want to collect log files that contain a certain phrase, or perhaps only the lines from those files. In this case, using a globbing pattern in combination with a for loop will allow us to perform many commands on many files, which we can find dynamically! Let's give it a go. Because this script will combine many of the lessons we've covered so far, we'll begin simple and expand it gradually:

reader@ubuntu:~/scripts/chapter_11$ vim for-globbing.sh 
reader@ubuntu:~/scripts/chapter_11$ cat for-globbing.sh 
#!/bin/bash

#####################################
# Author: Sebastiaan Tammer
# Version: v1.0.0
# Date: 2018-10-27
# Description: Combining globbing patterns in a for loop.
# Usage: ./for-globbing.sh 
#####################################

# Create a list of log files.   
for file in $(ls /var/log/*.log); do
  echo ${file}
done

reader@ubuntu:~/scripts/chapter_11$ bash for-globbing.sh 
/var/log/alternatives.log
/var/log/auth.log
/var/log/bootstrap.log
/var/log/cloud-init.log
/var/log/cloud-init-output.log
/var/log/dpkg.log
/var/log/kern.log

By using the $(ls /var/log/*.log) construct, we can create a list of all files that end in .log that are found in the /var/log/ directory. If you manually run the ls /var/log/*.log command, you will notice that the format is the same as the others we've seen when used in the Bash-style for syntax: single words, whitespace delimited. Because of this, we can now manipulate all of the files we found in order! Let's see what happens if we try to grep in these files:

reader@ubuntu:~/scripts/chapter_11$ cat for-globbing.sh 
#!/bin/bash

#####################################
# Author: Sebastiaan Tammer
# Version: v1.1.0
# Date: 2018-10-27
# Description: Combining globbing patterns in a for loop.
# Usage: ./for-globbing.sh 
#####################################

# Create a list of log files.   
for file in $(ls /var/log/*.log); do
  echo "File: ${file}"
  grep -i 'error' ${file}
done

Since we changed the content of the script, we've upped the version from v1.0.0 to v1.1.0. If you run this script now, you'll see that some of files return a positive match on the grep, while others do not:

reader@ubuntu:~/scripts/chapter_11$ bash for-globbing.sh 
File: /var/log/alternatives.log
File: /var/log/auth.log
File: /var/log/bootstrap.log
Selecting previously unselected package libgpg-error0:amd64.
Preparing to unpack .../libgpg-error0_1.27-6_amd64.deb ...
Unpacking libgpg-error0:amd64 (1.27-6) ...
Setting up libgpg-error0:amd64 (1.27-6) ...
File: /var/log/cloud-init.log
File: /var/log/cloud-init-output.log
File: /var/log/dpkg.log
2018-04-26 19:07:33 install libgpg-error0:amd64 <none> 1.27-6
2018-04-26 19:07:33 status half-installed libgpg-error0:amd64 1.27-6
2018-04-26 19:07:33 status unpacked libgpg-error0:amd64 1.27-6
<SNIPPED>
File: /var/log/kern.log
Jun 30 18:20:32 ubuntu kernel: [    0.652108] RAS: Correctable Errors collector initialized.
Jul  1 09:31:07 ubuntu kernel: [    0.656995] RAS: Correctable Errors collector initialized.
Jul  1 09:42:00 ubuntu kernel: [    0.680300] RAS: Correctable Errors collector initialized.

Great, so now we've accomplished the same thing with a complicated for loop that we could have also done directly with grep! Now, let's get our money's worth and do something with the files after we've determined that they contain the word error:

reader@ubuntu:~/scripts/chapter_11$ vim for-globbing.sh 
reader@ubuntu:~/scripts/chapter_11$ cat for-globbing.sh 
#!/bin/bash

#####################################
# Author: Sebastiaan Tammer
# Version: v1.2.0
# Date: 2018-10-27
# Description: Combining globbing patterns in a for loop.
# Usage: ./for-globbing.sh 
#####################################

# Create a directory to store log files with errors.
ERROR_DIRECTORY='/tmp/error_logfiles/'
mkdir -p ${ERROR_DIRECTORY}

# Create a list of log files. 
for file in $(ls /var/log/*.log); do
 grep --quiet -i 'error' ${file}

 # Check the return code for grep; if it is 0, file contains errors.
 if [[ $? -eq 0 ]]; then
 echo "${file} contains error(s), copying it to archive."
 cp ${file} ${ERROR_DIRECTORY} # Archive the file to another directory.
 fi

done

reader@ubuntu:~/scripts/chapter_11$ bash for-globbing.sh 
/var/log/bootstrap.log contains error(s), copying it to archive.
/var/log/dpkg.log contains error(s), copying it to archive.
/var/log/kern.log contains error(s), copying it to archive.

The next version, v1.2.0, does a quiet grep (no output, since we just want the exit status of 0 when something is found). Directly after the grep, we use a nested if-then to copy the files to an archive directory that we defined at the beginning of the script. When we run the script now, we can see the same files that generated output in the previous version of the script, but now it copies the entire file. At this point, the for loop is proving its value: we're now doing multiple operations on a single file that was found with the globbing pattern. Let's take this one step further and remove all of the lines that do not contain an error from the archived files:

reader@ubuntu:~/scripts/chapter_11$ vim for-globbing.sh 
reader@ubuntu:~/scripts/chapter_11$ cat for-globbing.sh 
#!/bin/bash

#####################################
# Author: Sebastiaan Tammer
# Version: v1.3.0
# Date: 2018-10-27
# Description: Combining globbing patterns in a for loop.
# Usage: ./for-globbing.sh 
#####################################

# Create a directory to store log files with errors.
ERROR_DIRECTORY='/tmp/error_logfiles/'
mkdir -p ${ERROR_DIRECTORY}

# Create a list of log files.   
for file in $(ls /var/log/*.log); do
  grep --quiet -i 'error' ${file}

  # Check the return code for grep; if it is 0, file contains errors.
  if [[ $? -eq 0 ]]; then
    echo "${file} contains error(s), copying it to archive ${ERROR_DIRECTORY}."
    cp ${file} ${ERROR_DIRECTORY} # Archive the file to another directory.

    # Create the new file location variable with the directory and basename of the file.
    file_new_location="${ERROR_DIRECTORY}$(basename ${file})"
    # In-place edit, only print lines matching 'error' or 'Error'.
    sed --quiet --in-place '/[Ee]rror/p' ${file_new_location} 
  fi

done

Version v1.3.0! To keep it a little readable, we have not included error checking on the cp and mkdir commands. However, due to the nature of this script (creating a subdirectory in /tmp/ and copying files there), the chance of issues there is very slim. We added two new interesting things: a new variable called file_new_location with the file name of the new location and sed, which ensures only the error lines remain in the archived files.

First, let's consider file_new_location=${ERROR_DIRECTORY}$(basename ${file}). What we're doing is pasting together two strings: first, the archive directory, followed by the basename of the processed file. The basename command strips the fully qualified path of a file, and only leaves the file name at the leaf of the path intact. If we were to look at the steps that Bash will undertake to resolve this new variable, it would probably look something like this:

  • file_new_location=${ERROR_DIRECTORY}$(basename ${file})
    -> resolve ${file}
  • file_new_location=${ERROR_DIRECTORY}$(basename /var/log/bootstrap.log)
    -> resolve $(basename /var/log/bootstrap.log)
  • file_new_location=${ERROR_DIRECTORY}bootstrap.log
    -> resolve ${ERROR_DIRECTORY}
  • file_new_location=/tmp/error_logfiles/bootstrap.log
    -> done, final value of variable!

With that out of the way, we can now run sed on that new file. The sed --quiet --in-place '/[Ee]rror/p' ${file_new_location} command simply replaces the content of the file with all lines that match the regular expression search pattern of [Ee]rror, which is (almost) what we initially grepped for. Remember, we need --quiet because, by default, sed prints all lines. If we were to omit this, we would end up with all of the lines in the file, but all of the error files would be duplicated: once from the non-quiet output of sed, and once from the search pattern match. However, with --quiet active, sed only prints the matching lines and writes those to the files. Let's see this in practice and verify the outcome:

reader@ubuntu:~/scripts/chapter_11$ bash for-globbing.sh 
/var/log/bootstrap.log contains error(s), copying it to archive /tmp/error_logfiles/.
/var/log/dpkg.log contains error(s), copying it to archive /tmp/error_logfiles/.
/var/log/kern.log contains error(s), copying it to archive /tmp/error_logfiles/.
reader@ubuntu:~/scripts/chapter_11$ ls /tmp/error_logfiles/
bootstrap.log  dpkg.log  kern.log
reader@ubuntu:~/scripts/chapter_11$ head -3 /tmp/error_logfiles/*
==> /tmp/error_logfiles/bootstrap.log <==
Selecting previously unselected package libgpg-error0:amd64.
Preparing to unpack .../libgpg-error0_1.27-6_amd64.deb ...
Unpacking libgpg-error0:amd64 (1.27-6) ...

==> /tmp/error_logfiles/dpkg.log <==
2018-04-26 19:07:33 install libgpg-error0:amd64 <none> 1.27-6
2018-04-26 19:07:33 status half-installed libgpg-error0:amd64 1.27-6
2018-04-26 19:07:33 status unpacked libgpg-error0:amd64 1.27-6

==> /tmp/error_logfiles/kern.log <==
Jun 30 18:20:32 ubuntu kernel: [    0.652108] RAS: Correctable Errors collector initialized.
Jul  1 09:31:07 ubuntu kernel: [    0.656995] RAS: Correctable Errors collector initialized.
Jul  1 09:42:00 ubuntu kernel: [    0.680300] RAS: Correctable Errors collector initialized.

As you can see, the three lines at the top of each file all contain the error or Error string. Actually, all of the lines in all of those files contained either of those strings; be sure to verify this on your own system since the content will undoubtedly be different.

Now that we've finished this example, we have a few challenges for the reader, should you like to take them on:

  • Make this script accept input. This could be the archive directory, the path glob, the search pattern, or even all three!
  • Make this script more robust by adding exception handling to commands that could fail.
  • Invert the functionality of this script, by using the sed '/xxx/d' syntax (hint: you'll probably need redirection for this).
While this example should illustrate a lot of things, we realize that just searching on the word error does not actually only return errors. Actually, most of what we saw being returned was related to an installed package, liberror! In practice, you might be working with log files that have a predefined structure when it comes to errors. In this case, it is much easier to determine a search pattern that only logs real errors.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
52.14.172.93