7
EXAMINATION OF INSTALLED SOFTWARE PACKAGES

This chapter covers the analysis of software installed on a Linux system, which includes software copied during the initial creation of a Linux system and software packages installed, updated, and removed during normal system administration. From a digital forensics perspective, we are interested in when software packages were installed on a system, what was installed, who installed them, and why. These same questions apply to software that has been removed (uninstalled). Linux systems and package managers have package databases and logs with timestamps that help to answer these questions.

In the very early days of Linux, there were no installation GUIs or package management systems. People installed software by downloading source files directly from the developer (usually via FTP), compiling source files into binaries, and installing them with provided install scripts, make install commands, or even just simple file copying. Fetching and installing software dependencies was done manually after reading the requirements listed in the documentation (README files, and so on). The initial installation was a similar manual process. Partitions and filesystems were created by hand, system directories were made, the kernel was copied into place, and the bootloader was installed. You can still experience this manual process today with the Linux From Scratch (LFS)¹ distribution, which is also an excellent way to learn Linux in depth.

Some of a Linux distribution’s defining features include its installation process and its package management system. These areas of Linux largely lack common standardization, and most distributions still have their own tools, scripts, remote package repositories, local package databases, and package file formats.

The Linux community is experiencing some fundamental changes in how it manages software. Some distributions are now using a rolling-release model, in which the system is updated as new software becomes available without having fixed version numbers or release dates. This model allows users to have the latest versions of software with the newest features and security fixes. Gentoo and Arch Linux were the first major distros to pioneer the rolling release concept. Complexity and compatibility has driven another change toward software bundled in self-contained archives with all the files needed to function (including files that are normally shared, like libraries). Both of these software packaging concepts are interesting from a forensics perspective, and digital evidence can be found in the metadata and logfiles.

Most distros use a traditional software development life cycle which has well-defined release dates, names, and version numbers. Version numbers are especially important when analyzing compromised systems and intrusions. Known vulnerabilities in a particular software version can be potentially linked to malicious activity and exploitation. This vulnerability identification also applies to rolling release distros, as they install released versions of individual software packages or Git-cloned packages from a specific date.

System Identification

When a Linux PC, laptop, or acquired image file arrives in your forensic lab for analysis, one of the first tasks is to determine which Linux distribution is installed. This knowledge helps focus an investigation along a more distro-specific analysis. Other artifacts to look for are unique identifiers that can be used to link and corroborate evidence from multiple sources. For example, a randomly generated unique identification string created during installation might be used to positively identify the machine in backup archives or in logs found on other machines.

Distro Release Information

The typical software development life cycle involves releasing software at distinct points in time, with alphas, betas, release candidates, and releases. This model includes pre-release testing, a fixed (frozen) stable release, and post-release updates. Fixed releases provide a higher degree of stability and allow for easier support. The distro version number is independent of the kernel version (even though it’s the kernel that makes it Linux in the first place). The individual software packages each have their own version numbers, which are also independent of the distro version number.

Modern Linux installations based on systemd provide detailed release information in the /etc/os-release file (usually a symlink to /usr/lib/os-release); for example:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

This file is designed to be readable from shell scripts (each line is an assigned variable). The variables in this example are mostly self-explanatory, but you can see the os-release(5) man page for more information. A systemd-based distro may also place information about the local machine (location, deployment, and so on) in the /etc/machine-info file. See the machine-info(5) man page for more information.

The Linux Standard Base (LSB) also defines /etc/distro.release and /etc/ lsb-release files that provide distro release information, and some distributions may include LSB information files. See the lsb_release(1) man page and lsb_release source code (it is a simple script) for more information. Here is one example:

$ cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=20
DISTRIB_CODENAME=ulyana
DISTRIB_DESCRIPTION="Linux Mint 20 Ulyana"

Some distros write version information to other small text files in the /etc/ directory. For example, in Fedora:

$ cat /etc/fedora-release
Fedora release 33 (Thirty Three)

Debian stores information in the /etc/debian_version file. A search for all files matching /etc/*release or /etc/*version will provide the most common distro and release information files.

Some distros also put version and release information into the /etc/issue or /etc/motd files, which are displayed when a user logs in via the shell or network. For example:

$ cat /etc/issue
Welcome to openSUSE Tumbleweed 20201111 - Kernel 
 (l).

Rolling release distros will often use the date of the last update as the version number.

Unique Machine ID

Modern Linux systems have a unique identifier that’s created during installation. The /etc/machine-id file (may be copied or symlinked with the D-Bus machine ID stored in /var/lib/dbus/machine-id) contains a randomly generated 128-bit hexadecimal string, as shown here:

$ cat /etc/machine-id
8635db7eed514661b9b1f0ad8b249ffd

This unique identification string can be used for matching identical copied/ duplicated machines deployed in multiple places, or for matching a system with full system backups. The creation timestamp of this file is a potential indicator of the installation time. See the machine-id(5) man page for details. Raspberry Pi images initially contain an empty /etc/machine-id file that’s initialized during the first boot.

POSIX-compliant systems also have a hostid that’s typically a hexadecimal representation of the IP address (derived from the /etc/hosts file or a DNS lookup). This ID can be stored in the /etc/hostid file (though most distros don’t have it) and is found on a running system by executing the hostid command or calling gethostid() from a program.

System Hostname

The machine’s hostname is another identifier. This hostname is set in the kernel at boot time or during network reconfiguration. The hostname can be manually specified during installation or dynamically assigned during DHCP network configuration. The system administrator chooses the hostname, which is likely to be unique among the machines under their responsibility or within a DNS domain. However, the hostname is not guaranteed

to be unique in general. The name of the system is typically stored in the /etc/hostname file in a non-FQDN format. Fully qualified domain names (FQDNs) are allowed but not preferred.

If a hostname is specified in /etc/hostname (or another distro-specific location) or returned from a DHCP request, the running kernel is configured accordingly. Hosts with multiple interfaces, multiple IP addresses (each resolving to a different DNS name), or roaming machines (laptops and mobile devices) will still have one hostname representing the whole system. Network configuration involving hostnames, DNS domain names, interfaces, and so on is explained in Chapter 8.

Distro Installer Analysis

Analysis of the initial installation of a Linux system involves identifying the locations of logs and files containing potentially interesting information. An initial Linux installation can be either user-interactive or automated/unattended (enterprise deployment). In both cases, a set of basic configuration parameters are specified to guide the installation process. The typical decision information needed for installing a system is as follows:

• Language, locale, keyboard layout, and time zone
• Drive partitioning, filesystems, and mount points
• Encryption of drives or home directories
• Initial username and password, and root password (unless using sudo)
• Basic system type (choice of desktop, headless server, and so on)
• Basic services (web server, remote access with SSH, printing, and so on)
• Choice of software repositories, non-free software

Automated enterprise installations (such as Red Hat’s Kickstart or SUSE’s AutoYaST, for example) are outside the scope of this book.

When analyzing the installation process, a digital forensic investigator is trying to answer several basic questions:

• When was the system installed?
• What were the initial settings provided during install?
• Is there any useful or interesting information that was saved?
• Was there anything unusual about the installation (or about the repositories)?

Depending on the type of incident or investigation in progress, other more specific questions related to the installation will need answering.

When building timelines, keep in mind that a system installation is not a single point in time, but rather a period with starting and ending timestamps. Depending on the speed of the machine, network connection, and number of installed packages, an installation may take more than a few minutes to complete. If an installation is interactive, and a user is not there to answer the prompted questions, the installation may appear to take hours or more to complete (whenever the user returned to the installation prompt).

Also note that the starting timestamps of an installation may be unreliable. When a computer is booted with the installation media, time has not yet been synchronized and the time zone has not been chosen. The installer might still generate logs, but it will use whatever time the PC or virtual machine (VM) host happened to have (in some obscure cases, this time difference could also be interesting from an investigative perspective). Once the network has been configured, the time zone has been determined, and the clock has been synchronized, the logs will contain more reliable timestamps.

A systemd service called systemd-firstboot is able to provide automated or interactive configuration on the first boot of a system. See the systemd-firstboot(1) man page for more information.

Debian Installer

The initial installation of a Debian system uses Debian Installer.² Debian Installer itself is a Linux system that can be booted from CD/DVD, USB stick, over a network, or from a downloaded image file (for VMs). The documentation defines multiple stages of a Debian installation:

Logs from a completed Debian installation are saved in /var/log/installer/ and provide a snapshot of information from the time the initial installation was made. This snapshot can be interesting. For example, consider this installer log directory from a typical Debian installation:

$ ls -lR /var/log/installer/
/var/log/installer/:
total 1208
drwxr-xr-x 2 root root   4096 Mar 5 02:43 cdebconf
-rw-r--r-- 1 root root  35283 Mar 5 02:43 hardware-summary
-rw-r--r-- 1 root root    160 Mar 5 02:43 lsb-release
-rw------- 1 root root  81362 Mar 5 02:43 partman
-rw-r--r-- 1 root root  72544 Mar 5 02:43 status
-rw------- 1 root root 988956 Mar 5 02:43 syslog
-rw------- 1 root root  43336 Mar 5 02:43 Xorg.0.log

/var/log/installer/cdebconf:
total 14668
-rw------- 1 root root   119844 Mar 5 02:43 questions.dat
-rw------- 1 root root 14896576 Mar 5 02:43 templates.dat

The hardware-summary file provides information about the machine hardware at the time of installation, including a list of devices on the PCI bus and attached USB devices. The lsb-release file contains information about the originally installed release (before any upgrades). The partman file is the output from the drive setup process, and it includes storage devices, partition information, and created filesystems. The status file contains a detailed list of all installed packages (including versions) at the time of installation. The syslog file contains information sent to the standard syslog during the entire installation process (with timestamps). Desktop systems may also have an Xorg.0.log file containing the startup output of the X11 server, which has information about the graphics card, monitors, and attached peripheral input devices. The cdebconf package has files containing the options and choices made during the install process. These files provide insight into the system’s state at the time of installation.

Ubuntu-based systems have a bootable live system (called Casper) with a graphical installer program called Ubiquity. The Debian Installer is used as a backend to Ubiquity and leaves files in /var/log/installer/ but with slightly different contents. Following is an example:

$ ls -l /var/log/installer/
total 1096
-rw------- 1 root root   1529 Mar 5 11:22 casper.log
-rw------- 1 root root 577894 Mar 5 11:22 debug
-rw-r--r-- 1 root root 391427 Mar 5 11:22 initial-status.gz
-rw-r--r-- 1 root root     56 Mar 5 11:22 media-info
-rw------- 1 root root 137711 Mar 5 11:22 syslog

The casper.log and debug files are the output from the installer scripts and contain error messages. The media-info file shows the release information at the time of install. Some Ubuntu-based distros (Mint, for example) may also have a version file. The initial-status.gz file (compressed) contains a list of initially installed packages.

Raspberry Pi Raspian

The Raspberry Pi uses a Debian-based distribution called Raspian. Debian Installer isn’t necessary, because Raspian is available as a preinstalled image file for download. This preinstalled image is available in two formats:

Because there is no “installation” in the usual sense, investigators will want to determine the time when the user first powered on the Pi and saved the initial settings. However, finding this initial setup time is tricky for a number of reasons. The initial filesystem timestamps are from the Raspian image that was downloaded, and not created from local installer scripts. The Raspberry Pi has no hardware clock with battery backup,³ so every time the Pi is powered on, the clock starts with the Unix epoch (00:00 January 1, 1970). The booting operating system sets the clock to the time nearest to the last power-off until network time synchronization is achieved (see Chapter 9 for more details on system time). By default, the filesystem is mounted with the noatime option, so the last-accessed timestamps are not updated. Other timestamps may have been updated and log entries written before the correct time was established, rendering those times unreliable.

When a Raspberry Pi is used for the first time, the filesystem is resized to fit the SD card. After a reboot, the piwiz application starts,⁴ which allows the user to configure a network, reset the password (the default is raspberry), and specify the country, language, and time-zone settings. The piwiz app starts automatically from the file /etc/xdg/autostart/piwiz.desktop, which is deleted after the user provides their initial preferences. If this piwiz.desktop file still exists, it indicates an unused Raspberry Pi installation. If your filesystem forensic analysis tool can determine the time when the file /etc/xdg/ autostart/piwiz.desktop was deleted, that would indicate an approximate time of a completed installation. An alternative is to find the timestamp of the first entry in the /var/log/dpkg.log file (or the oldest saved log rotation). Packages are updated for the first time when piwiz runs, which happens only after time synchronization was successful.

Fedora Anaconda

Fedora-based systems (CentOS, Red Hat, and so on) use an installer called Anaconda.⁵ After the initial desktop installation is complete and the new system reboots for the first time, a separate application called Initial Setup is run. This application can provide additional configuration possibilities, including the user acceptance of end-user licensing agreements (EULAs).

The Anaconda installer leaves logfiles of the initial installation in /var/ log/anaconda/ that look like this:

# ls -l /var/log/anaconda/
total 3928
-rw-------. 1 root root   36679 Mar 24 11:01 anaconda.log
-rw-------. 1 root root    3031 Mar 24 11:01 dbus.log
-rw-------. 1 root root  120343 Mar 24 11:01 dnf.librepo.log
-rw-------. 1 root root     419 Mar 24 11:01 hawkey.log
-rw-------. 1 root root 2549099 Mar 24 11:01 journal.log
-rw-------. 1 root root       0 Mar 24 11:01 ks-script-sot00yjg.log
-rw-------. 1 root root  195487 Mar 24 11:01 lvm.log
-rw-------. 1 root root  327396 Mar 24 11:01 packaging.log
-rw-------. 1 root root    7044 Mar 24 11:01 program.log
-rw-------. 1 root root    2887 Mar 24 11:01 storage.log
-rw-------. 1 root root  738078 Mar 24 11:01 syslog
-rw-------. 1 root root   22142 Mar 24 11:01 X.log

The anaconda.log file tracks the progress of various installation tasks. The X.log file shows the output from the Xorg server used by Anaconda and contains information about the graphics card, monitors, and attached peripheral input devices at the time of installation.

The journal.log and syslog files are very similar, with the main difference being that journal.log shows more dracut activity (see Chapter 6). They both include the initialization of the kernel (dmesg output) and systemd at the time of first installation. These logs can help determine the start and end times of an installation. Information about storage devices, partitioning, and volume management can be found in storage.log and lvm.log. The dnf.librepo.log file lists all of the packages downloaded for installation. The ks-script-*.log files contain log output from kickstart scripts. Other files contain logs with D-Bus activity and library calls. See https://fedoraproject.org/wiki/Anaconda/Logging for more information about Anaconda logging.

These logs provide information about the user-specified configuration, the hardware of the original machine, packages installed, and storage configuration at the time of installation.

SUSE YaST

SUSE Linux has one of the oldest distro installers still maintained today. YaST, or “Yet another Setup Tool,” was designed to combine the initial installation with other system configuration tasks into one tool.⁶ YaST can be used to install the system, set up peripherals like printers, install packages, configure hardware, configure the network, and more. SUSE also provides AutoYaST for unattended enterprise deployment.

The YaST log directory is /var/log/YaST2/. It contains logs from both the installation and other regular configuration tasks. Logs from the installation are stored in the compressed archive file yast-installation-logs.tar.xz, and are of particular interest from a forensics perspective. The following is an example (partial) list of contents:⁷

# tar -tvf yast-installation-logs.tar.xz
-rw-r--r-- root/root     938 2020-03-05 08:35 etc/X11/xorg.conf
drwxr-xr-x root/root       0 2020-02-12 01:14 etc/X11/xorg.conf.d/
-rw-r--r-- root/root     563 2020-03-03 20:30 linuxrc.config
-rw-r--r-- root/root     322 2020-02-26 01:00 etc/os-release
...
-rw-r--r-- root/root   21188 2020-03-05 08:35 Xorg.0.log
-rw-r--r-- root/root   25957 2020-03-05 08:38 linuxrc.log
-rw-r--r-- root/root   17493 2020-03-05 08:34 wickedd.log
-rw-r--r-- root/root   46053 2020-03-05 08:35 boot.msg
-rw-r--r-- root/root  104518 2020-03-05 08:55 messages
-rw-r--r-- root/root    5224 2020-03-05 08:55 dmesg
-rw-r--r-- root/root      17 2020-03-05 08:55 journalctl-dmesg
-rw-r--r-- root/root     738 2020-03-05 08:55 install.inf
-rw------- root/root    3839 2020-03-05 08:55 pbl-target.log
-rw-r--r-- root/root     141 2020-03-05 08:55 rpm-qa
-rw-r--r-- root/root   27563 2020-03-05 08:55 _packages.root

The release information at the time of install is found in the sub-directory etc/os-release. The file Xorg.0.log contains information about the graphics card, monitors, and attached peripheral input devices at the time of installation. The boot.msg, dmesg, and messages files contain logs from the installation, the kernel ring buffer, and other information at the time of install. The wickedd.log file from the network manager records the configuring of the network, including the system’s IP and other network configuration from the time of installation.

The start and end times of the logfile entries from this directory provide an approximate time period of when the installation took place.

Arch Linux

The native Arch Linux system does not have a comfortable installer. Booting the Arch installation media drops the user into a root shell with a reference to the wiki installation guide (earlier versions had an install.txt file containing further instructions). The user is expected to create the partitions and filesystem manually, and then run the pacstrap script that populates the mounted install target directory. After that, the user chroots into the directory and manually completes the installation. The installation process is documented at https://wiki.archlinux.org/index.php/Installation_guide.

A basic installer script called archinstall is included with Arch Linux install media. If used, this script logs the initial configuration settings and activity in /var/log/archinstall/install.log.

The creation (Birth:) timestamp of the root directory (if the filesystem supports it) is a rough indicator of the start of installation:

# stat /
  File: /
  Size: 4096       Blocks: 16     IO Block: 4096  directory
Device: fe01h/65025dInode: 2       Links: 17
Access: (0755/drwxr-xr-x) Uid: (  0/  root)  Gid: (  0/  root)
Access: 2020-03-05 10:00:42.629999954 +0100
Modify: 2020-02-23 10:29:55.000000000 +0100
Change: 2020-03-05 09:59:36.896666639 +0100
 Birth: 2020-03-05 09:58:55.000000000 +0100

Installing Arch is a manual and ongoing process. A user may continue installing and tweaking the system indefinitely, and an install “end” time may not make sense in this context.

The terse and non-intuitive process of installing Arch Linux has spawned several distros for users who want all the benefits of a bleeding-edge rolling distro, but with a comfortable installation. The most popular Arch-based distro is Manjaro.

Manjaro’s installer is called Calamares, and it provides minimal logging of the installation process. These logs are found in /var/log/Calamares.log. The content of Calamares.log includes specified configuration (time zone, locale, and so on), partition information, user information, and more. Calamares (on Manjaro) doesn’t log an IP address, but it does perform a Geo-IP lookup to determine the location of the system being installed:

# grep Geo /var/log/Calamares.log
2020-03-05 - 08:57:31 [6]: GeoIP result for welcome= "CH"
2020-03-05 - 08:57:33 [6]: GeoIP reporting "Europe/Zurich"

Calamares has become well known because of Manjaro, but it was developed with the intention of being a general installer for any distribution. See https://calamares.io/ for more information on Calamares.

Package File Format Analysis

This section covers the file formats of the individual software packages used in common Linux distributions. Linux distribution software packages are single archive files containing all the information and files needed to install and remove them from the Linux system. In addition, Linux systems typically have package management systems that keep track of installed packages, manage dependencies, perform updates, and so on.

Analysis of a software package file can reveal interesting artifacts. Some forensic analysis tasks that can be performed on a package file include:

• Discovering when a package was built
• Verifying package integrity
• Showing package metadata
• Listing package file contents
• Extracting supporting scripts
• Extracting individual files
• Identifying additional timestamps

In addition, a vulnerability assessment may involve matching the version numbers of individual packages with known published vulnerabilities; for example, matching a particular software version installed on a system with a CVE published by Mitre (https://cve.mitre.org/). This is typically the task of an enterprise vulnerability management function within an organization.

Debian Binary Package Format

The Debian binary package format (DEB), is used by Debian and Debian-based distributions. See the deb(5) man page on a Debian or Debian-based system for more information. A DEB file has the *.deb extension and an initial magic string of seven characters (!<arch>). Figure 7-1 on the following page shows the structure of a DEB file.

DEB files use the ar archive format and contain three standard components. In this example, the ed package (a line-oriented text editor) is listed using the GNU ar command:

$ ar -tv ed_1.15-1_amd64.deb
rw-r--r-- 0/0     4 Jan 3 15:07 2019 debian-binary
rw-r--r-- 0/0  1160 Jan 3 15:07 2019 control.tar.xz
rw-r--r-- 0/0 58372 Jan 3 15:07 2019 data.tar.xz

In this example, the flags (-tv) for ar specify a verbose listing of the contents. The file timestamps indicate when the DEB package archive was built.

The three files in the archive have the following contents:

These components can be extracted with ar:

$ ar -xov ed_1.15-1_amd64.deb
x - debian-binary
x - control.tar.xz
x - data.tar.xz

The (-xov) flags instruct ar to extract files, keep original timestamps, and show verbose output. The control.tar.xz and data.tar.xz files are compressed archives that can be further examined.

Figure 7-1: Debian “DEB” package format (modified from Wikipedia: https://upload.wikimedia.org/wikipedia/commons/6/67/Deb_File_Structure.svg)

The debian-binary file contains a single line with the package format version number (2.0). To list the contents of the archives, we rely on tar to decompress the file and list the archive contents:

$ cat debian-binary
2.0
$ tar -tvf control.tar.xz
drwxr-xr-x root/root       0 2019-01-03 15:07 ./
-rw-r--r-- root/root     506 2019-01-03 15:07 ./control
-rw-r--r-- root/root     635 2019-01-03 15:07 ./md5sums
-rwxr-xr-x root/root     287 2019-01-03 15:07 ./postinst
-rwxr-xr-x root/root     102 2019-01-03 15:07 ./prerm
$ tar -tvf data.tar.xz
drwxr-xr-x root/root       0 2019-01-03 15:07 ./
drwxr-xr-x root/root       0 2019-01-03 15:07 ./bin/
-rwxr-xr-x root/root   55424 2019-01-03 15:07 ./bin/ed
-rwxr-xr-x root/root      89 2019-01-03 15:07 ./bin/red
drwxr-xr-x root/root       0 2019-01-03 15:07 ./usr/
drwxr-xr-x root/root       0 2019-01-03 15:07 ./usr/share/
drwxr-xr-x root/root       0 2019-01-03 15:07 ./usr/share/doc/
drwxr-xr-x root/root       0 2019-01-03 15:07 ./usr/share/doc/ed/
-rw-r--r-- root/root     931 2012-04-28 19:56 ./usr/share/doc/ed/AUTHORS
-rw-r--r-- root/root     576 2019-01-01 19:04 ./usr/share/doc/ed/NEWS.gz
-rw-r--r-- root/root    2473 2019-01-01 18:57 ./usr/share/doc/ed/README.gz
-rw-r--r-- root/root     296 2016-04-05 20:28 ./usr/share/doc/ed/TODO
...

If we want to extract a particular file from the *.tar.xz archives, we can use the same command, but give tar specific instructions to extract the file:

$ tar xvf control.tar.xz ./control
./control
$ cat ./control
Package: ed
Version: 1.15-1
Architecture: amd64
Maintainer: Martin Zobel-Helas <[email protected]>
Installed-Size: 111
Depends: libc6 (>= 2.14)
Section: editors
Priority: optional
Multi-Arch: foreign
Homepage: https://www.gnu.org/software/ed/
Description: classic UNIX line editor
 ed is a line-oriented text editor. It is used to
...

The contents of the extracted control file list the version, CPU architecture, maintainer, dependencies, and other information. The control file is mandatory and the other files within the control.tar.xz component are optional. Other common package control files include pre-install, post-install, pre-remove, and post-remove scripts (preinst, postinst, prerm, and postrm, respectively). See the deb-control(5) man page for more information about the control file.

We can extract files and directories from the data archive the same way. However, doing so will extract a full directory tree to the current working directory with the file(s) specified. It is also possible to extract individual files to stdout, allowing redirection to a file or program. In this example, a single file is extracted to stdout using the -xOf flags (O is an uppercase O, not zero):

$ tar -xOf data.tar.xz ./usr/share/doc/ed/AUTHORS
Since 2006 GNU ed is maintained by Antonio Diaz Diaz.

Before version 0.3, GNU ed and its man page were written and maintained
(sic) by Andrew L. Moore.

The original info page and GNUification of the code were graciously
provided by FranÃğois Pinard.
...

Individual files can be saved using file redirection, or the entire archive can be unpacked to a local analysis directory.

Although not mandatory, it is normal practice for DEB packages to contain a list of MD5 hashes to verify file integrity. These are stored in the md5sums file in the control component of the package archive file. This example displays the list of expected MD5 hashes in the package, followed by the verification of an installed binary file:

$ tar -xOf control.tar.xz ./md5sums
9a579bb0264c556fcfe65bda637d074c bin/ed
7ee1c42c8afd7a5fb6cccc6fa45c08de bin/red
318f005942f4d9ec2f19baa878f5bd14 usr/share/doc/ed/AUTHORS
ad0755fb50d4c9d4bc23ed6ac28c3419 usr/share/doc/ed/NEWS.gz
f45587004171c32898b11f8bc96ead3c usr/share/doc/ed/README.gz
3eef2fe85f82fbdb3cda1ee7ff9a2911 usr/share/doc/ed/TODO
...
$ md5sum /bin/ed
9a579bb0264c556fcfe65bda637d074c /bin/ed

The md5sum tool has a flag (-c) that reads a list of MD5s from files like md5sums and performs checks on all files listed. There has been discussion of replacing the md5sums file with SHA hashes (for more information, see https://wiki.debian.org/Sha256sumsInPackages).

On a Debian system, the dpkg-deb tool performs all the above analysis tasks of listing files, extracting files, viewing control data, and so on. If you are trying to recover data from a corrupted DEB file, ar -tO (O is uppercase O, not zero) will provide hexadecimal offsets to the three components, which may allow extraction with tools such as dd.

Red Hat Package Manager

The Red Hat Package Manager (RPM) is a binary package format developed by Red Hat. RPM packages can be identified by an .rpm extension and a four-byte magic string (ED AB EE DB) at the start of the file. The structure of RPM package files is documented in the rpm tool’s source code, and the file /doc/manual/format describes four logical sections:

The rpm command, which can also be installed on non–Red Hat distros, can be used on a separate analysis machine. The query flag (-q) can be used to analyze various aspects of RPM files. In this example, the -q and -i flags provide an informational overview of the xwrits RPM package file:

$ rpm -q -i xwrits-2.26-17.fc32.x86_64.rpm
Name        : xwrits
Version     : 2.26
Release     : 17.fc32
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 183412
License     : GPLv2
Signature   : RSA/SHA256, Sat 01 Feb 2020 01:17:59 AM, Key ID 6c13026d12c944d0
Source RPM  : xwrits-2.26-17.fc32.src.rpm
Build Date  : Fri 31 Jan 2020 09:43:09 AM
Build Host  : buildvm-04.phx2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.lcdf.org/xwrits/
Bug URL     : https://bugz.fedoraproject.org/xwrits
Summary     : Reminds you take wrist breaks
Description :
Xwrits reminds you to take wrist breaks, which
should help you prevent or manage a repetitive
stress injury. It pops up an X window when you
...

You can view other RPM metadata with the following flags (after rpm -q) together with the RPM filename:

This list was taken from the rpm(9) man page, where you can find further information about rpm files. If a flag returns no output, that header field is empty.

Extracting individual files from RPM packages is a two-step process. First, the payload is extracted from the RPM, and then the desired file is extracted from that payload. The rpm2cpio and rpm2archive tools create a cpio or compressed tar (*.tgz) archive file containing the payload of the RPM. These are files that most file managers and forensic tools should be able to browse for file exporting/extracting.

In the following example, an individual file is extracted from an RPM. First, the RPM payload is extracted, and then an individual file is identified and extracted:

$ rpm2cpio xwrits-2.26-17.fc32.x86_64.rpm > xwrits-2.26-17.fc32.x86_64.rpm.cpio
$ cpio -i -tv < xwrits-2.26-17.fc32.x86_64.rpm.cpio
...
-rw-r--r--  1 root   root     1557 Oct 16 2008 ./usr/share/doc/xwrits/README
...
$ cpio -i --to-stdout ./usr/share/doc/xwrits/README < xwrits-2.26-17.fc32.x86_64.rpm.cpio
XWRITS VERSION 2.25
===================
ABOUT XWRITS
------------
  Xwrits was written when my wrists really hurt. They don't any more --
...

The rpm2cpio command is run and the output is redirected to a file (it can be any name, but for clarity, I used same filename with a *.cpio extension). The next command lists the cpio archive to find the desired file for extraction. The final command extracts the file to stdout where it can be piped or redirected to a program or file.

RPM package headers contain cryptographic signatures and hashes for verifying the payload’s integrity. Integrity checking is done with the rpmkeys⁸ command and can be viewed (verbose) with the -Kv flags:

$ rpmkeys -Kv xwrits-2.26-17.fc32.x86_64.rpm
xwrits-2.26-17.fc32.x86_64.rpm:
    Header V3 RSA/SHA256 Signature, key ID 12c944d0: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID 12c944d0: OK
    MD5 digest: OK

The GPG keys for signed RPM packages can be imported using the rpmkeys command. See the rpmkeys(8) man page for more information.

Arch Pacman Packages

Packages for Arch Linux are compressed tar files. The default compression is currently in transition from XZ to Zstandard, with file extensions *.xz and *.zst, respectively.⁹ The tar file contains both the package metadata and the files to be installed.

We can use tar to view the contents of a pacman package:

$ tar -tvf acpi-1.7-2-x86_64.pkg.tar.xz
-rw-r--r-- root/root     376 2017-08-15 19:06 .PKGINFO
-rw-r--r-- root/root    3239 2017-08-15 19:06 .BUILDINFO
-rw-r--r-- root/root     501 2017-08-15 19:06 .MTREE
drwxr-xr-x root/root       0 2017-08-15 19:06 usr/
drwxr-xr-x root/root       0 2017-08-15 19:06 usr/share/
drwxr-xr-x root/root       0 2017-08-15 19:06 usr/bin/
-rwxr-xr-x root/root   23560 2017-08-15 19:06 usr/bin/acpi
drwxr-xr-x root/root       0 2017-08-15 19:06 usr/share/man/
drwxr-xr-x root/root       0 2017-08-15 19:06 usr/share/man/man1/
-rw-r--r-- root/root     729 2017-08-15 19:06 usr/share/man/man1/acpi.1.

This example shows the simplicity of the package format. Several files in the root of the archive contain the package metadata. They are described in the Arch Linux Wiki (https://wiki.archlinux.org/index.php/Creating_packages) and include:

The .PKGINFO file is regular text and can be easily viewed, but using the pacman tool provides more complete output (including fields that are undefined). The -Qip flags specify a query operation, information option, and a package filename for a target, respectively:

$ pacman -Qip acpi-1.7-2-x86_64.pkg.tar.xz
Name            : acpi
Version         : 1.7-2
Description     : Client for battery, power, and thermal readings
Architecture    : x86_64
URL             : https://sourceforge.net/projects/acpiclient/files/acpiclient/
Licenses        : GPL2
Groups          : None
Provides        : None
Depends On      : glibc
Optional Deps   : None
Conflicts With  : None
Replaces        : None
Compressed Size : 10.47 KiB
Installed Size  : 24.00 KiB
Packager        : Alexander RÃÿdseth <[email protected]>
Build Date      : Di 15 Aug 2017 19:06:50
Install Script  : No
Validated By    : None
Signatures      : None

The .MTREE file is a compressed list of timestamps, permissions, file sizes, and cryptographic hashes. We can extract it by piping the tar output into zcat:

$ tar -xOf acpi-1.7-2-x86_64.pkg.tar.xz .MTREE | zcat
#mtree
/set type=file uid=0 gid=0 mode=644
./.BUILDINFO time=1502816810.765987104 size=3239 md5digest=0fef5fa26593908cb0958537839f35d6
sha256digest=75eea1aee4d7f2698d662f226596a3ccf76e4958b57e8f1b7855f2eb7ca50ed5
./.PKGINFO time=1502816810.745986656 size=376 md5digest=c6f84aeb0bf74bb8a1ab6d0aa174cb13
sha256digest=83b005eb477b91912c0b782808cc0e87c27667e037766878651b39f49d56a797
/set mode=755
./usr time=1502816810.602650109 type=dir
./usr/bin time=1502816810.685985311 type=dir
./usr/bin/acpi time=1502816810.682651903 size=23560 md5digest=4ca57bd3b66a9afd517f49e13f19688f
sha256digest=c404597dc8498f3ff0c1cc026d76f7a3fe71ea729893916effdd59dd802b5181
./usr/share time=1502816810.592649885 type=dir
./usr/share/man time=1502816810.592649885 type=dir
./usr/share/man/man1 time=1502816810.699318943 type=dir
./usr/share/man/man1/acpi.1.gz time=1502816810.609316926 mode=644 size=729
md5digest=fb0da454221383771a9396afad250a44
sha256digest=952b21b357d7d881f15942e300e24825cb3530b2262640f43e13fba5a6750592

This can be used to verify the integrity of the files in the package and provides timestamps for timeline reconstruction. We can use this information to analyze packages that are rogue, malicious, or have been tampered with.

Package Management System Analysis

The previous section focused on the file formats of individual software packages before they are installed. Here we shift the focus to the package management systems for software already installed (or previously installed) on a machine. This includes analysis of the repositories from where packages were downloaded, where the package contents were placed on the filesystem, databases to track the installed packages, installation logs, and more.

A Linux distribution’s software packaging system typically has the following components:

• Repositories to download compiled binary packages
• Repositories to download package source code
• Repositories with non-free or varying licenses
• Information to resolve dependencies and conflicts
• A database with a record of installed software
• Logfiles of package management activity (including uninstalls)
• Frontend user interfaces interacting with backend tools and libraries

Package management systems across Linux distributions are very similar. See https://wiki.archlinux.org/index.php/Pacman/Rosetta for a comparison of package management commands.

From a forensics perspective, we can ask many questions related to package management, such as the following:

• What packages are currently installed, and which versions?
• Who installed them, when, and how?
• Which packages were upgraded and when?
• Which packages were removed and when?
• Which repositories were used?
• Can we confirm the integrity of the packages?
• What logs, databases, and cached data can be analyzed?
• Given a particular file on the filesystem, to which package does it belong?
• What other timestamps are relevant?

Answering these questions will help reconstruct past activity, build timelines, and identify possible malicious or suspicious activity. Finding and validating cryptographic hashes can also be useful when using NSRL hashsets to exclude known software. Packages that have been removed may leave behind traces of custom or modified configuration files and data that was not deleted.

The next few sections describe the analysis of the most common distributions. Each section provides an introduction to the packaging system and describes the various files, databases, and directory locations that are of interest to a forensic examiner.

Debian apt

The Debian package management system is a collection of programs that manage package searching/selection, external repositories, downloads, dependency/conflict resolution, installation, removal, updates and upgrades, and other package housekeeping functions. The end user interacts with high-level programs like Apt, Aptitude, Synaptic, and others to choose which packages to install, remove, or upgrade. These high-level programs interact with the dpkg command,¹⁰ which manages the installation, removal, and querying of packages on a Debian-based system. Forensic investigators are mainly interested in the current package state of a system, reconstructing past package activity, and identifying other interesting artifacts.

The current installed package state of a Debian-based system is stored in the /var/lib/dpkg/status file (the package “database”). This is a plaintext file with each package entry starting with the string Package: and ending with a blank line (similar style to the email mbox format). Backup copies of this file are in the same directory, and may be named status-old or /var/backups/ dpkg.status.* (multiple copies of previous versions may also be available in compressed form).

The status file can be easily viewed and searched with any text editor or text-processing tool. In this example, the awk¹¹ tool is used to search the status file for a package name (Package: bc) and print the entire block of information:

$ awk ' /^Package: bc$/ , /^$/ ' /var/lib/dpkg/status
Package: bc
Status: install ok installed
Priority: standard
Section: math
Installed-Size: 233
Maintainer: Ryan Kavanagh <[email protected]>
Architecture: amd64
Multi-Arch: foreign
Source: bc (1.07.1-2)
Version: 1.07.1-2+b1
Depends: libc6 (>= 2.14), libncurses6 (>= 6), libreadline7 (>= 6.0), libtinfo6 (>= 6)
Description: GNU bc arbitrary precision calculator language
 GNU bc is an interactive algebraic language with arbitrary precision which
 follows the POSIX 1003.2 draft standard, with several extensions including
 multi-character variable names, an `else' statement and full Boolean
 expressions. GNU bc does not require the separate GNU dc program.
Homepage: http://ftp.gnu.org/gnu/bc/

The Status: line is interesting from a forensic reconstruction perspective. A normal installed package file will have Status: install ok installed. Packages that have been removed but still have user-modified configuration files are listed with a status of Status: deinstall ok config-files. Some packages may have a Conffiles: line followed by several lines indicating configuration files an administrator might modify, and the MD5 hash of the originally installed version of the file. For example, the default configuration files of the Apache web server are shown here:

Package: apache2
Status: install ok installed
...
Conffiles:
 /etc/apache2/apache2.conf 20589b50379161ebc8cb35f761af2646
...
 /etc/apache2/ports.conf a961f23471d985c2b819b652b7f64321
 /etc/apache2/sites-available/000-default.conf f3066f67070ab9b1ad9bab81ca05330a
 /etc/apache2/sites-available/default-ssl.conf 801f4c746a88b4228596cb260a4220c4
 ...

The MD5 hashes can help identify configuration files that deviate from the package defaults. See the dpkg-query(1) man page for more information about the fields in the status file.

The status file does not contain installation timestamps. For installation dates, you must analyze the logfiles. Several logfiles record the activity of the package management system and the frontend package manager tools. Common package management logs found on Debian-based systems include the following:

Rotated logs may be compressed and renamed to filenames with a number indicating the relative age of the logfile (dpkg.log.1.gz, for example). The larger the number, the older the log.

Configuration information for dpkg is stored in the /etc/dpkg/ directory. Configuration information for apt is stored in the /etc/apt/ directory. The /etc/apt/ directory contains the sources.list and sources.list.d/* files. These files are interesting because they define the configured external repositories for a particular Debian release. Explicitly added (legitimate or rogue) repositories will be appended to this file or saved to a file in the sources.list.d/ directory. Ubuntu also has Personal Package Archives (PPAs) that use its central Launchpad server to help users add sources for individual packages.

The /var/lib/dpkg/info/ directory contains several files for each installed package (this is the metadata from the DEB files). This information includes the file list (*.list), cryptographic hashes (*.md5sums), preinstall/postinstall and remove scripts, and more. The *.conffiles (if they exist) are a potentially useful resource for forensic investigators, as they list the location of configuration files and are often modified by the system owner.

The /var/cache/apt/archives/ directory contains *.deb files that have been downloaded in the past. The /var/cache/debconf/ directory is a central location for package configuration information and templates. Of potential interest here is the passwords.dat file that contains system-generated passwords needed for local daemons.

See the dpkg(1) and apt(8) man pages, as well as the Debian manual (https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_the_dpkg_command) for more information.

Fedora dnf

Fedora-based systems manage packages using dnf (Dandified Yum), the successor to yum (Yellow Dog Update Manager). The dnf tool is written in Python and uses the librpm library to manage the installed rpm packages.

The current installed package state is stored in a collection of Berkeley database files in the /var/lib/rpm/ directory. The easiest way to analyze this is to use the rpm command on a separate analysis machine¹² with the --dbpath flag pointing to a read-only copy of the database files. For example, to list the installed packages in a collection of database files stored in a separate directory, use the --dbpath and -qa flags:

$ rpm --dbpath=/evidence/ -qa
...
rootfiles-8.1-25.fc31.noarch
evince-libs-3.34.2-1.fc31.x86_64
python3-3.7.6-2.fc31.x86_64
perl-Errno-1.30-450.fc31.x86_64
OpenEXR-libs-2.3.0-4.fc31.x86_64
man-pages-de-1.22-6.fc31.noarch
...

To see the metadata for a specific installed package, use the --dbpath and -qai flags with the package name. Several examples using the Evince document viewer package are shown here:

$ rpm --dbpath=/evidence/ -qai evince
Name        : evince
Version     : 3.34.2
Release     : 1.fc31
Architecture: x86_64
Install Date: Tue Mar 3 06:21:23 2020
Group       : Unspecified
Size        : 9978355
License     : GPLv2+ and GPLv3+ and LGPLv2+ and MIT and Afmparse
Signature   : RSA/SHA256, Wed Nov 27 16:13:20 2019, Key ID 50cb390b3c3359c4
Source RPM  : evince-3.34.2-1.fc31.src.rpm
Build Date  : Wed Nov 27 16:00:47 2019
Build Host  : buildhw-02.phx2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://wiki.gnome.org/Apps/Evince
Bug URL     : https://bugz.fedoraproject.org/evince
Summary     : Document viewer
Description :
Evince is simple multi-page document viewer. It can display and print
...

To see a list of files belonging to a package, use the --dbpath and -ql flags (lowercase letter L, as in “list”) flags:

$ rpm --dbpath /evidence/ -ql evince
/usr/bin/evince
/usr/bin/evince-previewer
/usr/bin/evince-thumbnailer
/usr/lib/.build-id
/usr/lib/.build-id/21
/usr/lib/.build-id/21/15823d155d8af74a2595fa9323de1ee2cf10b8
...

To determine which package a file belongs to, use the --dbpath and -qf flags with the full path and filename:

$ rpm --dbpath /evidence/ -qf /usr/bin/evince
evince-3.34.2-1.fc31.x86_64

All of these commands can be used with read-only offline copies of the RPM database files found in the /var/lib/rpm/ directory of the Linux image under analysis. Be aware that running the rpm command on your forensic workstation will use the local RPM configuration (for example, /usr/lib/rpm/ rpmrc), but that shouldn’t affect the accuracy of the output shown in the examples above.

The RPM database files traditionally have been standard Berkeley DB files and could be analyzed individually with tools like db_dump. Fedora 33 transitioned to SQLite for the RPM database, and associated tools can be used to examine package data. In addition, the /var/lib/dnf/ directory contains SQLite databases with dnf package information, allowing analysis with SQLite tools.

The dnf command generates multiple logs, which are stored in the /var/ log/ directory and listed here:

• /var/log/dnf.librepo.log
• /var/log/dnf.log
• /var/log/dnf.rpm.log
• /var/log/dnf.librepo.log
• /var/log/hawkey.log

Some of these are less interesting from a forensics perspective and may show only that a machine was online at a particular time (checking for updates, and so on).

The dnf.log (or rotated versions) contain activity performed using the dnf command. Here’s an example:

2020-08-03T19:56:04Z DEBUG DNF version: 4.2.23
2020-08-03T19:56:04Z DDEBUG Command: dnf install -y openssh-server
2020-08-03T19:56:04Z DDEBUG Installroot: /
2020-08-03T19:56:04Z DDEBUG Releasever: 32

Here, the dnf install command was used to install openssh-server at a particular time.

The configuration data for dnf is potentially found in several locations:

See the dnf.conf(5) man page for more information about dnf configuration.

SUSE zypper

SUSE Linux originally had its own package manager tightly integrated with its YaST configuration tool. SUSE later switched to using RPM for the package format and developed the ZYpp package manager. The primary tool for interfacing with the ZYpp library (libzypp) is zypper. The configuration information is in the /etc/zypp/zypper.conf and /etc/zypp/zypp.conf files, which control the zypper tool and ZYpp library, respectively. The configuration files specify various parameters, including the locations of files and directories. See the zypper(8) man page for more information.

The ZYpp library calls the rpm tool to perform the low-level installation and removal tasks. Because the packages are standard RPMs, the installed package state can be analyzed in the same way as Fedora-based systems. The /var/lib/rpm/ directory contains the installed package databases, as described in the previous section.

ZYpp has several detailed logs of package management activity. The /var/log/zypp/history log records the actions of the ZYpp library, which multiple frontend tools might use. The following example shows logs for the installation and removal of the cowsay package:

# cat /var/log/zypp/history
...
2020-04-11 12:38:20|command|root@pc1|'zypper' 'install' 'cowsay'|
2020-04-11 12:38:20|install|cowsay|3.03-5.2|noarch|root@pc1|download.opensuse.
org-oss| a28b7b36a4e2944679e550c57b000bf06078ede8fccf8dfbd92a821879ef8b80|
2020-04-11 12:42:52|command|root@pc1|'zypper' 'remove' 'cowsay'|
2020-04-11 12:42:52|remove |cowsay|3.03-5.2|noarch|root@pc1|
...

The log contains basic libzypp actions, including package install/remove, repository add/remove, repository changes, and the commands used.

The /var/log/zypper.log file shows detailed activity of the zypper command line tool, and the /var/log/pk_backend_zypp has logs of PackageKit activity. Both of these logs contain a field with the local system’s hostname. This could be interesting from a forensics perspective if the hostname is dynamically generated from DHCP because it indicates the hostname during the time the tools were run. If the hostname is an FQDN, it could have a valid domain name resolving to an IP address.

The SUSE zypper-log tool can print formatted output of a zypper.log file:

$ zypper-log -l zypper.log
===============================================================================
Collect from zypper.log ...

TIME                PID      VER   CMD
2020-08-03 09:08   1039  1.14.37   /usr/bin/zypper appstream-cache
2020-08-03 09:08   1074  1.14.37   /usr/bin/zypper -n purge-kernels
2020-08-03 09:08   1128  1.14.37   zypper -n lr
2020-11-12 20:52  29972  1.14.37   zypper search hex
2020-11-12 20:52  30002  1.14.37   zypper search kcrash
2020-11-12 20:52  30048  1.14.37   zypper search dr.conqi
2020-11-13 09:21   2475  1.14.37   zypper updaet
2020-11-13 09:21   2485  1.14.37   zypper -q subcommand
2020-11-13 09:21   2486  1.14.37   zypper -q -h
2020-11-13 09:21   2489  1.14.37   /usr/bin/zypper -q help
2020-11-13 09:21   2492  1.14.37   zypper update
2020-11-13 09:22   2536  1.14.37   zypper dup
2020-11-13 10:02    671  1.14.40   /usr/bin/zypper -n purge-kernels

This output is similar to shell history in that all zypper commands entered are shown, including misspelled or failed attempts. The -l (lowercase letter L) flag specifies the name of the logfile to use if the log has been copied to an analysis machine.

The configuration of repositories is stored in definition files in the /etc/ repos.d/ and /etc/services.d/ directories. Service definition files manage the repositories and contain the lrf_dat variable, which is a timestamp (in Unix epoch format) indicating the date of last refresh. Information about remote package repositories (metadata) is cached locally in the /var/cache/zypp/* directories.

Some SUSE installations are configured to save bug report information whenever a distribution upgrade (zypper dist-upgrade) is run. This will create a directory in /var/log/updateTestcase-*, where * is a date and time. The directory will contain compressed XML files of available repository packages and installed packages (such as solver-system.xml.gz).

The zypper tool can also be run as an interactive shell (zypper shell), in which case, histories of commands are stored in the ~/.zypper_history file of the user who ran them.

The /var/lib/zypp/ directory also contains persistent information about the installed system. A unique identifier is generated during installation and used for statistics every time files are downloaded from SUSE. The file AnonymousUniqueId contains the string, as shown here:

# cat /var/lib/zypp/AnonymousUniqueId
61d1c49b-2bee-4ff0-bc8b-1ba51f5f9ab2

This string is embedded in the HTTP user-agent (X-ZYpp-AnonymousId:) and sent to SUSE’s servers when files are requested.

Arch pacman

Arch Linux uses the pacman command line tool for downloading and managing packages. The configuration file /etc/pacman.conf is used to control how pacman and the associated libalpm library are used. Packages are fetched from remote mirror sites, which are configured in /etc/pacman.d/mirrorlist and used in the order listed.

Arch Linux systems typically install packages from one of four sources:

The first three sources are official Arch repositories with compiled binary packages. The list of available packages in the official repositories are synchronized with files in the /var/lib/pacman/sync/ directory. These files are simply zipped tar archives (with a different filename extension) and can be extracted with regular tools:

$ file /var/lib/pacman/sync/* 
/var/lib/pacman/sync/community.db: gzip compressed data, last modified:
Mon Apr 6 07:38:29 2020, from Unix, original size modulo 2^32 18120192
/var/lib/pacman/sync/core.db:   gzip compressed data, last modified:
Sun Apr 5 19:10:08 2020, from Unix, original size modulo 2^32 530944
/var/lib/pacman/sync/extra.db:   gzip compressed data, last modified:
Mon Apr 6 07:43:58 2020, from Unix, original size modulo 2^32 6829568
...
$ tar tvf /var/lib/pacman/sync/core.db
drwxr-xr-x lfleischer/users 0 2019-11-13 00:49 acl-2.2.53-2/
-rw-r--r-- lfleischer/users 979 2019-11-13 00:49 acl-2.2.53-2/desc
drwxr-xr-x lfleischer/users  0 2020-04-04 07:11 amd-ucode-20200316.8eb0b28-1/
-rw-r--r-- lfleischer/users 972 2020-04-04 07:11 amd-ucode-20200316.8eb0b28-1/desc
drwxr-xr-x lfleischer/users  0 2020-01-09 08:14 archlinux-keyring-20200108-1/
-rw-r--r-- lfleischer/users 899 2020-01-09 08:14 archlinux-keyring-20200108-1/desc
...

The timestamps indicate when the repository package lists and individual packages were last updated.

The integrity of signed¹³ packages and databases is verified using GnuPG and described in the pacman(8) man page. The GPG keys used to verify signatures are stored in the /etc/pacman.d/gnupg/ directory.

The default location of installed package metadata is the /var/lib/pacman/ local/ directory. A separate directory for every installed package exists on the system and contains these files:

These correspond to the files listed earlier when describing the Arch Linux package format.

The mtree file contains the package’s filenames, timestamps, cryptographic hashes, and permissions needed to install the package. See the mtree(5) man page for more information about the format. The contents of mtree are gzip-compressed (but without a filename extension) and can be viewed with zless or zcat. In this example, the mtree file from the sfsimage¹⁴ package is analyzed:

$ zcat /var/lib/pacman/local/sfsimage-1.0-1/mtree
#mtree
/set type=file uid=0 gid=0 mode=644
./.BUILDINFO time=1586180739.0 size=58974 md5digest=352b893f2396fc6454c78253d5a3be5a
sha256digest=681193c404391246a96003d4372c248df6a977a05127bc64d49e1610fbea1c72
./.PKGINFO time=1586180739.0 size=422 md5digest=32a5ef1a7eab5b1f41def6ac57829a55
sha256digest=3dd26a5ca710e70e7c9b7c5b13043d6d3b8e90f17a89005c7871313d5e49a426
...
./usr/bin/sfsimage time=1586180739.0 size=10168
md5digest=e3dcfcb6d3ab39c64d733d8fa61c3097
sha256digest=1c19cc2697e214cabed75bd49e3781667d4abb120fd231f9bdbbf0fa2748c4a3
...
./usr/share/man/man1/sfsimage.1.gz time=1586180739.0 mode=644 size=1641
md5digest=2d868b34b38a3b46ad8cac6fba20a323
sha256digest=cb8f7d824f7e30063695725c897adde71938489d5e84e0aa2db93b8945aea4c1

When a package is removed, the installed files are deleted together with this package metadata directory.

The history of package installation, updates, and removal is logged in the /var/log/pacman.log file. The following example shows a package being installed and then removed:

$ cat /var/log/pacman.log
[2020-04-06T16:17:16+0200] [PACMAN] Running 'pacman -S tcpdump'
[2020-04-06T16:17:18+0200] [ALPM] transaction started
[2020-04-06T16:17:18+0200] [ALPM] installed tcpdump (4.9.3-1)
[2020-04-06T16:17:18+0200] [ALPM] transaction completed
...
[2020-04-06T16:18:01+0200] [PACMAN] Running 'pacman -R tcpdump'
[2020-04-06T16:18:02+0200] [ALPM] transaction started
[2020-04-06T16:18:02+0200] [ALPM] removed tcpdump (4.9.3-1)
[2020-04-06T16:18:02+0200] [ALPM] transaction completed
...

In the logs, PACMAN refers to pacman commands executed by the user, and ALPM refers to libalpm library activity (which includes installing dependencies).

Packages downloaded from the various repositories are cached in the /var/cache/pacman/pkg/ directory. This can be interesting from a forensics perspective because the directory contains previous versions of updated package files and does not delete removed package files. The filesystem timestamps will indicate when a package was downloaded for installation or update.

Packages in the AUR that are not part of the Arch community repository require several manual steps to install. This process is typically automated using AUR helper scripts (two examples of popular AUR helpers are yay and pacaur). These programs download the PKGBUILD and source files, unpack and compile source code, create and install a package, and then clean up any temporary files. These helper scripts may leave files and data in the user’s ~/.cache/ directory with filesystem timestamps from when the package was built. Many AUR helper programs are available, and each one might have its own configuration and save log information. See https://wiki.archlinux.org/index.php/AUR_helpers for a list of AUR helpers.

Universal Software Package Analysis

Some software installation and packaging systems bypass the standard mechanisms of Linux distributions. These are sometimes called universal software packages or universal package systems if they were designed to function independently of the chosen Linux distribution (or version of some particular distribution).

Some software packaging systems also are designed to function across non-Linux operating systems or enterprise container platforms (Docker, for example). This section focuses primarily on Linux-specific local packaging systems.

AppImage

AppImage was designed to provide compatible binaries that would work across multiple Linux distributions and versions by creating a self-contained portable file format. The most popular use of AppImage is to have the latest versions of desktop apps running on stable Linux distributions that have older app versions in their native package repository. AppImage can also be used to run old versions of software. The example presented later in this section will analyze a working AppImage of the NCSA Mosaic browser from the mid-1990s.

The AppImage format bundles all the needed binaries, libraries, and supporting files into a single executable file. Any user can download an AppImage file, give it execute permissions, and then run it. No further installation or root privileges are necessary. An AppImage binary embeds a squashfs filesystem where the directory structure of files is stored. When the binary is run, this squashfs filesystem is mounted (via FUSE), and execution is passed to an internal program called AppRun. AppImage binaries are not running in an isolated sandbox and have access to the rest of the filesystem. The user’s home directory may have configs, cache, and other files related to the AppImage program.

Every AppImage executable includes flags for file extraction, squashfs mounting, and more. The most interesting flag from a forensics perspective is --appimage-offset, which provides the byte offset of the embedded squashfs filesystem. This offset allows us to access the filesystem with the unsquashfs command to extract detailed information and files (including preserved timestamps). The problem with this flag is that we must execute the binary, which is a security risk (especially when analyzing suspicious or malicious files). To avoid this risk, the offset can be independently calculated using the readelf command.

The readelf tool provides information about the executable header with the -h flag:

$ readelf -h NCSA_Mosaic-git.6f488cb-x86_64.AppImage
ELF Header:
  Magic:  7f 45 4c 46 02 01 01 00 41 49 02 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       65
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x401fe4
  Start of program headers:          64 (bytes into file)
  Start of section headers:          110904 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 30

The squashfs filesystem starts after the section headers. This offset is easily calculated from the section header lines:

Start of section headers:      110904 (bytes into file)
Size of section headers:       64 (bytes)
Number of section headers:     31

The byte offset is calculated from the Start + (Size * Number) of the section headers, or in our example:

110904 + ( 64 * 31 ) = 112888

This byte offset number (112888) can be used with unsquashfs to extract information and files.

In the following unsquashfs example, the -o specifies the offset within the AppImage file, and the -s displays information about the filesystem (including a timestamp):

$ unsquashfs -s -o 112888 NCSA_Mosaic-git.6f488cb-x86_64.AppImage
Found a valid SQUASHFS 4:0 superblock on NCSA_Mosaic-git.6f488cb-x86_64.AppImage.
Creation or last append time Tue Apr 18 23:54:38 2017
Filesystem size 3022295 bytes (2951.46 Kbytes / 2.88 Mbytes)
Compression gzip
Block size 131072
...

We can use the offset and -ll flag (two lowercase Ls) for a more detailed file listing:

$ unsquashfs -ll -o 112888 NCSA_Mosaic-git.6f488cb-x86_64.AppImage
Parallel unsquashfs: Using 4 processors
19 inodes (75 blocks) to write

drwxrwxr-x root/root           96 2017-04-18 23:54 squashfs-root
-rw-rw-r-- root/root          653 2017-04-18 23:54 squashfs-root/.DirIcon
lrwxrwxrwx root/root           14 2017-04-18 23:54 squashfs-root/AppRun -> usr/bin/Mosaic
-rw-rw-r-- root/root          149 2017-04-18 23:54 squashfs-root/mosaic.desktop
-rw-rw-r-- root/root          653 2017-04-18 23:54 squashfs-root/mosaic.png
drwxrwxr-x root/root           50 2017-04-18 23:54 squashfs-root/usr
drwxrwxr-x root/root           29 2017-04-18 23:54 squashfs-root/usr/bin
-rwxrwxr-x root/root      2902747 2017-04-18 23:54 squashfs-root/usr/bin/Mosaic
...

The entire filesystem tree can be extracted, or we can extract individual files. In this example, a single file is extracted (unsquashfs will create the squashfs-root directory if it doesn’t exist):

$ unsquashfs -o 112888 NCSA_Mosaic-git.6f488cb-x86_64.AppImage mosaic.desktop
...
created 1 files
created 1 directories
created 0 symlinks
created 0 devices
created 0 fifos
$ ls -l squashfs-root/
total 4
-rw-r----- 1 sam sam 149 18. Apr 2017 mosaic.desktop

The byte offset can also be used to mount the embedded filesystem on your forensic analysis machine, where it can be browsed with other programs:

$ sudo mount -o offset=112888 NCSA_Mosaic-git.6f488cb-x86_64.AppImage /mnt
...
$ ls -l /mnt
total 2
lrwxrwxrwx 1 root root 14 18. Apr 2017 AppRun -> usr/bin/Mosaic
-rw-rw-r-- 1 root root 149 18. Apr 2017 mosaic.desktop
-rw-rw-r-- 1 root root 653 18. Apr 2017 mosaic.png
drwxrwxr-x 5 root root 50 18. Apr 2017 usr/

Because this is squashfs, it’s read-only, so there is no danger of accidentally modifying the mounted directory contents.

AppImage files can be found anywhere a user has write permission. Because they are normal ELF executables, they have the same magic string and other properties as other executables. The *.AppImage filename extension might be the only indicator of the file type. The filesystem timestamps (Birth and Modify) of the AppImage file may indicate when the file was downloaded, and the timestamps inside squashfs indicate when the AppImage file was built.

Flatpak

Flatpak (renamed from xdg-app) is designed for Linux distro-independent packaging and distribution of desktop apps. Flatpak uses repositories to transfer and update files using the OSTree system. OSTree is similar to Git, but it tracks binary files rather than source code. The apps are run in containers with explicit permissions to access local system resources.

Flatpak has several configuration files to examine. System-wide configuration in /etc/flatpak/ may contain config files (*.conf ) that override defaults and also configure the repositories used in a system.

$ cat /etc/flatpak/remotes.d/flathub.flatpakrepo
[Flatpak Repo]
Title=Flathub
Url=https://dl.flathub.org/repo/
Homepage=https://flathub.org/
Comment=Central repository of Flatpak applications
Description=Central repository of Flatpak applications
Icon=https://dl.flathub.org/repo/logo.svg
GPGKey=mQINBFlD2sABEADsiUZUOYBg1UdDaWkEdJYkTSZD682
...

The configuration file describes the repository, or repo, specifies the URL location, and stores the GPG public key used to verify signatures.

The system-wide directory is /var/lib/flatpak/, which contains runtime data and further configuration. Configuration describing the basic behavior of repos can be found in the /var/lib/flatpak/repo/config file:

$ cat /var/lib/flatpak/repo/config
[core]
repo_version=1
mode=bare-user-only
min-free-space-size=500MB
xa.applied-remotes=flathub;

[remote "flathub"]
url=https://dl.flathub.org/repo/
xa.title=Flathub
gpg-verify=true
gpg-verify-summary=true
xa.comment=Central repository of Flatpak applications
xa.description=Central repository of Flatpak applications
xa.icon=https://dl.flathub.org/repo/logo.svg
xa.homepage=https://flathub.org/

Individual users can also install Flatpak repos, data, and configuration, which are fully contained in their local home directory (~/.local/share/flatpak/).

Applications are installed into their own subdirectories and found in /var/lib/flatpak/app/*. Multiple versions may exist, and symlinks indicate the current or active version. The current/active/metadata file in the Flatpak application’s directory provides configuration data for running and setting up the sandbox environment; for example:

$ cat /var/lib/flatpak/app/org.jitsi.jitsi-meet/current/active/metadata
[Application]
name=org.jitsi.jitsi-meet
runtime=org.freedesktop.Platform/x86_64/20.08

sdk=org.freedesktop.Sdk/x86_64/20.08
base=app/org.electronjs.Electron2.BaseApp/x86_64/20.08
command=jitsi-meet-run

[Context]
shared=network;ipc;
sockets=x11;pulseaudio;
devices=all;

[Session Bus Policy]
org.gnome.SessionManager=talk
org.freedesktop.Notifications=talk
org.freedesktop.ScreenSaver=talk
org.freedesktop.PowerManagement=talk

[Extension org.jitsi.jitsi_meet.Debug]
directory=lib/debug
autodelete=true
no-autodownload=true

[Build]
built-extensions=org.jitsi.jitsi_meet.Debug;org.jitsi.jitsi_meet.Sources;

Here, the different permissions, policies, paths, and more can be defined. See the flatpak-metadata(5) man page for a description of this file format.

Flatpak explicitly records installations, updates, and uninstalls in the systemd journal, which can be viewed with the flatpak history command. See the flatpak-history(1) man page for more information about Flatpak logging.

The installing and uninstalling of Flatpaks is logged to the systemd journal, as shown here:

...
Dec 05 10:14:07 pc1 flatpak-system-helper[131898]: system:
Installed app/org.sugarlabs.MusicKeyboard/x86_64/stable from flathub
...
Dec 05 10:18:24 pc1 flatpak-system-helper[131898]: system:
Uninstalled app/org.sugarlabs.MusicKeyboard/x86_64/stable
...

Here, two log entries in the systemd journal show that the Flatpak for Sugar Labs’s Music Keyboard was installed and then uninstalled a few minutes later.

The starting and stopping of Flatpak apps may also be logged in the journal:

...
Dec 05 10:14:44 pc1 systemd[400]: Started
app-flatpak-org.sugarlabs.MusicKeyboard-144497.scope.
...
Dec 05 10:16:42 pc1 systemd[400]:
app-flatpak-org.sugarlabs.MusicKeyboard-144497.scope: Succeeded.
...

Here, two log entries show the application was started and run for a few minutes before being closed. This information is also stored in the systemd user journal and can be used in a forensic examination to reconstruct past application usage.

It’s also possible to have Flatpak package bundles. They are called single-file bundles, and they have a *.flatpak file extension. Flatpak files start with a magic string of flatpak and contain the files needed to install:

00000000  66 6C 61 74 70 61 6B flatpak

This file format is taken from Docker’s Open Container Initiative (OCI). Using single-file bundles is less common than the developer-recommended use of repositories.

Snap

Software developers at Canonical created a self-contained package format called Snap together with a central app store (https://snapcraft.io/). Snap packages are designed to be distribution-independent, but Ubuntu is the only mainstream distro that uses them by default. In a forensic investigation of a system using snaps, we can determine which snaps are installed, when they were installed or updated, and information about the snap contents (files, configs, and so on).

Snap packages have a *.snap extension, but they are regular squashfs compressed filesystems. They can be easily mounted and browsed for additional information:

$ sudo mount gnome-calculator_238.snap /mnt
$ ls -l /mnt
total 1
drwxr-xr-x 2 root root  37 10. Sep 2018 bin/
-rwxr-xr-x 1 root root 237 10. Sep 2018 command-gnome-calculator.wrapper
-rw-r--r-- 1 root root  14 10. Sep 2018 flavor-select
drwxr-xr-x 2 root root   3 10. Sep 2018 gnome-platform/
drwxr-xr-x 2 root root  40 10. Sep 2018 lib/
drwxr-xr-x 3 root root  43 10. Sep 2018 meta/
drwxr-xr-x 3 root root  82 10. Sep 2018 snap/
drwxr-xr-x 5 root root  66 10. Sep 2018 usr/

Once installed, these squashfs files are mounted under the /snap/ directory on a running system (not visible during a postmortem forensic examination). Information about the package is found in meta/snap.yaml file.

Installed snaps can be found in the /var/lib/snapd/snaps/ directory, with a single file per application (and version), as shown in this example:

# ls -l /var/lib/snapd/snaps/* 
-rw------- 1 root root 179642368 Nov 20 23:34 /var/lib/snapd/snaps/brave_87.snap
-rw------- 1 root root 187498496 Dez 4 00:31 /var/lib/snapd/snaps/brave_88.snap
-rw------- 1 root root 254787584 Nov 18 18:49 /var/lib/snapd/snaps/chromium_1411.snap
-rw------- 1 root root 254418944 Dez 3 18:51 /var/lib/snapd/snaps/chromium_1421.snap
...

The example output here shows multiple versions of the Brave and Chromium browsers. The mounting is done using systemd mount unit files, which can be found in the /etc/systemd/system/ directory with a snap-*.mount filename.

Snaps rely on the snapd daemon to manage basic housekeeping. Various snapd actions are logged in the journal (or syslog):

...
Apr 07 15:21:25 pc1 snapd[22206]: api.go:985: Installing snap "subsurface" revision unset
...
Sep 28 14:41:32 pc1 snapd[8859]: storehelpers.go:438: cannot refresh snap "subsurface":
snap has no updates available
...
Nov 14 16:10:14 pc1 systemd[1]: Unmounting Mount unit for subsurface, revision 3248...
...
Nov 14 16:10:59 pc1 systemd[1]: Mounting Mount unit for subsurface, revision 3231...
...

This journal output shows the snapd logs for the Subsurface snap package.¹⁵ The output indicates the installation date, refresh (update) checks, and mounting/unmounting activity (which also corresponds to system reboots).

See the snap(8) man page and https://snapcraft.io/ for more information about snap packages.

Software Centers and GUI Frontends

Historically, package management has been highly distro specific. A collaborative effort between the major distributions began working toward a common solution to this problem. PackageKit was developed to unify package management across different distros. It provides an interface between generic frontend software management applications and backend (distro-specific) package management systems (apt, dnf, and so on). Universal package systems like Flatpak or Snap can also be managed through the same PackageKit applications. A specification for generic package metadata called AppStream was created for use across distributions and package management systems.

Installed applications can store an AppStream metadata XML file in the /usr/share/metainfo/ directory. This file contains information such as descriptions (including translations), license and version information, the project team’s homepage and contact person, the URL of screenshots displayed, and more. The screenshots are fetched from the URL specified by the project team when the user views the application in the software center. This web location and associated network traffic may be of interest in a forensic investigation. See https://www.freedesktop.org/software/appstream/docs/chap-Quickstart.html for more information about what is stored in the AppStream metadata.

The configuration files for PackageKit are found in the /etc/PackageKit/ directory. An SQLite database of packages installed by PackageKit is stored in the file /var/lib/PackageKit/transactions.db.

This effort to harmonize package management resulted in the development of universal package managers called software centers that are easy-to-use graphical applications that run on any Linux distribution. The concept of software centers is analogous to the app store programs that are popular on mobile devices and other operating systems. The following list includes some examples of Linux software centers with their command line and graphical app names:

These tools all have a similar look and feel (see Figure 7-2 for an example).

Figure 7-2: GNOME Software

In addition to generic frontends using PackageKit and AppStream, many distributions have graphical frontend tools that interface directly with their local package management system. Examples include Debian’s Synaptic or SUSE’s YaST.

In the background, these graphical tools are typically running low-level tools (like apt or dnf) or calling libraries (like libalpm or libdnf). For a forensic examination, the package management activity should be seen in the logs and local package databases as discussed earlier in this chapter. Individual tools may have their own logs (for example, they may have a daemon logging to a file or to syslog). Persistent or cache data may also reside in the user’s ~/.cache/ or ~/.local/ directories. Configuration information will usually be in /etc/ (for system-wide defaults) and in ~/.config/ (for user-customized settings).

Other Software Installation Analysis

Several other methods exist for adding software either manually or as plug-ins to existing software packages. These examples completely bypass the software package management done by the Linux distribution. However, they may still leave traces of information useful in a forensic context.

Manually Compiled and Installed Software

GNU software packages can be compiled and installed manually, bypassing any package management systems (leaving no traces in the package management logs or databases). The GNU Coding Standards documentation can be found at https://www.gnu.org/prep/standards/. The typical process involves finding the source software package online (usually a compressed tar file), downloading it to a working directory, unpacking, and running configure and make scripts. Here’s an example:

$ wget http://ftp.gnu.org/gnu/bc/bc-1.07.1.tar.gz
...
Length: 419850 (410K) [application/x-gzip]
Saving to: 'bc-1.07.1.tar.gz'
...
$ tar -xvf bc-1.07.1.tar.gz
...
bc-1.07.1/bc/bc.h
bc-1.07.1/bc/bc.c
...
$ cd bc-1.07.1/
$ ./configure
checking for a BSD-compatible install... /bin/install -c
checking whether build environment is sane... yes
...
$ make
make all-recursive
make[1]: Entering directory '/home/sam/Downloads/bc/bc-1.07.1'
...
$ sudo make install
Making install in lib
...
 /bin/mkdir -p '/usr/local/bin'
 /bin/install -c bc '/usr/local/bin'
...

The install directory can be specified and non-privileged users may install software in their home directory (like ~/.local/bin/, for example). Typically, the download site will include a separate file containing a cryptographic hash of the compressed archive file so it can be verified.

Manual downloads may also involve synchronizing (or cloning) with a software development repository like Git. A manual installation may also involve simply copying stand-alone scripts and binaries to a location in the executable path. With manual installs, there is no package management or tracking with install timestamps. The filesystem timestamps are the best indicator of when a file was installed (in particular, matching timestamps of files in the compile directory with timestamps of the installed files). The manual removal of software may involve a make uninstall command or script. If source code directories are found, it is worth examining the Makefiles to understand what was modified on the filesystem during the install (and uninstall) process. The shell history can also be examined for evidence of manual downloading, compiling, and installing of software packages.

Programming Language Packages

Some programming languages, especially interpreted languages, have their own package manager for adding additional code modules and libraries that provide extended functionality. These packages may use the distribution’s package management system or bypass it completely. This section describes a few examples of software packages that were installed directly using the programming language’s package management system.

The Python programming language has several package managers, the most popular being pip, the Python Package Installer. The pip tool is used to fetch, install, and manage Python packages. If a non-privileged user installs a package, it will be written to their home directory in ~/.local/lib/python*/ site-packages/. If it is a site installation (intended for all users) it is installed in /usr/lib/python*/site-packages/. Files or directories with the extension .egg-info contain the package metadata.

The Perl programming language has CPAN, the Comprehensive Perl Archive Network. The cpan command is used to fetch, install, and manage Perl modules. The user’s installed modules are found in ~/.cpan.

Another example is Ruby Gems (https://rubygems.org/), which downloads Ruby code from a central repository and stores it in a user’s home directory or to a site-wide location.

During a forensic examination, every user’s home directory should be analyzed to determine whether they were programmers and under which programming languages they developed. The programming languages may have a module or library package management system that was used.

Application Plug-ins

Application plug-ins are mentioned only briefly here, as the analysis is outside the scope of this book. Many large applications are extensible with themes, plug-ins, add-ons, or extensions, which are installed from within the app. This is typical of web browsers, file managers, office suites, window environments, and other programs. Plug-ins are not only used by big graphical programs, but also smaller utilities (for example, vim or neovim).

In some cases, these plug-ins are available from the distro’s package repository and are installed in standard locations that are available to other users. In other cases, a user may install plug-ins for their own use. In the latter case, the plug-ins are usually stored in the user’s home directory (in a hidden dot “.” directory together with other files associated with the application). If the application has a log or history of activity, a timestamp of installation might be found; otherwise, the filesystem timestamps are the best indicator of when the installation happened.

Summary

In this chapter, I’ve described how to examine the installed software on a Linux system. You now should be able to identify the installed distro and version numbers, and reconstruct the initial installation process. You also should be able to determine which additional software packages are installed and how to analyze the details of those packages.