Hardware

There are many different kinds of computers from single-board computers (SBC) like the Arduino and the Raspberry Pi to desktop computers, servers, mainframes, and supercomputers. Many of these use Intel or AMD processors, but others do not. For the purposes of this series of books, I will work with Intel X86_64 hardware. Generally, if I say Intel, you can also assume I mean the X86_64 processor series and supporting hardware, and that AMD X86_64 hardware should produce the same results, and the same hardware information will apply.

The processor

Let’s take a moment to explore the CPU and define some terminology in an effort to help reduce confusion. Five terms are important when we talk about processors: processor, CPU, socket, core, and thread. The Linux command lscpu, as shown in Figure 2-1, gives us some important information about the installed processor(s) as well as clues about terminology. I use my primary workstation for this example.

The first thing to notice in Figure 2-1 is that the term “processor” never appears. The common usage for the term “processor”⁵ refers generically to any hardware unit that performs some form of computations. It can refer to the CPU⁶ – central processing unit – of the computer, to a graphic processing unit (GPU⁷) that performs calculations relating to graphical video displays, or any number of other types of processors. The terms processor and CPU tend to be used interchangeably when referring to the physical package that is installed in your computer.

Using Intel terminology, which can be a bit fluid, the processor is the physical package that can contain one or more computing cores. Figure 2-2 shows an Intel i5-2500 processor which contains four cores. Because the processor package is plugged into a socket and a motherboard may have multiple sockets, the lscpu utility numbers the sockets. Figure 2-1 shows the information for the processor in socket number 1 on the motherboard. If this motherboard had additional sockets, lscpu would list them separately.

A core, which is sometimes referred to as a compute core, is the smallest physical hardware component of a processor that can actually perform arithmetic and logical computations, that is, it is composed of a single arithmetic and logic unit (ALU)⁸ and its required supporting components. Every computer has at least one processor with one or more cores. Most modern Intel processors have more – two, four, or six cores, and many processors have eight or more cores. They make up the brains of the computer. They are the part of the computer which is responsible for executing each of the instructions specified by the software utilities and application programs.

The line in the lscpu results that specifies the number of cores contained in the processor package is “Core(s) per socket.” For this socket on my primary workstation, there are sixteen (16) cores. That means that there are 16 separate computing devices in the processor plugged into this socket.

But wait – there’s more! The line “CPU(s)” shows that there are 32 CPUs on this socket. How can that be? Look at the line with the name “Thread(s) per core,” and the number there is 2, so 16 x 2 = 32. Well that is the math but not the explanation. The short explanation is that compute cores are really fast. They are so fast that a single stream of instructions and data is not enough to keep them busy all the time even in a very compute intensive environment. The details of why this is so are beyond the scope of this book but suffice it to say that before hyper-threading, most compute cores would sit waiting with nothing to do, while the slower external memory circuitry tried to feed them sufficient steams of program instructions and data to them to keep them active.

Rather than let precious compute cycles go to waste in high-performance computing environments, Intel developed hyper-threading technology that allows a single core to process two streams of instructions and data by switching between them. This enables a single core to perform almost as well as two. So the term CPU is used to specify that a single hyper-threading core is reasonably close to the functional equivalent of two CPUs.

But there are some caveats. Hyper-threading is not particularly helpful if all you are doing is word processing and spreadsheets. Hyper-threading is intended to improve performance in high-performance computing environments where every CPU compute cycle is important in speeding the results.

The operating system

All of these hardware pieces of the computer must work together. Data must be gotten into the computer and moved about between the various components. Programs must be loaded from long-term storage on the hard drive into RAM where they can be executed. Processor time needs to be allocated between running applications. Access to the hardware components of the computer such as RAM, disk drives, and printers by application programs must be managed.

It is the task of the operating system to provide these functions. The operating system manages the operation of the computer and of the application software which runs on the computer.

Memory management

Linux and other modern operating systems use advanced memory management strategies to virtualize real memory – random-access memory⁹ (RAM) and swap memory (disk) – into a single virtual memory space which can be used as if it were all physical RAM. Portions of this virtual memory¹⁰ can be allocated by the memory management functions of the kernel to programs that request memory.

The memory management components of the operating system are responsible for assigning virtual memory space to applications and utilities and for translation between virtual memory spaces and physical memory. The kernel allocates and deallocates memory and assigns physical memory locations based upon requests, either implicit or explicit, from application programs. In cooperation with the CPU, the kernel also manages access to memory to ensure that programs only access those regions of memory which have been assigned to them. Part of memory management includes managing the swap partition or file and the movement of memory pages between RAM and the swap space on the hard drive.

Virtual memory eliminates the need for the application programmer to deal directly with memory management because it provides a single virtual memory address space for each program. It also isolates each application’s memory space from that of every other, thus making the program’s memory space safe from being overwritten or viewed by other programs.

Multitasking

Linux, like most modern operating systems, can multitask. That means that it can manage two, three, or hundreds of processes at the same time. Part of process management is managing multiple processes that are all running on a Linux computer.

I usually have several programs running at one time such as LibreOffice Write which is a word processor, an e-mail program, a spreadsheet, a file manager, a web browser, and usually multiple terminal sessions in which I interact with the Linux command-line interface (CLI). Right now, as I write this sentence, I have multiple documents open in several LibreOffice Write windows. This enables me to see what I have written in other documents and to work on multiple chapters at the same time.

But those programs usually do little or nothing until we give them things to do by typing words into the word processor or clicking an e-mail to display it. I also have several terminal emulators running and use them to log in to various local and remote computers for which I manage and have responsibility.

Linux itself always has many programs running in the background – called daemons – programs that help Linux manage the hardware and other software running on the host. These programs are usually not noticed by users unless we specifically look for them. Some of the tools you will learn about in this book can reveal these otherwise hidden programs.

Even with all of its own programs running in the background and users’ programs running, a modern Linux computer uses a few compute cycles and wastes most of its CPU cycles waiting for things to happen. Linux can download and install its own updates while performing any or all of the preceding tasks simultaneously – without the need for a reboot. Wait... what?! That’s right. Linux does not usually need to reboot before, during, or after installing updates or when installing new software. After a new kernel or glibc (General C Libraries) is installed, however, you may wish to reboot the computer to activate it, but you can do that whenever you want and not be forced to reboot multiple times during an update or even stop doing your work while the updates are installed.

Multiuser

The multitasking functionality of Linux extends to its ability to host multiple users – tens or hundreds of them – all running the same or different programs at the same time on one single computer.

Multiuser capabilities means a number of different things. First, it can mean a single user who has logged in multiple times via a combination of the GUI desktop interface and via the command line using one or more terminal sessions. We will explore the extreme flexibility available when using terminal sessions a bit later in this course. Second, multiuser means just that – many different users logged in at the same time, each doing their own thing, and each isolated and protected from the activities of the others. Some users can be logged in locally and others from anywhere in the world with an Internet connection if the host computer is properly configured.

The role of the operating system is to allocate resources to each user and to ensure that any tasks, that is, processes, they have running have sufficient resources without impinging upon the resources allocated to other users.

Process management

The Linux kernel manages the execution of all tasks running on the system. The Linux operating system is multitasking from the moment it boots up. Many of those tasks are the background tasks required to manage a multitasking and – for Linux – a multiuser environment. These tools take only a small fraction of the available CPU resources available on even modest computers.

Each running program is a process. It is the responsibility of the Linux kernel to perform process management.¹¹

The scheduler portion of the kernel allocates CPU time to each running process based on its priority and whether it is capable of running. A task which is blocked – perhaps it is waiting for data to be delivered from the disk, or for input from the keyboard – does not receive CPU time. The Linux kernel will also preempt a lower priority task when a task with a higher priority becomes unblocked and capable of running.

In order to manage processes, the kernel creates data abstractions that represent that process. Part of the data required is that of memory maps that define the memory that is allocated to the process and whether it is data or executable code. The kernel maintains information about the execution status such as how recently the program had some CPU time, how much time, and a number called the “nice” number. It uses that information and the nice number to calculate the priority of the process. The kernel uses the priority of all of the process to determine which process(es) will be allocated some CPU time.

Note that not all processes need CPU time simultaneously. In fact, for most desktop workstations in normal circumstances, usually only two or three processes at the most need to be on the CPU at any given time. This means that a simple quad-core processor can easily handle this type of CPU load.

If there are more programs – processes – running than there are CPUs in the system, the kernel is responsible for determining which process to interrupt in order to replace it with a different one that needs some CPU time.

Interprocess communication

Interprocess communication (IPC) is vital to any multitasking operating system. Many programs must be synchronized with each other to ensure that their work is properly coordinated. Interprocess communication is the tool that enables this type of inter-program cooperation.

The kernel manages a number of IPC methods. Shared memory is used when two tasks need to pass data between them. The Linux clipboard is a good example of shared memory. Data which is cut or copied to the clipboard is stored in shared memory. When the stored data is pasted into another application, that application looks for the data in the clipboard’s shared memory area. Named pipes can be used to communicate data between two programs. Data can be pushed into the pipe by one program, and the other program can pull the data out of the other end of the pipe. A program may collect data very quickly and push it into the pipe. Another program may take the data out of the other end of the pipe and either display it on the screen or store it to the disk, but it can handle the data at its own rate.

Device management

The kernel manages access to the physical hardware through the use of device drivers. Although we tend to think of this in terms of various types of hard drives and other storage devices, it also manages other input/output (I/O) devices such as the keyboard, mouse, display, printers, and so on. This includes management of pluggable devices such as USB storage devices and external USB and eSATA hard drives.

Access to physical devices must be managed carefully, or more than one application might attempt to control the same device at the same time. The Linux kernel manages devices so that only one program actually has control of or access to a device at any given moment. One example of this is a COM port.¹² Only one program can communicate through a COM port at any given time. If you are using the COM port to get your e-mail from the Internet, for example, and try to start another program which attempts to use the same COM port, the Linux kernel detects that the COM port is already in use. The kernel then uses the hardware error handler to display a message on the screen that the COM port is in use.

For managing disk I/O devices, including USB, parallel and serial port I/O, and filesystem I/O, the kernel does not actually handle physical access to the disk but rather manages the requests for disk I/O submitted by the various running programs. It passes these requests on to the filesystem, whether it be EXT[2,3,4], VFAT, HPFS, CDFS (CD-ROM file system), or NFS (Network Filesystem, or some other filesystem types), and manages the transfer of data between the filesystem and the requesting programs.

We will see later how all types of hardware – whether they are storage devices or something else attached to a Linux host – are handled as if they were files. This results in some amazing capabilities and interesting possibilities.

Error handling

Errors happen. As a result, the kernel needs to identify these errors when they occur. The kernel may take some action such as retrying the failing operation, displaying an error message to the user, and logging the error message to a log file.

In many cases, the kernel can recover from errors without human intervention. In others, human intervention may be required. For example, if the user attempts to unmount¹³ a USB storage device that is in use, the kernel will detect this and post a message to the umount program which usually sends the error message to the user interface. The user must then take whatever action necessary to ensure that the storage device is no longer in use and then attempt to unmount the device.

Starting with UNICS

The history of Linux begins with UNICS which was originally written as a gaming platform to run a single game. Ken Thompson was an employee at Bell Labs in the late 1960s – before the breakup – working on a complex project called Multics. Multics was an acronym that stood for Multiplexed Information and Computing System. It was supposed to be a multitasking operating system for the GE (yes, General Electric) 645¹⁷ mainframe computer. It was a huge, costly, complex project with three very large organizations, GE, Bell Labs, and MIT, working on it.

Although Multics never amounted to much more than a small bump along the road of computer history, it did introduce a good number of then innovative features that had never before been available in an operating system. These features included multitasking and multiuser capabilities.

Ken Thompson,¹⁸ one of the developers of Multics, had written a game called Space Travel¹⁹ that ran under Multics. Unfortunately, due at least in part to the committee-driven design of Multics, the game ran very slowly. It was also very expensive to run at about $50 per iteration. As with many projects developed by committees, Multics died a slow, agonizing death. The platform on which the Space Travel game was run was no longer available.

Thompson then rewrote the game to run on a DEC PDP-7 computer similar to the one in Figure 2-3 that was just sitting around gathering dust. In order to make the game run on the DEC, he and some of his buddies, Dennis Ritchie²⁰ and Rudd Canaday, first had to write an operating system for the PDP-7. Because it could only handle two simultaneous users – far fewer than Multics had been designed for – they called their new operating system UNICS for UNiplexed Information and Computing System as a bit of geeky humor.

UNIX

At some time later, the UNICS name was modified slightly to UNIX, and that name has stuck ever since.

In 1970, recognizing its potential, Bell Labs provided some financial support for the Unix operating system and development began in earnest. In 1972 the entire operating system was rewritten in C to make it more portable and easier to maintain than the assembler it had been written in allowed for. By 1978, Unix was in fairly wide use inside AT&T Bell Labs and many universities.

Due to the high demand, AT&T decided to release a commercial version of Unix in 1982. Unix System III was based on the seventh version of the operating system. In 1983, AT&T released Unix System V Release 1. For the first time, AT&T promised to maintain upward compatibility for future versions. Thus programs written to run on SVR1 would also run on SVR2 and future releases. Because this was a commercial version, AT&T began charging license fees for the operating system.

Also, in order to promote the spread of Unix and to assist many large universities in their computing programs, AT&T gave away the source code of Unix to many of these institutions of higher learning. This caused one of the best and one of the worst situations for Unix. The best thing about the fact that AT&T gave the source code to universities was that it promoted rapid development of new features. It also promoted the rapid divergence of Unix into many distributions.

System V was an important milestone in the history of Unix. Many Unix variants are today based on System V. The most current release is SVR4 which is a serious attempt to reconverge the many variants that split off during these early years. SVR4 contains most of the features of both System V and BSD. Hopefully they are the best features.

A (very) brief history of Linux

Linus Torvalds, the creator of Linux, was a student at Helsinki University in 1991. The university was using a very small version of Unix called Minix for school projects. Linus was not very happy with Minix and decided to write his own Unix-like operating system.²⁴

Linus wrote the kernel of Linux and used the then ubiquitous PC with an 80386 processor as the platform for his operating system because that is what he had on hand as his home computer. He released an early version in 1991 and the first public version in March of 1992.

Linux spread quickly, in part because many of the people who downloaded the original versions were hackers like Linus and had good ideas that they wanted to contribute. These contributors, with guidance from Torvalds, grew into a loose international affiliation of hackers dedicated to improving Linux.

Linux is now found in almost all parts of our lives.²⁵ It is ubiquitous, and we depend upon it in many places that we don’t normally even think about. Our mobile phones, televisions, automobiles, the International Space Station, most supercomputers, the backbone of the Internet, and most of the web sites on the Internet all utilize Linux.

For more detailed histories of Linux, see Wikipedia²⁶ and its long list of references and sources.

GNU coreutils

To understand the origins of the GNU core utilities, we need to take a short trip in the Wayback Machine to the early days of Unix at Bell Labs. Unix was originally written so that Ken Thompson, Dennis Ritchie, Doug McIlroy, and Joe Ossanna could continue with something they had started while working on a large multitasking and multiuser computer project called Multics. That little something was a game called “Space Travel.” As is true today, it always seems to be the gamers that drive forward the technology of computing. This new operating system was much more limited than Multics as only two users could log in at a time, so it was called Unics. This name was later changed to UNIX.

Over time, UNIX turned out to be such a success, that Bell Labs began essentially giving it away to universities and later to companies, for the cost of the media and shipping. Back in those days, system-level software was shared between organizations and programmers as they worked to achieve common goals within the context of system administration.

Eventually the PHBs²⁷ at AT&T decided that they should start making money on Unix and started using more restrictive – and expensive – licensing. This was taking place at a time when software in general was becoming more proprietary, restricted, and closed. It was becoming impossible to share software with other users and organizations.

Some people did not like this and fought it with – free software. Richard M. Stallman²⁸ led a group of rebels who were trying to write an open and freely available operating system that they call the “GNU Operating System.” This group created the GNU utilities but did not produce a viable kernel.

When Linus Torvalds first wrote and compiled the Linux kernel, he needed a set of very basic system utilities to even begin to perform marginally useful work. The kernel does not provide these commands or even any type of command shell such as bash. The kernel is useless by itself. So Linus used the freely available GNU core utilities and recompiled them for Linux. This gave him a complete operating system even though it was quite basic.

You can learn about all of the individual programs that comprise the GNU utilities by entering the command info coreutils at a terminal command line. The utilities are grouped by function to make specific ones easier to find. Highlight the group you want more information on, and press the Enter key.

There are 102 utilities in that list. It does cover many of the basic functions necessary to perform some basic tasks on a Unix or Linux host. However, many basic utilities are missing. For example, the mount and umount commands are not in this list. Those and many of the other commands that are not in the GNU coreutils can be found in the util-linux collection.

util-linux

The util-linux package of utilities contains many of the other common commands that SysAdmins use. These utilities are distributed by the Linux Kernel Organization, and virtually every distribution uses them. These 107 commands were originally three separate collections, fileutils, shellutils, and textutils, which were combined into the single package, util-linux, in 2003.

These two collections of basic Linux utilities, the GNU core utilities and util-linux, together provide the basic utilities required to administer a basic Linux system. As I researched this book, I found several interesting utilities in this list that I never knew about. Many of these commands are seldom needed. But when you do, they are indispensable. Between these two collections, there are over 200 Linux utilities. Linux has many more commands, but these are the ones that are needed to manage the most basic functions of the typical Linux host. The lscpu utility that I used earlier in this chapter is distributed as part of the util-linux package.

I find it easiest to refer to these two collections together as the Linux core utilities.

Copyleft

Just because Linux and its source code are freely available does not mean that there are no legal or copyright issues involved. Linux is copyrighted under the GNU General Public License Version 2 (GPL2). The GNU GPL2 is actually called a copyleft instead of a copyright by most people in the industry because its terms are so significantly different from most commercial licenses. The terms of the GPL allow you to distribute or even to sell Linux (or any other copylefted software), but you must make the complete source code available without restrictions of any kind, as well as the compiled binaries.

The original owner – Linus Torvalds in the case of parts of the Linux kernel – retains copyright to the portions of the Linux kernel he wrote, and other contributors to the kernel retain the copyright to their portions software no matter by whom or how much it is modified or added to.