Downloading Source Code

Once developers are ready to release their open-source applications to the world, they publish them on the Internet. Developers for most open-source packages use a website to host their code and documentation, and many even provide user forums that allow customers to discuss issues and possible improvements.

While you can use a graphical browser to connect to a website and download source code, that’s not always available, especially in Linux server environments. Linux provides a couple of command-line tools to help us download source code files directly from the command line.

The wget application is a command-line tool from the GNU Project that allows you to retrieve files from remote servers using FTP, FTPS, HTTP, or HTTPS. You specify the protocol, server name, and file to download using a standard URL format, where remotehost is the full hostname for the location hosting the files, and filename is the name of the source code file you wish to retrieve, including the folder path required:

wget http://remotehost/filename

Yet another solution is the cURL application. It does the same thing as wget but supports many more protocols, such as DAP, DICT, FILE, Gopher, IMAP, LDAP, POP3, RTSP, SCP, SFTP, SMTP, and TFTP. It too uses the standard URL format for you to specify the protocol, server name, and file to download.

One nice feature of cURL is its ability to work with the secure HTTPS protocol. It will warn you if the remote website is using a self-signed certificate or if the certificate is signed by an untrusted certificate authority (CA).

Bundling Source Code Packages

Distributing the source code for applications can be a bit tricky. Source code projects often comprise many different files:

• Source code files
• Header files
• Library files
• Documentation files

Trying to distribute a large batch of files for a project can be a challenge. Linux provides somewhat of an odd solution for that.

The tar program was originally developed for archiving files and folders to tape drives for backups (the tar name originally stood for tape archiver). These days it also comes in handy for bundling project files to distribute on the Internet.

The tar command allows you to specify multiple files, or even multiple folders of files, to bundle together into a single output file. You can then transfer the entire project bundle as a single file and extract the files and folders on a remote system. It’s so versatile in what it can do that there is a long list of command-line options available, which can become somewhat imposing.

For most bundling operations, three basic option groups are commonly used for the tar command:

• -cvf: Create a new tar file
• -tvf: Display the contents of a tar file
• -xvf: Extract the contents of a tar file

To create a new tar archive file, specify the output file name and then the list of files and folders to bundle, as shown in the example in Listing 13.1.

Listing 13.1: Using the tar command to bundle files

$ tar -cvf test.tar test1.txt test2.txt test3.txt
test1.txt
test2.txt
test3.txt
$ ls -al
total 32
drwxr-xr-x  2 rich rich  4096 Dec  5 08:33 .
drwxr-xr-x 19 rich rich  4096 Dec  5 08:28 ..
-rw-r--r--  1 rich rich   795 Dec  5 08:19 test1.txt
-rw-r--r--  1 rich rich  1020 Dec  5 08:19 test2.txt
-rw-r--r--  1 rich rich  2280 Dec  5 08:20 test3.txt
-rw-r--r--  1 rich rich 10240 Dec  5 08:33 test.tar
$

In Listing 13.1, test.tar is the name of the archive file to create. For the input files and folders, you can use wildcard characters to specify the names, or even redirect a listing of files to the tar command, making it very versatile in scripts. One of the advantages of bundling folders with tar is that it preserves the folder structure of your environment, including file and folder ownership, making it easier to extract the files and re-create the original environment.

If you need to see what’s in a tar archive file, use the -tvf option group:

$ tar -tvf test.tar
-rw-r--r-- rich/rich       795 2018-12-05 08:19 test1.txt
-rw-r--r-- rich/rich      1020 2018-12-05 08:19 test2.txt
-rw-r--r-- rich/rich      2280 2018-12-05 08:20 test3.txt
$

Notice that both the file ownerships and the file permissions are retained within the tar archive file. When you extract the files onto another system, they’ll be assigned to the userid that matches the user number assigned to the original files.

Extracting the files and folders from a tar file is just a matter of using the -xvf option group:

$ tar -xvf test.tar
test1.txt
test2.txt
test3.txt
$ ls -al
total 32
drwxr-xr-x  2 rich rich  4096 Dec  5 08:38 .
drwxr-xr-x 20 rich rich  4096 Dec  5 08:38 ..
-rw-r--r--  1 rich rich   795 Dec  5 08:19 test1.txt
-rw-r--r--  1 rich rich  1020 Dec  5 08:19 test2.txt
-rw-r--r--  1 rich rich  2280 Dec  5 08:20 test3.txt
-rw-r--r--  1 rich rich 10240 Dec  5 08:38 test.tar
$

While the tar archive method makes bundling files for distribution easy, it does tend to create a very large file, which can be awkward to handle. Linux developers usually compress the final tar archive file using some type of file compression utility.

In Linux there is a plethora of ways to create compressed files. Table 13.1 lists the most popular methods you’ll run into.

By far the most common zip utility used in Linux for tar archive files is the GNU gzip package. To compress a single file, use the gzip utility with the file name, as shown in Listing 13.2.

Listing 13.2: Compressing a tar archive file

$ gzip test.tar
$ ls -al
total 24
drwxr-xr-x  2 rich rich 4096 Dec  5 08:53 .
drwxr-xr-x 20 rich rich 4096 Dec  5 08:39 ..
-rw-r--r--  1 rich rich  795 Dec  5 08:19 test1.txt
-rw-r--r--  1 rich rich 1020 Dec  5 08:19 test2.txt
-rw-r--r--  1 rich rich 2280 Dec  5 08:20 test3.txt
-rw-r--r--  1 rich rich  204 Dec  5 08:33 test.tar.gz
$

As seen in Listing 13.2, the gzip program adds a .gz file name extension to the end of the file that’s compressed.

To decompress a compressed tarball and extract the original files, you have a couple of options. One option is to use a two-step approach. First use the gunzip command directly on the compressed tar file:

$ gunzip test.tar.gz

This restores the original test.tar file. Then you extract the tar file using the standard -xvf options of the tar command.

The second option is to decompress and extract the tarball file in one step by just adding the -z option to the tar command line:

$ tar -zxvf test.tgz
test1.txt
test2.txt
test3.txt
$ ls -al
total 24
drwxr-xr-x 2 rich rich 4096 Dec  5 09:03 .
drwxr-xr-x 3 rich rich 4096 Dec  5 09:02 ..
-rw-r--r-- 1 rich rich  795 Dec  5 08:19 test1.txt
-rw-r--r-- 1 rich rich 1020 Dec  5 08:19 test2.txt
-rw-r--r-- 1 rich rich 2280 Dec  5 08:20 test3.txt
-rw-r--r-- 1 rich rich  204 Dec  5 09:02 test.tgz
$

Compiling Source Code

Once you have the source code package files downloaded onto your Linux system, you’ll need to compile them to create an executable file to run the application. Linux supports a wide variety of programming languages, so you’ll need to know just what programming language the application was written in. Once you know that, you’ll need to install a compiler for the program code. A compiler converts the source code into an executable file the Linux system can run.

The most common tool used for compiling programs in Linux is the GNU Compiler Collection (gcc). While originally created to support only the C programming language, gcc now supports an amazing array of different programming languages, such as Ada, C++, Fortran, Go, Java, Objective-C, Objective-C++, and OpenMP.

To compile simple one-file programs, just run the gcc command-line command against the source code file to produce the executable file that you run on your system. The -o command-line option allows you to specify the name of the compiled output file; otherwise it defaults to the ugly a.out file name:

$ cat hello.c
#include <stdio.h>
int main() {
   printf("Hello, this is my first C program!
");
   return 0;
}
$ gcc -o hello hello.c
$ ./hello
Hello, this is my first C program!
$

As mentioned earlier, most larger applications require additional header and library files besides the source code files to build the final application file. Depending on just how many source code, header, and library files are required for an application, the gcc command process can get very long and complicated. Separate library files need to be compiled in the proper order before the main program file can be compiled, creating a difficult road map to follow to generate the application.

There’s a simple solution available for you to help keep track of all that. The make utility allows developers to create scripts that guide the compiling and installation process of application source code packages so that even novices can compile and install an application from source code.

Usually there are three steps involved with installing an application that uses a make script:

1. Run the configure utility, which analyzes your Linux system and customizes the make script to build the application for your environment.
2. Run the make utility by itself to build the necessary library files and executable files for the application.
3. Run make install as the root user account to install the application files in the appropriate locations on your system.

What makes C language programs so complicated is that they often split the application functions into separate library files. Each library file contains one or more specialized functions used in the application code.

The benefit of splitting functions into separate library files is that multiple applications that use the same functions can share the same library files. These files, called shared libraries, make it easier to distribute applications but more complicated to keep track of what library files are installed with which applications.

While not necessary for compiling the application source code, the ldd utility can come in handy if you need to track down missing library files for an application. It displays a list of the library files required for the specified application file:

$ ldd hello
    linux-vdso.so.1 (0x00007fff0f378000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e16063000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f8e16656000)
$

My simple hello application requires two external library files, the standard linux-vdso.so.1 and libc.so.6 files, which provide the ability for the printf() function to display the output. The ldd utility also shows where those files were found on the Linux system. That in itself can be helpful when troubleshooting issues with applications picking up the wrong library files.

Installing and Managing Packages

Both the Debian and Red Hat package management systems have similar sets of tools for working with software packages in the package management system. We’ll now take a look at both systems and the tools to use with them.

dpkg [options] action package-file

$ sudo dpkg -i zsh_5.3.1-4+b2_amd64.deb
Selecting previously unselected package zsh.
(Reading database ... 204322 files and directories currently installed.)
Preparing to unpack zsh_5.3.1-4+b2_amd64.deb ...
Unpacking zsh (5.3.1-4+b2) ...
dpkg: dependency problems prevent configuration of zsh:
 zsh depends on zsh-common (= 5.3.1-4); however:
  Package zsh-common is not installed.

dpkg: error processing package zsh (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Errors were encountered while processing:
 zsh
$

$ dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===========================
ii  accountsservic 0.6.45-1ubun amd64        query and manipulate accounts
ii  acl            2.2.52-3buil amd64        Access control list utilities
ii  acpi-support   0.142        amd64        scripts for handling ACPI
ii  acpid          1:2.0.28-1ub amd64        Advanced Config and Power
ii  adduser        3.116ubuntu1 all          add and remove users
ii  adium-theme-ub 0.3.4-0ubunt all          Adium message style for Ubuntu
ii  adwaita-icon-t 3.28.0-1ubun all          default icon theme of GNOME
ii  aisleriot      1:3.22.5-1   amd64        GNOME solitaire card game
...

$ dpkg -l openssh*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=============================
ii  openssh-client 1:7.6p1-4ubu amd64        secure shell (SSH) client
un  openssh-server <none>       <none>       (no description available)
$

rpm action [options] package-file

$ sudo rpm -Uvh zsh-5.0.2-31.el7.x86_64.rpm
Preparing...                    ################################# [100%]
Updating / installing...
  1:zsh-5.0.2-31.el7            ################################# [100%]
$

$ rpm -q zsh
zsh-5.0.2-31.el7.x86_64
$

$ sudo rpm -e zsh
$ sudo rpm -q zsh
package zsh is not installed
$

Understanding Repositories

The dpkg and rpm commands are useful tools, but they both have their limitations. If you’re looking for new software packages to install, it’s up to you to find them. Also, if a package depends on other packages to be installed, it’s up to you to install those packages first and in the correct order. That can become somewhat of a pain to keep up with.

To solve that problem, each Linux distribution has its own central clearinghouse of packages, called a repository. The repository contains software packages that have been tested and known to install and work correctly in the distribution environment. By placing all known packages into a single repository, the Linux distribution can create a one-stop shopping environment for installing all applications for the system.

Most Linux distributions create and maintain their own repositories of packages. There are also additional tools for working with package repositories. These tools can interface directly with the package repository to find new software and even automatically find and install any dependent packages the application requires to operate.

Besides the officially supported distribution package repositories, many third-party package repositories have sprung up on the Internet. Often specialized or custom software packages aren’t distributed as part of the normal Linux distribution repository but are available in third-party repositories. The repository tools allow you to retrieve those packages as well.

The following sections walk through how to use the Debian and Red Hat repository tools.

$ sudo apt-get install zsh
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Suggested packages:
  zsh-doc
The following NEW packages will be installed:
  zsh
...
Setting up zsh (5.4.2-3ubuntu3.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
$

$ cd /etc/yum.repos.d
$ ls -al
total 44
drwxr-xr-x.   2 root root  187 Sep 17 21:47 .
drwxr-xr-x. 142 root root 8192 Dec 15 16:55 ..
-rw-r--r--.   1 root root 1660 Sep 17 21:39 CentOS-Base.repo
-rw-r--r--.   1 root root 1309 Aug 13 10:34 CentOS-CR.repo
-rw-r--r--.   1 root root  649 Aug 13 10:34 CentOS-Debuginfo.repo
-rw-r--r--.   1 root root  314 Aug 13 10:34 CentOS-fasttrack.repo
-rw-r--r--.   1 root root  630 Aug 13 10:34 CentOS-Media.repo
-rw-r--r--.   1 root root 1331 Aug 13 10:34 CentOS-Sources.repo
-rw-r--r--.   1 root root 4768 Aug 13 10:34 CentOS-Vault.repo
$

$ sudo yum install zsh
[sudo] password for rich:
...
========================================================================
 Package        Arch              Version                 Repository       Size
========================================================================
Installing:
 zsh            x86_64            5.0.2-31.el7            base            2.4 M

Transaction Summary
========================================================================
Install  1 Package

Total download size: 2.4 M
Installed size: 5.6 M
Is this ok [y/d/N]: y
...
Installed:
  zsh.x86_64 0:5.0.2-31.el7                                      

Complete!
$

Graphical Package Tools

Both the Debian-based and Red Hat–based package management systems have graphical tools for making it easier to install software in desktop environments. One tool that is available in both the Ubuntu and CentOS distributions is gnome-software.

The gnome-software program is a graphical front end to the PackageKit tool, which itself is a front end that standardizes the interface to multiple package management tools, including apt and yum. By including both PackageKit and gnome-software, Linux distributions can provide a standard graphical interface for users to manage their software packages. Figure 13.1 shows the gnome-software package as it appears in the Ubuntu 18.04 Linux distribution.

You can search for packages, view the installed packages, and even view the updated packages available in the repository. If you’re using the CentOS Linux distribution, the gnome-software interface looks the same, as shown in Figure 13.2.

Finally, some standardization is happening across Linux distributions, at least where it comes to graphical software package management tools.

Exercise 13.1 Working with packages

This exercise demonstrates how to work with a package management system to install software.

1. Log into your Linux system and open a new command prompt.
2. Display the packages currently installed on your system. For Debian-based systems such as Ubuntu, use the command sudo apt-cache pkgnames. For Red Hat–based systems such as CentOS, use the command sudo yum list.
3. If it’s not already installed on your system, install the zsh shell package. For Debian-based systems, use the command sudo apt-get install zsh. For Red Hat–based systems, use the command sudo yum install zsh. If the zsh package is already installed, try installing the tcsh package, which is an open-source version of the C shell found in many Unix systems.
4. Display the installed packages on your system again to see if the newly installed package appears.
5. Now remove the package from your system. For Debian-based systems, use the command sudo apt-get remove zsh. For Red Hat–based systems, use the command sudo yum remove zsh.
6. Display the installed packages yet again to see if the package was properly removed.