Chapter Objectives
Introduce some of the criteria for choosing a cluster operating system
Describe some of the major advantages of the Linux operating system
Explain why the Linux operating system makes a good basis for clusters
Provide a survey of the most popular Linux distributions used for building clusters
This book is about building clusters using the Linux operating system. It should come as no surprise that we have to start describing the application of the Linux operating system to clusters at some point in our discussion. I will try to take a realistic view of the requirements for using Linux in building clusters.
Building a cluster's hardware from commodity pieces is only the start. To make the cluster a usable solution, it must run an operating system and applications. In short, there must be software to run on the cluster hardware. But which operating system forms a reasonable basis for a cluster environment?
In a book on Linux clusters, it should come as no surprise that Linux is the operating system of choice. But for fairness's sake, let's start by identifying some of the characteristics that a cluster software environment requires. This should help us understand the proper choice of operating system for our cluster.
Some attributes we want are
Hardware support
Stability
Low cost
Manageability
Flexibility
Scalability
Openness
Software availability and cost
Multiple support options
Support for the hardware architecture in our cluster is a must for any selected operating system. To minimize complexity, a cluster will usually consist of systems with the same processor architecture, and often the same processor speed. The operating system we choose should support the processor type we select for our cluster, and an added benefit would be support for the same functionality on multiple processor types. In this way, we can reuse our development efforts across multiple platforms.
Linux runs on a wide range of processor architectures. Currently, Linux supports systems built with IA-32, Itanium, Alpha, Opteron, PA-RISC, SPARC, and many more processors. Sometimes support for a particular processor or system configuration entails selecting a particular “flavor” of Linux, but the support is there.
Any operating system we choose must be stable and reliable, or our cluster's software environment will fail and overall reliability will suffer. Because the cluster is viewed as a single system by its users, the overall stability of that system relies, at its lowest level, on the stability of the operating system software. Note that stability in this sense refers to the operating system's ability to run long periods of time without crashing, locking up, or hanging.
Linux is very stable in both UP and multiprocessor SMP hardware configurations. Linux is increasingly used in areas in which stability and commodity costs are important. It has been used as the basis for ISPs infrastructure, provides the operating system platform for parallel Web and database servers and other business-critical systems, and has proved itself in cluster environments at many installations.
Our cluster will comprise multiple independent hardware systems, each running a copy of the chosen operating system software. Because of this, the per-system licensing cost for the operating system and other software becomes increasingly important as the cluster grows in size. Spending $300 or $700 or $1500 per node for the operating system and associated software licenses is very expensive (and not very practical) in a large cluster. The whole advantage of building clusters from commodity components is the expected cost savings over noncommodity ingredients.
Linux is available from many sources, in many different configurations. The license cost of Linux ranges all the way from free to very expensive, depending on the required support level and the vendor. Although “free is a very, very good price” (to quote a local Portland, Oregon, furniture store owner, Tom Peterson), choosing the proper distribution and associated support options will affect the cost. I will discuss this in an upcoming section.
The operating system choice must provide tools for managing the multiple systems in the cluster, along with the ability to enable the automation of the custom processes that we will create. Management in this sense requires secure remote access, remote display features, robust scripting facilities, well-designed system installation and software deployment vehicles, and a strong ability to monitor system and application performance.
The individual Linux system is every bit as manageable as any other UNIX-like system, with secure remote access, scripting, software development facilities, and other essential management components. The addition of freely available tools to the base operating system enhances manageability without necessarily increasing the software license costs. The operating system provides all the features mentioned as requirements, and packages are readily available to manage networked environments like a cluster.
A good portion of the software infrastructure that will run in our cluster is based on client–server architectures (DHCP, DNS, NFS, and so forth). The marketing departments in some software companies feel the need to create artificial packaging for the operating system and tools. Instead of providing all functionality in a single product, the operating system is divided into “desktop” and “server” configurations. There is an extra (frequently large) charge for the server portion of the software. The fundamental assumption appears to be that there will be fewer servers than clients, so it is a better revenue-generating scheme to charge more money for server functionality.
We will avoid any operating system that follows this artificial packaging scheme. We need the flexibility to choose our own architectural configurations for our cluster's internal software. Linux provides that flexibility by allowing the system manager to choose which services run where in the cluster. Artificial marketing divisions of software functionality need not get in the way of our design choices and software costs.
The Linux operating system has support for multiple programming languages, multiple scripting languages, and a variety of software development tools. Its focus on providing popular development tools, along with the operating system, makes it a good choice for software development, whether your target operating system is Linux. Linux is a software developer's dream in terms of tools and environment.
The operating system specifications, and indeed the operating source code itself, is freely available. Because of the openness, the potential to run tools and applications from multiple vendors or private sources is enhanced.
To borrow two terms from the current commercial IT world, we want to be able to scale up and scale out with our operating system. What this means is that we require the ability to scale “up” the number of processors inside a given compute slice, if this gives us an advantage, and at the same time our operating system should support “scaling out” by interconnecting the required number of individual systems in our cluster. This requires SMP scaling “inside” a system, and network connectivity scaling “outside” the system.
Linux supports both UP and multiprocessor SMP hardware in configurations most likely to be found in our cluster compute slices and infrastructure systems. Along with the ability to scale “up” the number of processors in a system, the operating system supports TCP/IP networking for both Ethernet and HSI hardware, along with support for other connection types to interconnect systems in the cluster and to connect the cluster to existing network infrastructure. Addition of MPI and the appropriate drivers allows parallel communication between application components on separate systems, if the cluster requires it.
In addition to the operating system, there are numerous other software packages needed to construct a cluster, including load-balancing (batch management, queueing, and so on) software, monitoring and management tools, software update and installation tools, compilers, communication libraries, and a wealth of other categories. If we choose an operating system environment that charges money for each of these tools, we face the same issues mentioned previously with regard to operating system license expense. We want a readily available selection of tools, at minimal cost.
Linux certainly provides a wealth of available software. As a matter of fact there is such a wide selection of Linux-based software that it often becomes difficult to choose between the many alternatives. Some of the software packages are available in free versions, as well as versions that cost money, but provide support, additional documentation, or other benefits.
Experience levels with operating system environments vary as widely as the required availability for a particular cluster. A custom solution, with very experienced local administrators requires less support than a solution in a business-critical environment. The ability to contract for required levels of both hardware and operating system support is essential to maintaining availability of the cluster.
Support is available from the major Linux vendors, such as Red Hat or SUSE, for the components contained in their distributions. Several of the major hardware vendors, like Hewlett-Packard, also provide support for Linux through their normal support channels. It is possible to choose a level of support that matches your own specific needs and budget. You also have the option to support your Linux installation yourself and work directly with the open-source community to resolve issues.
A rose by any other name would smell as sweet, and Linux, no matter how you name it, is an exciting development. However, Linux, like any other technology, is only useful once we apply it to solving a particular problem. Building a cluster with Linux is creating just such a solution. By configuring subsystems, tuning the operating system, adding the proper software packages, and creating a final configuration for the cluster environment, we are crafting a solution from raw technology.
What is commonly referred to as Linux is actually the operating system kernel alone, originally developed by Linus Torvalds. In addition to the kernel, however, there are many tools from the GNU (an acronym for “GNU is Not Unix”[1] ) project at the Free Software Foundation (FSF) along with other publicly available software. The kernel alone does not make up the whole of what we call Linux.
Much of Linux, in addition to the software that comes directly from the GNU project, is available from companies and software engineers that developed it under the auspices of the GNU general public license (GPL). Because a lot of the software that surrounds the Linux kernel comes from the GNU project, a more correct way to refer to Linux might be “GNU/Linux,” although for simplicity we will continue to call it just “Linux.” There is still some argument over this point in the community. I will take the easiest path here.
Linux is a UNIX-like operating system (a combination of the kernel and surrounding tools) that is being developed and licensed in a new and revolutionary way. The development method is termed open-source software development (OSSD), or commonly “open source.” Although the FSF would prefer that everybody refer to the software as “free software,” common usage still applies the term open source. They may not mean exactly the same thing, but it's probably too late to correct the usage at this point.
The GNU project was started in 1984 to produce an operating system that is free software. Here, the word “free” is used in a different sense than one would expect, so it is important to get the intention correct. Quoting from the FSF Web site http://www.fsf.org/philosophy/free-sw.html:
Free software is a matter of the users' freedom to run, copy, distribute, study, change and improve the software. More precisely, it refers to four kinds of freedom, for the users of the software: The freedom to run the program, for any purpose (freedom 0).
The freedom to study how the program works, and adapt it to your needs (freedom 1). Access to the source code is a precondition for this.
The freedom to redistribute copies so you can help your neighbor (freedom 2).
The freedom to improve the program, and release your improvements to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.
A program is free software if users have all of these freedoms. Thus, you should be free to redistribute copies, either with or without modifications, either gratis or charging a fee for distribution, to anyone anywhere. Being free to do these things means (among other things) that you do not have to ask or pay for permission.[2] [FSF 2004]
This is a very important distinction about the use of “free” software, focusing on the word “free.” The software does not have to be free in terms of price, but it does have to be freely available in the sense used by the GPL and the other open-source licenses under which it is distributed. You simply cannot take free software, and convert it to proprietary, closed-source products; the GPL was written to prevent this.
For a further understanding of the GPL and other “public” licenses, like the “Artistic” license (http://language.perl.com/misc/artistic.html) and the Berkeley Software Division BSD-style license (http://www.debian.org/misc/bsd.license), nothing substitutes for reading them (the GPL, in several formats, is available at http://www.fsf.org/licenses/licenses.html).
There are a lot of people with a lot of ideas and misconceptions about what they really say and mean, but only some of the opinions come from actual readings of the license text. I urge you to read at least the text of the GPL if you are not familiar with it.[3]
All arguments about origins and content aside, the primary way to get Linux is by choosing and installing a particular Linux distribution. Each Linux distribution has its own selection of open-source software packages, “value-added” content like installation tools, a particular package management system, a system structure and “philosophy,” and a particular version of the Linux kernel (along with applied patches, or “errata”) at its center. Along with the word distribution, you will also hear and see Linux distro or just distro used by the community as shorthand.
As we discussed in the previous section, a Linux distribution does not have to be free. The distribution's provider may charge for documentation, packaging, support, and other content, but the free software's source code must be provided (or be freely available) as specified by whatever open-source license applies to it. The source code for the distribution may be in the commercially packaged CD-ROMs or it may be downloadable from a Web site, but it is available.
If you go to http://www.linux.org and use their search tool to search for Intel-compatible Linux distributions, you will find 185 separate listings. Many of these have their own particular slant on how the software in the distribution may be applied: embedded systems, software development, desktop computing, minimal environments, and many more design centers. You need to choose the proper distribution for your needs carefully, and it might pay to check with http://www.distrowatch.com for the top ten distributions and their status. In the interest of space, I cover a minimal number of the available distributions here.
Among the criteria for selecting a Linux distribution for your cluster are
Support options, including security updates
Software package content
Expected longevity of the distribution
Licensing cost per system, if applicable
Installation tools and ease of installation
Manageability features
Kernel version
Hardware architecture support
Support software issues
You can see from this list that many of the same criteria we used to justify selecting Linux as the operating system for our cluster are also issues when choosing a particular Linux distribution to use for a cluster.
Choosing a nonmainstream distribution for your cluster may be acceptable, if you are ready to assume support responsibilities yourself. Linux distributions, especially those that are focused on narrow objectives, seem to come and go on a regular basis. It is likely that distributions that do not have wide acceptance may simply fade away. Frequently, the level of support required for business-critical clusters is only available for commercial distributions.
For these reasons it may be in your best interest to choose a commercially available Linux distribution to power your cluster. There are also several free distributions that may meet your requirements if you plan on supporting your own software environment. Some commercially available distributions include Red Hat Linux and SUSE Linux, and some free distributions include The Fedora Project and Debian Linux. We examine some of these distributions in upcoming sections.
The Linux kernel, and the open-source software packages that surround it, are under constant development. Features are added, bugs are fixed, and security vulnerabilities are patched, all of which generates revisions in the individual packages. The term churn, as I use it, refers to the never-ending stream of new packages that become available for a Linux environment, the effort required to install them, and the potential effects on an existing environment.
One of the great strengths of open-source development is the lack of direct financial constraints on the creativity of the programmers supporting a particular package. Bugs and issues may be very quickly resolved by contacting the developers directly via e-mail, Internet relay chat (IRC) channels, or newsgroups. The “time to resolution” for some issues is truly astounding—sometimes measured in terms of hours.
For system stability, it is normal practice to test and qualify all new software configurations before releasing them into a production environment. As new package versions are released, identifying new packages and applying the qualification process can take a substantial amount of effort, along with direct contact with the open-source developers. Your organization may not have the time or resources to sustain this level of involvement and the associated “churn.”
This situation brings us back to the definition of the word free discussed previously. The open-source Linux-based software may be free in the sense of “unencumbered,” but to apply it in your environment, and to your cluster, will have associated costs and responsibilities that match your organization's processes and requirements for reliability and stability. This is where you need to consider carefully the benefits of paying for the proper level of support for your Linux environment against expected savings for software licensing.
Your options include either assuming complete responsibility for supporting the Linux environment yourself, by hiring the appropriate people to manage the OSSD interface, buying support for Linux from your hardware vendor, or directly purchasing commercial Linux distributions that include the appropriate level of support. The message is that the use of open-source software in general, and Linux in particular, requires a different approach than you might be customary—but remember that “different” should not be equated with “bad.” The good news is that you have several choices regarding how you approach the issue.
Aside from making choices about portions of your chosen environment, your independent software vendors (ISVs) will also add some constraints to the mixture. If you are using third-party software from an ISV like Oracle, you will need to choose the distribution, and version of that distribution, that supports the ISV's software. An ISV may require a particular distribution (Red Hat or SUSE), a particular version of the distribution (Red Hat 7.3 or SUSE 8.0), and specific package revisions within that version of the distribution (glibc-2.3.2-4.80.8, XFree86-4.2.1-23, and so on).
Almost every Linux distribution has a package update tool that is intended to make updating the operating system less of a chore. Currently installed versions of packages are compared with a list of available updates, selection criteria are applied, and the packages are downloaded and automatically installed. The update tools, such as Red Hat's up2date can mitigate the software churn in the distributions that have frequent updates.
The need to generate revenue and minimize software churn for ISVs and customers is driving some of the commercial distributions toward the same “desktop” versus “server” packaging issues we described previously, along with less-frequent updates and expanded support offerings. The packaging choices complicate our efforts to build a cluster with supported versions of Linux.
We would like to have all the server software components we might need available, without having to buy, and manage, separate operating system products for compute slices, file servers, and administrative nodes in the cluster. An alternative, buying one distribution's product and adding the necessary packages ourselves, creates extra work for us in the support area and may actually invalidate support from the vendor. We will take a quick look at the current features in two commercial Linux distributions available from Red Hat and SUSE, along with the free distribution from The Fedora Project[4] and Debian Linux.
The current form of the Red Hat Linux distribution had its origins in a version of Linux that could be purchased commercially, downloaded from their Web site, and freely copied (both source and binaries) across as many systems as the customer desired. Red Hat made its money by providing optional support and update services. Today, however, the business model has changed, and you should read the Red Hat Enterprise Linux (RHEL) “frequently asked questions” list (FAQ), found at http://www.redhat.com/software/rhel/faq, for complete details.
The FAQ explains that the original Red Hat Linux products were primarily a vehicle for distributing the Linux operating system and open-source components to early adopters, Linux fans, and developers. As the Linux operating system matured, however, customers and ISVs began requesting a more stable, supported environment for their business-critical software. The current RHEL products are refocused on commercial, officially supported installations. The free version of Linux from Red Hat, called Fedora, is described later in this chapter.
The current Linux product offerings from Red Hat are purchased by yearly subscription only, and are restricted with regard to free copying. (This change occurred after the Red Hat version 9.0 software release.) The simplest way to explain the current situation is that you will need to purchase a license for every system on which you install the Red Hat software, if you use the Red Hat binaries. But, you ask, what about the free software (and the associated open-source licenses) upon which Red Hat is built?
The source code for the software open-source software that makes up the distribution is still downloadable from the Red Hat Web site. Most of the code contained in the Red Hat releases (with exceptions like Java) is all open source and covered by the GPL or a similar license. The source code is freely available from the Red Hat file transfer protocol (FTP) site, so the conditions of the open-source license are met.
Red Hat does not, however, allow you to distribute the binaries of RHEL freely. The system subscription that you buy from Red Hat consists of the binaries, access to upgrades, and some level of support services when you buy a particular packaging of the RHEL distribution. Every system on which you install RHEL must be properly licensed.
If you require a supported, low-churn version of Linux, and can afford to pay the subscription cost, then the Red Hat offerings are a very good choice. If you require a zero-dollar product, similar to the older Red Hat offerings like 6.x, 7.x, 8.x, and 9.x, then the Fedora distribution is probably a better choice. The primary difference will be the level of available “formal” support and the amount of “churn” you will need to manage.
As of this writing, Red Hat provides three Enterprise Linux subscriptions: WS (workstation), ES (enterprise server), and AS (advanced server). Within each of these product subscriptions, there are three possible “editions”: basic edition, standard edition, and premium edition. Not all of the three subscription choices make all editions available. For example, basic and standard editions are available for RHEL WS and ES subscriptions, but not for the AS subscription. Standard editions are available for WS, ES, and AS subscriptions, but a premium edition is only available for the AS subscription.
To make matters even more complicated, there are different availabilities and prices, depending on the processor architecture within the subscriptions. RHEL ES basic and standard editions only support the Intel x86 architecture. To get server support for Intel Itanium and AMD Opteron, customers need to select the RHEL AS subscription, with either the standard or premium editions. RHEL WS basic will support Itanium, but at an extra cost over and above the x86 price.
What used to be an easy choice for cluster environments is now fairly complex and can be expensive, particularly if you are deploying a cluster with Itanium or Opteron systems (see Table 8-1 for details). The Red Hat Web site mentions high-performance computing and recommends deploying the RHEL WS subscription in clusters, but this option is between $179 (x86) and $792 (Itanium and Opteron) at list price.[5] The Web site also mentions that Red Hat is working with partners to provide solutions specifically aimed at high performance technical computing (HPTC) clusters.
Table 8-1. RHEL Product Structure[a]
Red Hat Subscription and Processor Support | Basic Edition | Standard Edition | Premium Edition | |
|---|---|---|---|---|
Red Hat Enterprise Linux WS | -------------------- | -------------------- | -------------------- | |
Intel 32 bit (IA-32) | Available | Available | N/A | |
Intel Itanium (IA-64), AMD64 | N/A | Available, extra cost over IA-32 | N/A | |
Red Hat Enterprise Linux ES | ------------------- | ------------------- | -------------------- | |
Intel 32 bit (IA-32) | Available | Available | N/A | |
Red Hat Enterprise Linux AS | ------------------- | ------------------- | -------------------- | |
Intel 32 bit (IA-32) | N/A | Available | Available | |
N/A | Available, extra cost over IA-32 | Available, extra cost over IA-32 | ||
[a] As of February 24, 2004. Check the Red Hat Web site for current information. | ||||
What we are left with, if we need a supported version of Red Hat Linux for a production cluster, is a pretty hefty set of license fees (especially if you are running on Itanium or Opteron hardware) that need annual renewal to continue support. Because all compute slices in the cluster are usually identical, one has to question the usefulness of purchasing a support subscription for every identically deployed system in a cluster. If a bug or security vulnerability shows up on one system, it will show up on all of them, and the fix for one system is the same fix that is applied to all the others.
Several possible deployment strategies, based on the current product pricing and marketing decisions about processor support and packaging from Red Hat, are as follows.
Pay the license fee for the same version of RHEL (the greatest common denominator) and deploy it on all systems in the cluster (for example, RHEL ES basic edition for x86-based clusters).
Deploy the WS subscription, basic edition on all compute slices (extra cost for Itanium), and ES subscription basic edition on infrastructure, administrative, and file server systems (AS subscription premium edition will be required to support Itanium servers).
Work with your hardware vendor, or even Red Hat itself, for better pricing. Some systems come prelicensed with RHEL.
Deploy the minimal cost Red Hat configuration (lowest common denominator) and add the required open-source functionality yourself. This will have support consequences.
Search for distributions, like NWLinux,[6] which are based on recompiled sources from Red Hat, and are unencumbered by the Red Hat binary licenses.
One of the nice things about Linux and open source is that there are multiple places to get the same solution. Which approach you choose will depend on your individual circumstances, such as budget, ISV software, and need for business-critical support.
The Linux distribution from SUSE is broken down into desktop and server product offerings. The number of configurations for the server offering is only two as of this writing: SUSE Linux Standard Server and SUSE Linux Enterprise Server. The Standard Server offering supports up to two AMD32 or IA-32 (32-bit) CPUs per system and provides the server versions of subsystems that we require for our cluster infrastructure and administrative nodes. The Enterprise Server offering provides support for up to 32-way, 32-bit SMP systems or up to 64-way, 64-bit (AMD64 or Itanium) SMP systems, and provides the required server components of client–server software.The SUSE Linux desktop product structure is shown in Table 8-2.
Table 8-2. SUSE Linux Desktop Product Structure[a]
SUSE Desktop Product | Personal | Professional | |
|---|---|---|---|
SUSE Linux 9.0 | ------------------ | ----------------- | |
Intel 32 bit (IA-32) | Supported | Supported | |
AMD64 | N/A | Supported at extra cost | |
[a] Product structure as of February 24, 2004. Check the SUSE Web site for current information. | |||
The SUSE Linux desktop offerings are primarily aimed at providing an interactive environment for office automation. The only processors supported for the desktop versions of the SUSE distribution are the Intel x86 and AMD64 architectures. This means that a server version of the distribution must be purchased to support Itanium processors. The SUSE Linux server product structure is shown in Table 8-3.
Table 8-3. SUSE Linux Product Structure[a]
SUSE Linux Server Product | Up to 2 CPUs | Up to 8 CPUs | |
|---|---|---|---|
SUSE Linux Standard Server 8 | ----------------------- | --------------------- | |
Supported | N/A | ||
AMD 64 bit, Intel 64 bit Itanium (AMD64, IA-64) | N/A | N/A | |
SUSE Linux Enterprise Server 8 | ----------------------- | -------------------- | |
Intel/AMD 32 bit (IA-32, AMD32) | Supported | Supported | |
AMD 64 bit, Intel 64 bit Itanium (AMD64, IA-64) | Supported | Supported | |
[a] Product structure as of February 24, 2004. Check the SUSE Web site for current information. | |||
On the surface, this would appear to be a substantially easier product structure than the Red Hat offerings. Once you get beyond the surface level of the two server products, however, the complexity of the SUSE server product offerings equals Red Hat product complexity. There are different pricing options for SUSE's Linux Standard Server and Linux Enterprise Server, based on the number (either “up to two” or “up to eight”) of processors and the type of processor. Support is purchased in units of two or eight processors.
As with Red Hat products, the SUSE product's license only allows you to install their distribution on systems that have a valid maintenance agreement, even though the software packages in their distribution are covered by the GPL and other public, open-source licenses. Thus, selecting SUSE as your cluster's Linux distribution will depend more on preference and application support rather than on simplification and reduction of licensing costs. Once again, the inclusion of support substantially raises the cost of deploying a cluster solution.
To make money, remain in business, and grow, commercial Linux companies have to sell support services and their own “value-added” products along with the open-source software (which they cannot directly sell in the traditional way) in their distributions. The prices they charge are aimed primarily at commercial IT environments[7] and the replacement of existing proprietary UNIX servers, and are not easily adapted to cost savings in a cluster environment. This places cluster designers in an interesting position (some would say “catch-22”) with regard to providing a supported cluster operating system environment, and yet maintaining commodity cost savings with cluster solutions.
Licenses on current commercial distributions of Linux from Red Hat and SUSE restrict the number of systems you can install with their binaries, without purchasing support, to a single system. Because a license is needed for every copy of the operating system that is installed, the cost negates part of the price savings that justified a commodity cluster approach in the first place. Although some distributions that comprise the recompiled sources from commercial distributions are available, they do not provide the level of support required for business-critical situations.
If you are not willing to produce an in-house Linux configuration from a free distribution, your option is to select a commercial version that includes support in its price. This is certainly true if you are running third-party software that was qualified against a specific distribution and version of that distribution. To provide a supported, qualified environment for your ISV software will narrow the range of your choices. Some of the commercial Linux distributions appear to be recognizing the difficulties presented for cluster solutions by the current licensing costs. Concrete solutions do not appear to be available as of this writing.
If you are running your own software, or have the resources to support a recompiled distribution yourself, then you have a wider range of choices. You may be able to deploy a freely available distribution on compute slices (the majority of the cluster members), and deploy a commercial Linux distribution only where there is a specific need for application or specialized support (file servers, administrative nodes, and so on). The next section describes two freely available distributions.
There are several free Linux distributions available. Two of them are Fedora, from The Fedora Project, and Debian Linux. Both of these distributions are free in the sense of “no charge,” may be downloaded from their respective Web sites, and may be installed on as many systems as desired. The next two sections discuss these distributions.
If you require a freely copyable version of Linux, one that tracks the most current open-source developments and follows the standard open-source level of support (e-mail, Web, newsgroups, and IRC), you should consider the Linux distribution provided by The Fedora Project. This distribution includes all software components (client and server) in the distribution package, making no distinction between “workstations,” “desktops,” or “servers.” As of this writing, the functionality of Fedora closely resembles Red Hat Linux version 9.0, with some changes. It is likely to move away from this resemblance in the future (particularly with the addition of the new 2.6 kernel version in the Core 2 releases).
The objectives for Fedora are also different from the commercial Linux distributions like Red Hat and SUSE, which are attempting to provide the stability required by ISVs and commercial customers. The Fedora Project is not a supported product provided by Red Hat. To see the exact definitions and goals of the project, visit The Fedora Project Web site at http://fedora.redhat.com. Some of the goals are
To provide a trial environment for technologies that may become part of the RHEL products
To provide a complete Linux operating system distribution that is built from free software
To provide regular releases somewhere around two or three times a year
If you are willing to create your own support and qualification process, the Fedora distribution may be a potential solution, particularly if you are running your own software and do not have to rely on qualified operating system and software combinations from ISVs.
The development status of this distribution is at once an advantage and a disadvantage. As an advantage, Fedora is likely to have the new 2.6 version of the Linux kernel available before the other commercial distributions release it. As a disadvantage, the Fedora distribution will have a higher than average amount of churn because of its development and “leading-edge” focus.
As of this writing, the Fedora distribution release is currently “Core 1,” which supports the Intel IA-32 (x86) processor architraves. A test version of this release is available for AMD64 processors. The Core 2 release is expected to support the 2.6 version of the Linux kernel and was released in early spring of 2004.
One of the primary objectives for the Debian distribution is to provide a Linux environment consisting of free software. The Debian Project Web site, http://www.debian.org/intro/about, states:
The Debian Project is an association of individuals who have made common cause to create a free operating system. This operating system that we have created is called Debian GNU/Linux, or simply Debian for short. [Debian 2004]
The Debian distribution is in no way compromised by providing only free software. There are more than 8,700 packages listed on their Web site. For a list of the available packages for the Debian distribution, see http://www.debian.org/distrib/packages. Commercial software packages are also available for Debian Linux.
The Debian distribution supports a wide range of hardware architectures. Supported architectures include Alpha, PA-RISC, Itanium (IA-64), IA-32 (x86), SPARC, ARM, and quite a few others. As of this writing, Debian does not support the AMD processor architecture. It is worth checking the Debian distribution's Web site to see if it currently supports the processor architecture you are targeting.
If you are very cost conscious, you will need to consider the free Linux distributions as a way to save license expense. Of course, this will only be possible if you do not have applications that are qualified against a specific commercially supported distribution. The savings in license costs need to be weighed against the cost of supporting the Linux environment yourself.
You should not immediately dismiss the possibility of supporting a free distribution yourself, or looking for alternative support channels. There is no reason that you cannot adopt a more measured release strategy using the high-churn development distributions. Even some proprietary UNIX operating systems are infamous for the number of patches issued per unit of time.
System administrators have crafted workable release and qualification processes to deal with minimizing risk to production environments from high-churn commercial operating systems, and the circumstances with free Linux distributions are not entirely different. If you identify the portions of the environment that are really in use, particularly on the compute slices, you may be surprised at how minimal the Linux installation needs to be.
You should be aware of the potential difference in directory structure and package management between the two free distributions. Fedora, because it is derived from the Red Hat distributions, uses the Red Hat package manager (the rpm command and .rpm file extension for software packages) and has a similar directory structure (this may change or evolve). The Debian distribution uses a different set of package management tools (the dpkg command and .deb file extension for software packages) and will have a slightly different system structure and administration approach.
We have taken a pretty hard look at the product structure of two commercial Linux distributions that may be used in a cluster environment. Because of the product structure and pricing for commercial Linux distributions, there are a number of issues to overcome if you are extremely cost sensitive. I have several important points to make about using commercial versions of Linux in a cluster.
The license costs for commercial distributions include support and updates.
The availability of free tools may more than offset the expense of a commercial distribution (such as compilers, monitoring tools, management tools, and so forth).
You may deploy less-capable versions of a commercial offering (for example, RHEL WS) on compute slices if your applications allow it.
An application vendor, such as Oracle, will qualify their software against one or more specific versions of commercial distributions (for example, RHEL AS and SUSE Linux Enterprise Server 8.0), and may include specific kernel or package version requirements.
The processor type you choose for your cluster will affect licensing costs.
Hardware vendors may include licenses for some commercial distributions along with the system.
Some of the issues associated with using free distributions of Linux (Debian or Fedora) in a cluster are the following.
Support is tied to communication with the open-source software developers or to providing your own Linux expertise.
Although the acquisition of the software is free, the use and support of it within your organization still has associated costs that must be weighed against the cost of commercial distributions.
Applications from ISVs may not be qualified and supported for free distributions.
It is possible to manage open-source software churn by creating your own qualification and release process, thus gaining the benefits from a free distribution.
As with any design decision, choosing the proper Linux distribution will depend on your own situation and the characteristics of your cluster. Research, scientific, or experimental clusters in organizations with extensive Linux and open-source experience may have no difficulty employing free Linux distributions. Business-critical clusters with third-party software, or organizations with limited Linux and open-source experience, will have a more limited set of choices. But at least there are choices.
The recognition, by commercial Linux distribution vendors, that a cluster represents a special case for support, would make our selection of a distribution—commercial or otherwise—a whole lot easier. Although there is a tangible benefit to having support on all systems in a cluster, paying for the service on 256 compute slices in a cluster does not deliver 256 times the value of buying support on a single business-critical server. Scaling “up” does not currently equate with scaling “out” using commercial Linux distributions. It should not be more expensive to license eight two-CPU systems than a single 16-CPU system. Until this realization is made clear and reasonable solutions are found, cluster designers will have to weigh all operating system costs and benefits very carefully, potentially making less-than-optimum choices.
With all support issues aside, Linux still provides the most flexible operating system environment for clusters. The wide range of freely available, open-source software, like compilers, network installation tools, management tools, monitoring tools, Web servers, databases, infrastructure services, and other essential components, adds to the potential for savings. These components, along with the flexibility and openness to choose custom configurations to meet specific needs, make Linux a very cost-effective environment for cluster solutions. What we need is a version of a commercial Linux distribution that is targeted for compute slices, and a special support pricing scheme that acknowledges that clusters are a special case.
[1] You will notice that the GNU acronym is “recursive” in that the G stands for GNU, which reintroduces the acronym “inside” the acronym. UNIX programmers, systems people, and Linux developers love this sort of “in” or tricky semantic joke.
[2] Copyright © 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111, USA. Verbatim copying and distribution of this entire article is permitted in any medium without royalty provided this notice is preserved.
[3] This is also referred to as a copyleft in reference to the familiar term copyright, because the software is first copyrighted, then the distribution terms (free distribution, and so forth) are specifically added to the copyright. The terms of the copyleft are specified in the GPL.
[4] Please note that information provided is provided for informational purposes only and is obtained from the public Web sites for Red Hat Linux http://www.redhat.com, SUSE Linux http://www.suse.com, and The Fedora Project http://fedora.redhat.com. Product offerings, contents, and other specifications may change at any time. You should check with the specific Web site to verify the current information before making purchasing or design decisions.
[5] Price is US list price on February 23, 2004, from the Red Hat Web site. See the information listed at http://www.redhat.com/software/rhel/purchase for current details.
[6] Information is available at http://www.emsl.pnl.gov/nwlinux. This distribution is specifically for Itanium-based systems and is based on Red Hat Advanced Server 2.1 for Itanium.
[7] Only time will tell how successful this approach will be. Linux is currently, by most reports, displacing mainly proprietary UNIX systems in IT departments. Many research or educational institutions are using free or special versions of Linux distributions to provide the operating system software for their clusters. We are apparently on the cusp of some changes, at least with regard to general cluster solution acceptance and commercial Linux pricing.