—Lance Tost

Figure 4-1. Multiple xterms and a multixterm control window

Another nice benefit of MediaWiki is that it’s a breeze to install. It does require PHP, and its creators strongly suggest using MySQL as the backend database. Depending on the features you want to use (for example, image thumbnailing or LDAP authentication), you may need PHP to be compiled against specific libraries, but the requirements to run a simple site are pretty slim.

If you’re hosting your own site (i.e., you have root privileges), installation takes, quite literally, seconds. All you do is untar the distribution into the document root of your web server, and go to the site! MediaWiki knows if it’s your first visit, and prompts you to perform an installation. Once you supply the MySQL administrator password, MediaWiki will create a new user, a new database, and all the necessary tables, which is 90% of the installation process.

If you’re running MediaWiki on a hosted remote server, however, you’re not likely to have a root password or an administrative password for MySQL. In this case, you’ll want to create the MediaWiki database first, and then point the installation at this database to create its tables. Unfortunately, I can’t tell you how to do this, as every hosting service will provide different tools to assist you.

Once the installation has been performed successfully, you’ll be presented with a link to visit your new site.

Installing it was easy, but how do you lock this thing down? There’s about a metric ton of documentation available on how to do this, but I’ll summarize some of the features that are of primary importance to administrators.

First and foremost is site access. Many sites haven’t deployed Wikis because they’re under the illusion that they can’t be secured. Not so!

With Version 1.4 of MediaWiki, it’s possible to use the configuration file and/or a few SQL statements to change the functions available to different types of users. Version 1.5, on the other hand, has quite a robust collection of potential roles that users can play, implemented via user groups. We’ll work with Version 1.5 here, because it’ll likely be in its final form by the time this goes to press.

I’m working with 1.5rc4, which can be managed largely in a browser. There are separate pages for adding, deleting, and blocking users. There is also a page for changing the groups to which users belong, which will affect the rights they have when they visit your site. In addition, there are plug-ins available to help you correlate users with all the known IP addresses used by them and perform other functions not available in the main distribution.

However, there isn’t yet an interface for changing the rights for a group, or adding/deleting groups. For those tasks, you’ll need to have shell access to the web server, or you’ll need to create a local copy of the LocalSettings.php file, edit it, and copy it back into place to make the changes take effect. The file is simple to edit, and the documentation for making the changes is more than adequate, but I’ll go over examples of one or two quick changes you might want to make.

If you just want to change the group a user is associated with, you can log in as an administrative user and go to the Special Pages link. At the bottom of the screen you’ll see Restricted special pages that are listed only when the admin is logged in. This section contains the link to the user rights management page, which is currently just an interface to change the group memberships of specific users.

If you want to create a group, you’ll need to edit LocalSettings.php and set up the rights available to the group. To see how the default groups are set up, check the documentation or open up the includes/DefaultSettings.php file in your install directory. Here are the lines you would add to LocalSettings.php to add a group called newgroup with permission to read and edit but not to delete:

	$wgGroupPermissions['newgroup']['edit'] = true;
	$wgGroupPermissions['newgroup']['read'] = true;
	$wgGroupPermissions['newgroup']['delete']= false;

As you can see, there’s no explicit “create group” function. Assigning rights to a nonexistent group, as I’ve done here, will cause the group to be created, and it will be listed as an available group the next time you go to the user rights page.

Keep in mind that there are global settings as well, for the all group (represented in the configuration as *). Here are a few default settings for that group from the DefaultSettings.php file:

	$wgGroupPermissions['*' ]['createaccount'] = true;
	$wgGroupPermissions['*' ]['read'] = true;
	$wgGroupPermissions['*' ]['edit'] = true;

If you want to override these settings, just place similar lines in the LocalSettings.php file, setting the appropriate permissions to true or false as desired. The LocalSettings.php file overrides any corresponding settings that may be found in the DefaultSettings.php file.

This model gives you the flexibility to, for example, disable anonymous users from creating accounts at all or allow them only to read, and to require users to log in to edit anything. There are also additional rights you can give users to make them quasi-administrators, allowing them to create accounts for other users, delete files, and roll back bad edits.

Once user access is out of the way, probably the most important decisions you’ll make in running your Wiki have to do with how the content on your site will be structured and how your content best maps to the organizational elements available to you in MediaWiki. There are many useful tools you can use, and all of them are fairly generic—again making no assumptions about the purpose of the site.

There are many ways to use the various organizational elements. If you have just one project group, they can have their own Wiki devoted to their project. However, you could potentially have several projects share a single Wiki by providing separate namespaces on the site. A namespace is the highest-level data element provided in MediaWiki. Within the namespaces are categories, that the project maintainers themselves can use to break up their sites into various pieces that make sense for their needs.

Lest you think the pages of the site need to be completely static documents, have a look at the Templates feature of MediaWiki, which allows you to embed documents within various pages. This gives you the flexibility to, for example, make your main page nothing but a collection of various other documents placed into the main page. Maintainers of the various templated pages can then update their own content, and changes will be reflected on the main page without affecting templates created by other users.

The nohup command provides a quick and easy mechanism for keeping a process running regardless of whether its parent process is still active. To take advantage of this capability, run your favorite command, preceded by the nohup command:

	$ nohup command

This executes the specified command and keeps it running even if the parent session ends. If you don’t redirect output from this process, both its output and error messages (stdout and stderr) will be sent to a file called nohup.out in the current directory. If this file can’t be created there, it will be created in the home directory of the user that ran the command.

You can monitor what’s being written to nohup.out using the tail command:

	$ tail –f ~/nohup.out

You can also explicitly direct the output of your command to a specified file. For example, the following command line runs the specified command in the background, sends its output to a file called my_test_output.txt in your home directory and continues running it even if the parent session ends:

	$ nohup command > ~/my_test_output.txt &

If you don’t want to save the output of the specified command, you can discard it (and not create the nohup.out file) by redirecting output to /dev/null, the Linux bit-bucket:

	$ nohup command > /dev/null &

This runs the command or program in the background, ignores its output by sending it to /dev/null, and continues running it even if the parent session ends.

If you’re using the bash shell, you can tell an existing process to ignore SIGHUPs by using the shell’s disown built-in command:

	$ disown -h jobnumber

This tells a job already running in the background to keep running when its parent process shuts down. You can find its job number using the shell’s jobs command. If you use the disown built-in’s -h option, the running job won’t be removed from the jobs table when you disown it, but it will keep running if the current shell ends. You can still reconnect to this process using the standard bash %job-number mechanism. If you use the disown built-in with no options, the running job will be removed from the jobs table: it will still continue running after you log out, but you won’t be able to reconnect to it.

You can also use the disown command to keep all current background jobs running:

	$ disown -ar

This tells all running jobs to keep running even if the current shell closes.

—Jon Fox

If you manage more than a few machines, you’ve probably come up with some way of automating the process of connecting to some subset of your service machines at login time, or with a desktop icon, or by some other means that is more automated than manually opening up terminal windows and typing the commands to connect to each host. If you use SSH keys ( Hack #66 in the original Linux Server Hacks), you can create a simple shell script to automate this process for you. Here’s an example shell script:

	#!/bin/bash

	screen -d -m -S svr1 -t jonesy@svr1 ssh server1.linuxlaboratory.org
	screen -d -m -S svr2 -t jonesy@svr2 ssh server2.linuxlaboratory.org
	screen -d -m -S svr3 -t jonesy@svr3 ssh server3.linuxlaboratory.org

Save this script to your ~/bin directory, and make sure you make it executable!

What makes this script work well is calling screen with the –d –m flags, which tell screen to start the session, but not to attach to it. Note as well that I’ve used –S to specify a session name, so when I want to attach to, say, svr1, I can just type screen –r svr1. In addition, I’ve used the –t flag to specify a title for my shell, which will show in the titlebar of my terminal emulator to help me keep track of where I am.

The list of components you’ll need to configure may sound slightly intimidating, but it’s much easier than it looks, and once you get it to work once, automating the setup process and replicating the ease of installation is a breeze. Before you do anything, though, make sure that the hosts you want to install have network cards that support a Preboot eXecution Environment (PXE). This is a standard booting mechanism supported by firmware burned into the network card in your server. Most server-grade network cards, and even recent desktop network cards, support PXE. The way to check is generally to enter the BIOS settings and see if there’s an option to enable PXE, or to carefully watch the boot messages to see if there are settings there for PXE booting. On a lot of systems, simply hitting a function key during bootup will cause the machine to boot using PXE.

	host pxetest {
		hardware ethernet 0:b:db:95:84:d8;
		fixed-address 192.168.198.112;
		next-server 192.168.101.10;
		filename "/tftpboot/linux-install/pxelinux.0";
		option ntp-servers 192.168.198.10, 192.168.198.23;
	}

	Aug 9 06:05:55 livid dhcpd: [ID 702911 daemon.info] DHCPDISCOVER from 00:
	40:96:35:22:ff (jonesy-thinkpad) via 172.16.1.1
	Aug 9 06:05:55 livid dhcpd: [ID 702911 daemon.info] DHCPOFFER on 192.168.
	198.101 to 00:40:96:35:22:ff (jonesy-thinkpad) via 192.168.198.100

	/tftpboot/
		linux-install/
		pxelinux.0
		vmlinuz
		initrd.img
		pxelinux.cfg/
			default

	$ mkdir -p /tftpboot/linux-install/pxelinux.cfg

	default linux

	label linux
		kernel vmlinuz
		append ksdevice=eth0 load_ramdisk=1 prompt_ramdisk=0 network
		ks=nfs:myserver:/kickstart/Profiles/pxetest

Your host is PXE-enabled; your DHCP server is configured to deliver the necessary information to the target host; and the TFTP server is set up to provide the host with a boot file, a configuration file, a kernel, and a ramdisk image. All that’s left to do now is boot! Here’s the play-by-play of what takes place, for clarity’s sake:

Here are some of the problems you may run into and how to tackle them:

How is this possible? There’s an age-old solution that’s provided as a binary package by just about every Linux distribution, and it’s called minicom. If you need to build from source, you can download it at http://alioth.debian.org/projects/minicom/. minicom can do a multitude of great things, but what I use it for is to provide a console interface to a server over a serial connection, using a null modem cable (otherwise known as a crossover serial cable).

Actually, that’s a big, fat lie. My laptop, as it turns out, doesn’t have a serial port! I didn’t even look to confirm that it had one when I ordered it, but I’ve found that many newer laptops don’t come with one. If you’re in the same boat, fear not! Available at online shops everywhere, for your serial connection pleasure, are USB-to-serial adapters. Just plug this thing into a USB port, then connect one end of the null modem cable to the adapter and the other end to the server’s serial port, and you’re in business.

With hardware concerns taken care of, you can move on to configuring minicom. A default configuration directory is usually provided on Debian systems in /etc/minicom. On Red Hat systems, the configuration files are usually kept under /etc and do not have their own directory. Customizing the configuration is generally done by running this command as root:

	# minicom –s

This opens a text-based interface where you can make the necessary option changes. The configuration gets saved to a file called minirc.dfl by default, but you can use the “Save setup as” menu option to give the configuration a different name. You might want to do that in order to provide several configuration files to meet different needs—the profile used at startup time can be passed to minicom as a lone argument.

For example, if I run minicom -s, and I already have a default profile stored in minicom.dfl, I can, for instance, change the baud rate from the default 9,600 to 115,200 and then save this as a profile named fast. The file created by this procedure will be named minicom.fast, but when I start up I just call the profile name, not the filename, like this:

	$ minicom fast

Of course, this assumes that a regular user has access to that profile. There is a user access file, named minicom.users, that determines which users can get to which profiles. On both Debian and Red Hat systems, all users have access to all profiles by default.

A slightly simpler way to get a working configuration is to steal it. Here is a barebones configuration for minicom. Though it’s very simple, it’s really the only one I’ve ever needed:

	# Machine-generated file - use "minicom -s" to change parameters.
	pu port			/dev/ttyUSB0
	pu baudrate		9600
	pu bits			8
	pu parity		N
	pu stopbits		1
	pu minit
	pu mreset
	pu mconnect
	pu mhangup

I included here the options stored to the file by default, even though they’re not used. The unused settings are specific to situations in which minicom needs to perform dialups using a modem. Note in this config file that the serial device I’m using (the local device through which minicom will communicate) is /dev/ttyUSB0. This device is created and assigned by a Linux kernel module called usbserial. If you’re using a USB-to-serial adapter and there’s no indication that it’s being detected and assigned to a device by the kernel, check to make sure that you have this module. Almost every distribution these days provides the ubserial module and dynamically loads it when needed, but if you build your own kernels, make sure you don’t skip over this module! In your Linux kernel configuration file, the option CONFIG_USB_SERIAL should be set to y or m. It should not be commented out.

The next setting is the baudrate, which has to be the same on both the client and the server. In this case, I’ve picked 9,600, not because I want to have a turtle-slow terminal, but because that’s the speed configured on the servers to which I usually connect. It’s plenty fast enough for most things that don’t involve tailing massive logfiles that are updated multiple times per second.

The next three settings dictate how the client will be sending its data to the server. In this case, a single character will be eight bits long, followed by no parity bit and one stop bit. This setting (referred to as “8N1”) is by far the most common setting for asynchronous serial communication. These settings are so standard that I’ve never had to change them in my minicom.conf file—in fact, the only setting I do change is the baud rate.

Once you have your configuration in place, connect your null modem or USB-to-serial adapter to your laptop and connect the other end to the serial console port on the server. If you’re doing this for the first time, the serial console port on the server is a 15-pin male connection that looks a lot like the male version of a standard VGA port. It’s also likely to be the only place you can plug in a null modem cable! If there are two of them, generally the one on the top (in a vertical configuration) or on the left (in a horizontal configuration) will be ttyS0 on the server, and the other will be ttyS1.

After you’ve physically connected the laptop to the server, the next thing to do is fire up a terminal application and launch minicom:

	$ minicom

This command will launch minicom with its default configuration. Note that on many systems, launching the application alone doesn’t do much: you have to hit Enter once or twice to get a login prompt returned to you.

I’ve rarely had trouble using minicom in this way, especially when the server end is using agetty to provide its of the communication, because agetty is pretty forgiving and can adjust for things like seven-bit characters and other unusual settings. In the event that you have no output or your output looks garbled, check to make sure that the baud rate on the client matches the baud rate on the server. Also make sure that you are, in fact, connected to the correct serial port! On the server, try typing the following to get a quick rundown of the server settings:

	$ grep agetty /etc/inittab
	co:2345:respawn:/sbin/agetty ttyS0 9600 vt100-nav
	$

This output shows that agetty is in fact running on ttyS0 at 9600 baud. The vt100-nav option on the end is put there by the Fedora installation program, which sets up your inittab entry by default if something is connected to the console port during installation. The vt100-nav option sets the TERM environment variable. If you leave this setting off, most Linux machines will just set this to vt100 by default, which is generally fine. If you want, you can tell minicom to use an alternate terminal type on the client end with the -t flag.

If you’re having trouble launching minicom, make sure you don’t have restrictions in place in the configuration file regarding who is allowed to use the default profile.

The best way to explain this is with an example. I have a file that is the result of the dumping of all the data in my LDAP directory. It consists of the LDIF entries of all the users in my environment.

One entry looks like this:

	dn: cn=jonesy,ou=People,dc=linuxlaboratory,dc=org
	objectClass: top
	objectClass: person
	objectClass: organizationalPerson
	objectClass: inetOrgPerson
	objectClass: posixAccount
	objectClass: evolutionPerson
	uid: jonesy
	sn: Jones
	cn: Brian K. Jones
	userPassword: {crypt}eRnFAci.Ie2Ny
	loginShell: /bin/bash
	uidNumber: 3025
	gidNumber: 410
	homeDirectory: /u/jonesy
	gecos: Brian K. Jones,STAFF
	mail: [email protected]
	roomNumber: 213
	fileas: Jones, Brian K.
	telephoneNumber: NONE
	labeledURI: http://www.linuxlaboratory.org
	businessRole: NONE
	description: NONE
	homePostalAddress: NONE
	birthDate: 20030101
	givenName: Brian
	displayName: Brian K. Jones
	homePhone: 000-000-0000
	st: NJ
	l: Princeton
	c: UStitle: NONE
	o: Linuxlaboratory.orgou: Systems Group

There are roughly 1,000 entries in the file. What I need to do, for every user, is tag the end of every labeledURI line with a value of ~username. This will reflect a change in our environment in which every user has some web space accessible in her home directory, which is found on the Web using the URL http://www.linuxlibrary.org/~username. Some entries have more lines than others, so there’s not a whole heckuva lot of consistency or predictability to make my job easy. You could probably write some really ugly shell script or Perl script to do this, but you don’t actually even have to leave the cozy confines of Vim to get it done. First, let’s record a macro. Step 1 is to type (in command mode) q n, where n is a register label. Valid register labels are the values 0–9 and a–z. Once you do that, you’re recording, and Vim will store in register n every single keystroke you enter, so type carefully! Typing q again will stop the recording.

Here are the keystrokes I used, including my keystrokes to start and stop recording:

	qz
	/uid:<Enter>
	ww
	yw
	/labeledURI<Enter>
	A
	/~
	<Esc>
	p
	q

The first line starts the recording and indicates that my keystrokes will be stored in register z. Next, I search for the string uid: (/uid:), move two words to the right (ww), and yank (Vim-ese for copy) that word (yw). Now I have the username, which I need to paste on the end of the URL that’s already in the file. To accomplish this, I do a search for the labeledURI attribute (/labeledRUI), indicate that I am going to append to the end of the current line (A), type a /~ (because those characters need to be there and aren’t part of the user’s ID), and then hit Esc to enter command mode and immediately hit p to paste the copied username. Finally, I hit q to stop recording.

Now I have a nice string of keystrokes stored in register z, which I can view by typing the following command:

	:register z
	"z /uid: ^Mwwyw/labeledURI: ^MA/~^[p

If you can see past the control characters (^M is Enter and ^[ is Escape), you’ll see that everything I typed is there. Now I can call up this string of keystrokes any time I want by typing (again, in command mode) @z. It so happens that there are 935 entries in the file I’m working on (I used wc –l on the file to get a count), one of which has been edited already, so if I just place my cursor on the line underneath the last edit I performed and type 934@z, that will make the changes I need to every entry in the file. Sadly, I have not found a way to have the macro run to the end of the file without specifying a number.

I happen to really like the concept of WYSIWYG HTML editors. I like the idea of not having to be concerned with tag syntax. To that extent, these editors represent a decent abstraction layer, enabling me to concentrate more on content than form. They also do away with the need to remember the tags for things such as greater than and less than characters and nonbreaking spaces, which is wonderful.

Unfortunately, none of these shiny tools allows me to use Vim keystrokes to move around within a file. I’m not even asking for search and replace or any of the fancy register stuff that Vim offers—just the simple ability to move around with the h, j, k, and l keys, and maybe a few other conveniences. It took me a long time to figure out that I don’t need to compromise anymore! I can have the full power of Vim and use it to create an environment where the formatting, while not completely invisible, is really a no-brain-required activity.

Here’s a perfect example of one way I use Vim keyboard shortcuts every day. I have to write some of my documentation at work in HTML. Any time my document contains a command that has to be run, I enclose that command in <code></code> tags. This happens a lot, as the documentation I write at work is for an audience of sysadmins like me. The other two most common tags I use are the <p></p> paragraph tags and the <h2></h2> tags, which mark off the sections in the documentation. Here’s a line I’ve entered in my ~/.vimrc file so that entering code tags is as simple as hitting F12 on my keyboard.

	imap <F12> <code> </code> <Esc>2F>a

The keyword imap designates this mapping as being active only in insert mode. I did this on purpose, because I’m always already in insert mode when I realize I need the tags. Next is the key I’m mapping to, which is, in this case, F12. After that are the actual tags as they will be inserted. Had I stopped there, hitting F12 in insert mode would put in my tags and leave my cursor to the right of them. Because I’m too lazy to move my cursor manually to place it between the tags, I put more keystrokes on the end of my mapping. First, I enter command mode using the Esc key. The 2F> bit says to search from where the cursor is backward to the second occurrence of >, and then the a places the cursor, back in insert mode, after the > character. I never even realize I ever left insert mode—it’s completely seamless!

Let’s have a look at some code. This first script is really simple; it’s a simplified version of a script I use to avoid having to use the standard ldapsearch tool with a whole bunch of flags. For example, if I want to search a particular server in another department for users with the last name Jones and get back the distinguished name (dn) attribute for each of these users, here’s what I have to type:

	$ ldapsearch -x -h ldap.linuxlaboratory.org -b"dc=linuxlaboratory,dc=org" '
	(sn=Jones)' dn

Yucky. It’s even worse if you have to do this type of search often. I suppose you could write a shell script, but I found that PHP was perfectly capable of handling the task without relying on the ldapsearch tool being on the system at all. In addition, PHP’s universality is a big plus—everyone in my group has seen PHP before, but some of them code in tcsh, which is different enough from ksh or bash to be confusing. Don’t forget that the code you write today will become someone else’s problem if a catastrophic bug pops up while you’re on a ship somewhere sipping margaritas, far from a cell phone tower. Anyway, here’s my script, which I call dapsearch:

	#!/usr/bin/php
	<?php

	$conn=ldap_connect("ldap.linuxlaboratory.org")
	 or die("Connect failed
");

	$bind = ldap_bind($conn)
	 or die("Bind failed
");

	$answer = ldap_search($conn, "dc=linuxlaboratory,dc=org", "($argv[1])")	;
	$output = ldap_get_entries($conn, $answer);

	for ($i=0; $i < count($output); $i++) {
			if(!isset($output[$i])) break;
			echo $output[$i]["dn"]."
";
	}
	echo $output["count"]." entries returned
";
	?>

There are a couple of things to note in the code above. On the first line is your everyday “shebang” line, which contains the path to the binary that will run the code, just like in any other shell or Perl script. If you’re coding on your desktop machine for later deployment on a machine you don’t control, you might replace that line with one that looks like this:

	#!/usr/bin/env php

This does away with any assumption that the PHP binary is in a particular directory by doing a standard PATH search for it, which can be more reliable.

In addition, you’ll notice that the <?php and ?> tags are there in the shell script, just like they are in web scripts. This can be useful in cases where you have static text that you’d like output to the screen, because you can put that text outside the tags instead of using echo statements. Just close the tag, write your text, then open a new set of tags, and the parser will output your text, then start parsing PHP code when the tags open again.

Also, you can see I’ve simplified things a bit by hard-coding the attribute to be returned (the dn attribute), as well as the server to which I’m connecting. This script can easily be altered to allow for that information to be passed in on the command line as well. Everything you pass on the command line will be in the argv array.

Save the above script to a file called dapsearch, make it executable, and then run it, passing along the attribute for which you want to search. In my earlier ldapsearch command, I wanted the distinguished name attributes of all users with the last name “Jones.” Here’s the (greatly shortened) command I run nowadays to get that information:

	$ dapsearch sn=Jones

This calls the script and passes along the search filter, which you’ll see referenced in the code as $argv[1]. This might look odd to Perl coders who are used to referencing a lone argument as either @_, $_,or $argv[0]. In PHP, $argv[0] returns the command being run, rather than the first argument handed to it on the command line.

Speaking of the argv array, you can run into errors while using this feature if your installation of PHP doesn’t enable the argv and argc arrays by default. If this is the case, the change is a simple one: just open up your php.ini file (the configuration file for the PHP parser itself) and set register_argc_argv to on.

—Lance Tost

You can obtain the latest version of distcc from http://distcc.samba.org/download.html. Just download the archive, uncompress it, and run the standard build commands:

	dbrick@rivendell:$ tar -jxvf distcc-2.18.3.tar.bz2
	dbrick@rivendell:$ cd distcc-2.18.3
	dbrick@rivendell:$ ./configure && make && sudo make install

You must install the program on each machine you want included in your compile farm. On each of the compiling machines, you need to start the distccd daemon:

	root@bree:# distccd –daemon –N15
	root@moria:# distccd –daemon –N15

These daemons will listen on TCP port 3632 for instructions and code from the local machine (the one which you are actually compiling software for). The –N value sets a niceness level so the distributed compiles won’t interfere too much with local operations. Read the distccd manpage for further options.

On the client side, you need to tell distcc which computers to use for distributed compiles. You can do this by creating an environment variable:

	dbrick@rivendell:$ export DISTCC_HOSTS='localhost bree moria'

Specify localhost to make sure your local machine is included in the compiles. If your local machine is exceptionally slow, or if you have a lot of processors to distribute the load to, you should consider not including it at all. You can use machine IP addresses in place of names. If you don’t want to set an environment variable, then create a distcc hosts file in your home directory to contain the values:

	dbrick@rivendell:$ mkdir ~/.distcc
	dbrick@rivendell:$ echo "localhost bree moria" > ~/.distcc/hosts

To run a distributed compile, simply pass a CC=distcc option to the make command:

	dbrick@rivendell:$ make –j7 CC=distcc

It’s that simple to distribute your compiles. Read the manpages for distcc and distccd to learn more about the program, including how to limit the number of parallel makes a particular computer in your farm will perform.

Though some clever people have come up with very interesting ways to distribute compiles to a Windows machine using Cygwin, there is an easier way to perform the same task using a live CD distribution known as distccKnoppix, which you can download from http://opendoorsoftware.com/cgi/http.pl?p=distccKNOPPIX. Be sure to download the version that has the same major version number of gcc as your local machine.

To use distccKnoppix, simply boot the computer using the CD, note it’s IP address, and then enter that in your distcc hosts file or environment variable as instructed earlier. Happy compiling!

—David Brickner

Try really hard to forget about root. Here’s a quick comparison of the usage of root by a seasoned vet versus by a junior administrator.

Solid, experienced administrators will occasionally forget that they need to be root to perform some function. Of course they know they need to be root as soon as they see their terminal filling with errors, but running su -root occasionally slips their mind. No big deal. They switch to root, they run the command, and they exit the root shell. If they need to run only a single command, such as a make install, they probably just run it like this:

	$ su -c 'make install'

This will prompt you for the root password and, if the password is correct, will run the command and dump you back to your lowly user shell.

A junior-level admin, on the other hand, is likely to have five terminals open on the same box, all logged in as root. Junior admins don’t consider keeping a terminal that isn’t logged in as root open on a production machine, because “you need root to do anything anyway.” This is horribly bad form, and it can lead to some really horrid results. Don’t become root if you don’t have to be root!

Building software is a good example. After you download a source package, unzip it in a place you have access to as a user. Then, as a normal user, run your ./configure and make commands. If you’re installing the package to your ~/bin directory, you can run make install as yourself. You only need root access if the program will be installed into directories to which only root has write access, such as /usr/local.

My mind was blown one day when I was introduced to an entirely new meaning of “taking the root name in vain.” It doesn’t just apply to running commands as root unnecessarily. It also applies to becoming root specifically to grant unprivileged access to things that should only be accessible by root!

I was logged into a client’s machine (as a normal user, of course), poking around because the user had reported seeing some odd log messages. One of my favorite commands for tracking down issues like this is ls -lahrt/etc, which does a long listing of everything in the directory, reverse sorted by modification time. In this case, the last thing listed (and hence, the last thing modified) was /etc/shadow. Not too odd if someone had added a user to the local machine recently, but it so happened that this company used NIS+, and the permissions had been changed on the file!

I called the number they’d told me to call if I found anything, and a junior administrator admitted that he had done that himself because he was writing a script that needed to access that file. Ugh.

Junior admins tend to get really into customizing their environments. They like to show off all the cool things they’ve recently learned, so they have custom window manager setups, custom logging setups, custom email configurations, custom tunneling scripts to do work from their home machines, and, of course, custom shells and shell initializations.

That last one can cause a bit of headache. If you have a million aliases set up on your local machine and some other set of machines that mount your home directory (thereby making your shell initialization accessible), things will probably work out for that set of machines. More likely, however, is that you’re working in a mixed environment with Linux and some other Unix variant. Furthermore, the powers that be may have standard aliases and system-wide shell profiles that were there long before you were.

At the very least, if you modify the shell you have to test that everything you’re doing works as expected on all the platforms you administer. Better is just to keep a relatively bare-bones administrative shell. Sure, set the proper environment variables, create three or four aliases, and certainly customize the command prompt if you like, but don’t fly off into the wild blue yonder sourcing all kinds of bash completion commands, printing the system load to your terminal window, and using shell functions to create your shell prompt. Why not?

Well, because you can’t assume that the same version of your shell is running everywhere, or that the shell was built with the same options across multiple versions of multiple platforms! Furthermore, you might not always be logging in from your desktop. Ever see what happens if you mistakenly set up your initialization file to print stuff to your terminal’s titlebar without checking where you’re coming from? The first time you log in from a dumb terminal, you’ll realize it wasn’t the best of ideas. Your prompt can wind up being longer than the screen!

Just as versions and build options for your shell can vary across machines, so too can “standard” commands—drastically! Running chown -R has wildly different effects on Solaris than it does on Linux machines, for example. Solaris will follow symbolic links and keep on truckin', happily skipping about your directory hierarchy and recursively changing ownership of files in places you forgot existed. This doesn’t happen under Linux. To get Linux to behave the same way, you need to use the -H flag explicitly. There are lots of commands that exhibit different behavior on different operating systems, so be on your toes!

Also, test your shell scripts across platforms to make sure that the commands you call from within the scripts act as expected in any environments they may wind up in.

Many environments have strict rules about how software gets installed, how new machines are built and pushed into production, and so on. However, there are also thousands of sites that don’t enforce any such rules, which quite frankly can be a bit scary.

Not having the funds to come up with a proper testing and development environment is one thing. Having a blatant disregard for the availability of production services is quite another. When performing software installations, configuration changes, mass data migrations, and the like, do yourself a huge favor (actually, a couple of favors):

The process of scripting these procedures also forces you to think about the consequences of what you’re doing, which can have surprising results. I once got a quarter of the way through a script before realizing that there was an unmet dependency that nobody had considered. This realization saved us a lot of time and some cleanup as well.

The best tip any administrator can give is to be conscious of your own ignorance. Don’t assume you know every conceivable side effect of everything you’re doing. Ask. If the senior admin looks at you like you’re an idiot, let him. Better to be thought an idiot for asking than proven an idiot by not asking!

If you approach the powers that be and lead with a line about how Linux is free (as in beer), you’re likely doing yourself a disservice, for multiple reasons. First, if you point an IT manager at the Debian web site (home of what’s arguably the only “totally free in all ways” Linux distribution) and tell him to click around because this will be his new server operating system, he’s going to ask you where the support link is. When you show him an online forum, he’s going to think you are completely out in left field.

Linux IRC channels, mailing lists, and forums have given me better support for all technology, commercial or not, than the vendors themselves. However, without spending money on vendor support, your IT manager will likely feel that your company has no leverage with the vendor and no contractual support commitment from anyone. There is no accountability, no feel-good engineer in vendor swag to help with migrations, and no “throat to choke” if something goes wrong.

To be fair, you can’t blame him much for thinking this—he’s just trying to keep his job. What do you think would happen if some catastrophic incident occurred and he was called into a meeting with all the top brass and, when commanded to report status, he said “I’ve posted the problem to the linuxgoofball.org forums, so I’ll keep checking back there. In the meantime, I’ve also sent email to a mailing list that one of the geeks in back said was pretty good for support…”? He’d be fired immediately!

IT departments are willing to spend money for software that can get the job done. They are also willing to spend money for branded, certified vendor support. This is not wasted money. To the extent that a platform is only one part of a larger technology deployment, the money spent on the software and on support is their investment in the success of that deployment. If it costs less for the right reasons (fewer man hours required to maintain, greater efficiency), that’s great. But “free” is not necessary, expected, or even necessarily good.

It is also not Linux’s greatest strength, so leading with “no money down” is also doing an injustice to the people who create and maintain it. The cost of Linux did many things that helped it get where it is today, not the least of which was to lower the barrier of entry for new users to learn how to use a Unix-like environment. It also lowered the barrier of entry for developers, who were able to grow the technological foundation of Linux and port already trusted applications such as Sendmail and Apache to the platform, making it a viable platform that companies were willing to adopt in some small way. Leading with the monetary argument implies that that’s the best thing about Linux, throwing all of its other strengths out the window.

It’s useless (at best) to talk about running Linux in your shop without talking about it in the context of a solution that, when compared to the current solution, would be more useful or efficient.

To get Linux accepted as a viable platform, you have to start somewhere. It could be a new technology deployment, or it could be a replacement for an existing service. To understand the best way to get Linux in the door, it’s important to understand all of the aspects of your environment. Just because you know that management is highly displeased with the current office-wide instant messaging solution doesn’t mean that Jabber is definitely the solution for them. Whining to your boss that you should just move to Jabber and everything would be great isn’t going to get you anywhere, because you’ve offered no facts about Jabber that make your boss consider it an idea with any merit whatsoever. It also paints you in a bad light, because making blanket statements like that implies that you think you know all there is to know about an office-wide IM solution.

Are you ready for the tough questions? Have you even thought about what they might be? Do you know the details of the current solution? Do you know what might be involved in migrating to another solution? Any other solution? Do you know enough about Jabber to take the reins or are you going to be sitting at a console with a Jabber book open to Section 1.3 when your boss walks in to see how your big, high-profile, all-users-affected project is going?

“Linux is better” isn’t a credible statement. “A Linux file-sharing solution can work better at the department level because it can serve all of the platforms we support” is better. But what you want to aim for is something like “I’ve seen deployments of this service on the Linux platform serve 1,500 users on 3 client platforms with relatively low administrative overhead, whereas we now serve 300 clients on only 1 platform, and we have to reboot twice a week. Mean-while, we have to maintain a completely separate server to provide the same services to other client platforms.” The first part of this statement is something you might hear in a newbie Linux forum. The last part inspires confidence and hits on something that IT managers care about—server consolidation.

When talking to decision makers about Linux as a new technology or replacement service, it’s important to understand where they perceive value in their current solution. If they deployed the current IM solution because it was inexpensive to get a site license and it worked with existing client software without crazy routing and firewall changes, be ready. Can existing client software at your site talk to a Jabber server? Is there infrastructure in place to push out software to all of your clients?

It’s really simple to say that Linux rocks. It’s considerably more difficult to stand it next to an existing solution and justify the migration cost to a manager whose concerns are cost recovery, ROI, FTEs, and man-hours.

Linux is well suited to performing an enormous variety of tasks that are currently performed using lower-quality, higher-cost, proprietary software packages (too many to name—see the rest of this book for hints). There’s no reason to pitch it for tasks it can’t handle, as this will only leave a bad taste in the mouths of those whose first taste of Linux is a complete and utter failure.

What Linux is suitable for is 100% site-dependent. If you have a large staff of mobile, non-technical salespeople with laptops who use VPN connections from wireless hotspot sites around the globe, and you have a few old ladies manning the phones in the office all day, the desktop might not be the place for Linux to shine.

On the other hand, if you have an operator on a switchboard built in the 1920s, and the lifeblood of the business is phone communication, a Linux-based Asterisk PBX solution might be useful and much appreciated!

The point is, choose your battles. Even in Unix environments, there will be resistance to Linux, because some brands of Unix have been doing jobs for decades that some cowboy now wants Linux to perform. In some cases, there is absolutely no reason to switch.

Sybase databases have run really well on Sun servers for decades. Sybase released a usable version of their flagship product for Linux only about a year ago. This is not an area you want to approach for a migration (new deployments may or may not be another story). On the other hand, some features of the Linux syslog daemon might make it a little nicer than Solaris as a central log host. Some software projects readily tell you that they build, develop, and test on Linux. Linux is the reference Unix implementation in some shops, so use that leverage to help justify a move in that direction. Do your homework and pick your battles!

Personally, I’d rather have a deployment be nearly flawless than have it done yesterday. Both would be wonderful, but if history is any indication, that’s asking too much.

Don’t bite off more than you can chew. Let Linux grow on your clients, your boss, and your users. Get a mail server up and running. Get SpamAssassin, procmail, and a webmail portal set up on an Apache server. Then maintain it, optimize it, and secure it. If you do all this, Linux will build its own track record in your environment. Create a mailing list server. Build an LDAP-based white pages directory that users can point their email applications at to get user information. If you play your cards right, a year from now people will begin to realize that relatively few resources have been devoted to running these services, and that, generally, they “just work.” When they’re ready to move on to larger things, whom do you think they’ll turn to? The guy who wanted to replace an old lady’s typewriter with a dual-headed Linux desktop?

Think again. They’ll be calling you.

Managing your tasks won’t only ensure you get everything done in a timely manner. It will also help you make better predictions as to when work can be done and, more importantly, it will make your customers happier because you’ll do a better job of meeting their expectations about when their requests will be met. The next few sections discuss the methods you can use to order your tasks.

Prioritizing based on customer expectations.

Here’s a little secret I picked up from Ralph Loura when he was my boss at Bell Labs. If you have a list of tasks, doing them in any order takes (approximately) the same amount of time. However, if you do them in an order that is based on customer expectations, your customers will perceive you as working faster. Same amount of work for you, better perception from your customers. Pretty cool, huh?

So what are your customer expectations? Sure, all customers would love all requests to be completed immediately, but in reality they do have some conception that things take time. User expectations may be unrealistic, and they’re certainly often based on misunderstandings of the technology, but they still exist.

We can place user expectations into a few broad categories:

Some requests should be handled quickly.

Examples include requests to reset a password, allocate an IP address, and delete a protected file. One thing these requests have in common is that they often involve minor tasks that hold up larger tasks. Imagine the frustration a user experiences when she can’t do anything until a password is reset, but you take hours to get it done.

“Hurry up and wait” tasks should be gotten out of the way early.

Tasks that are precursors to other tasks are expected to happen quickly. For example, ordering a small hardware item usually involves a lot of work to push the order through purchasing, then a long wait for the item to arrive. After that, the item can be installed. If the wait is going to be two weeks, there is an expectation that the ordering will happen quickly so that the two-week wait won’t stretch into three weeks.

Some requests take a long time.

Examples include installing a new PC, creating a service from scratch, or anything that requires a purchasing process. Even if the vendor offers overnight shipping, people accept that overnight is not “right now.”

All other work stops to fix an outage.

The final category is outages. Not only is there an expectation that during an outage all other work will stop to resolve the issue, but there is an expectation that the entire team will work on the project. Customers generally do not know that there is a division of labor within a sysadmin team.

Now that we understand our customers’ expectations better, how can we put this knowledge to good use? Let’s suppose we had the tasks shown in Figure 4-4 on our to-do list.

Figure 4-4. Tasks that aren’t prioritized by customer expectations

If we did the tasks in the order listed, completing everything on the day it was requested in six and a half hours of solid work (plus an hour for lunch), we could be pretty satisfied with our performance. Good for us.

However, we have not done a good job of meeting our customers’ perceptions of how long things should have taken. The person that made request “T7” had to wait all day for something that he perceived should have taken two minutes. If I was that customer, I would be pretty upset. For the lack of an IP address, the installation of a new piece of lab equipment was delayed all day.

(Actually, what’s more likely to happen is that the frustrated, impatient customer wouldn’t wait all day. He’d ping IP addresses until he found one that wasn’t in use at that moment and “temporarily borrow” that address. If this were your unlucky day, the address selected would conflict with something and cause an outage, which could ruin your entire day. But I digress….)

Let’s reorder the tasks based on customer perceptions of how long things should take. Tasks that are perceived to take little time or to be urgent will be batched up and done early in the day. Other tasks will happen later. Figure 4-5 shows the reordered tasks.

Figure 4-5. Tasks ordered based on customer expectations

We begin the day by doing the two tasks (T1 and T7) that customers expect to happen quickly and that will hold up other, larger, projects. We succeed in meeting the perceived amount of time that these tasks should take, and everyone’s happy.

The previous section described ways to prioritize individual tasks. Now I’ll present some useful techniques for prioritizing projects.

Prioritization for impact.

Let’s say that you and your fellow sysadmins brain-stormed 20 great projects to do next year. However, you only have the budget and manpower to accomplish a few of them. Which projects should you pick?

It’s tempting to pick the easy projects and do them first. You know how to do them, and there isn’t much controversy surrounding them, so at least you’ll know that they’ll be completed.

It’s also very tempting to pick out the fun projects, or the politically safe projects, or the projects that are the obvious next steps based on past projects.

Ignore those temptations, and find the projects that will have the biggest positive impact on your organization’s goals. It’s actually better to do one big project that will have a large positive impact than many easy projects that are superficial—I’ve seen it many times. Also, an entire team working on one goal works better than everyone having a different project, because we work better when we work together.

Here’s another way to look at it. All projects can fit into one of the four categories listed in Figure 4-6.

Figure 4-6. Project impact versus effort

Doing category A projects first seems like the obvious course. An easy project that will have a big impact is rare, and when such projects magically appear in front of us, doing them always looks like the right choice. (Warning: Be careful, because their A status may be a mirage!)

It’s also obvious to avoid category D projects. A project that is difficult and won’t change much shouldn’t be attempted.

However, most projects are either in category B or C, and it’s human nature to be drawn to the easy C projects. We can fill our year with easy projects, list many accomplishments, and come away looking very good. However, highly successful companies train management to reward workers who take on category B projects—the difficult but necessary ones.

If you think about it in terms of Return on Investment (ROI), it makes sense. You’re going to spend a certain amount of money this year. Do you spend it on many small projects, each of which will not have a big impact? No, you look at what will have the biggest positive impact and put all your investment into that effort.

It’s important to make sure that these “big-impact” projects are aligned with your company’s goals—important for the company and important for you too. You will be more highly valued that way.

Managing your priorities ensures that you meet your customers’ expectations and get the work with the biggest impact done in a timely manner. However, prioritization is just one part of a time-management solution. Though you can go to general time-management books for more ideas, I humbly suggest a reading of my book, Time Management for System Administrators (O’Reilly).

—Tom Limoncelli