Performance, Benchmarks, and Gigahertz
It's said in the computer world that there are lies, damn lies, and benchmarks. Like Mark Twain's original observation about statistics, the numbers don't tell the whole story, or even the half of it. Measuring the speed of a microprocessor chip might seem to be a simple, straightforward task, but it's anything but.
As an example of the problem, try defining the world's best car. Do you measure top speed? Trunk space? Fuel efficiency? Price? Resale value? You could combine and weight all these factors, but other people will inevitably argue that you've given too much importance to speed and not enough to passenger comfort, for example. You can't please everyone with a single score. This is the problem with measuring microprocessor chips (and computers). Everyone would like to see a single score for microprocessor “goodness,” but nobody agrees on what that should measure.
Tech Talk
Megahertz, Gigahertz, and Beyond
The speed of microprocessors (or any type of digital chip) is often measured in hertz. One hertz means one “change per second.” If you flip a coin every second, that coin is changing at the rate of 1 hertz. If you can somehow do that trick 1,000 times per second you'd be flipping a coin at 1,000 hertz, or 1 kilohertz(KHz). One million hertz is a megahertz (MHz) and 1 billion hertz is a gigahertz (GHz).
Inexpensive microprocessors might run at 20 MHz or so; more expensive processors for PCs run at more than 1 GHz. The unit of measurement is named after Heinrich Hertz, a German scientist who began measuring the frequency of radio waves back in 1883.
There is no theoretical limit to how high these chip frequencies can go; it's been creeping up year after year with advances in silicon manufacturing technology. If you're a radio operator, you might recognize some of these frequencies from the radio spectrum. Today's microprocessors do indeed broadcast tiny radio signals in the megahertz and gigahertz bands, which is why computers are shielded in metal boxes to minimize interference with TVs and radios.
The easiest thing to measure is the clock speed of a microprocessor, also called its frequency. A chip might run at 5 MHz, 500 MHz, or 5,000 MHz, but at least it's a verifiable number. Unfortunately, it's also pretty meaningless. The clock speed of a microprocessor is like the revolutions per minute (RPM) of a car's engine; it makes a nice “zoom, zoom” noise when you rev it up, but it doesn't tell you anything about real horsepower. Big frequency numbers add sizzle to advertisements and marketing literature, but they're not the steak.
To try to bring order to this chaos, a number of people have created special computer programs called benchmarks. Benchmarks instruct a microprocessor to perform repetitive tasks and then measure the time it took to finish them. They are the microprocessor equivalent of drag racing. (The word benchmark comes from an ages-old carpenters' habit of cutting regularly spaced marks on a workbench to measure materials.)
Benchmarks are fine, as far as they go, but people often read too much into them. Like the miles per gallon (MPG) sticker on a car, benchmarks are only an estimate. Your mileage may vary. Even if the estimate is correct, one number can't tell you very much about the whole product.
Tech Talk
Measuring Dhrystone MIPS
DEC made a computer called the VAX 11/780, which was the first computer to perform 1 million instructions per second, in 1978. (This turned out to be untrue, but that's another story.) It wasn't long before other computer companies wanted to prove their products were as fast as the VAX.
They looked for a program that was easy to translate for different types of computers, and Dhrystone was selected. The VAX happened to run the Dhrystone program 1,757 times per second. This set the bar. Now any computer that could do more than 1,757 runs per second was judged to be faster than a VAX.
Bizarrely, the magic number of 1,757 Dhrystone runs per second began to be used as the definition of 1 MIPS. Whether a computer was really performing 1 million instructions per second (1 MIPS) or not, the Dhrystone score was used as proof. Even computers that were faster than 1 MIPS were downgraded to 1 MIPS based on their Dhrystone performance. The same scale is still used today.
Misleading MIPS
If you're optimistic, MIPS stands for “millions of instructions per second.” If you're cynical, it's “meaningless indicator of performance for salesmen.” The MIPS number was for a long time the accepted way to measure a microprocessor's performance. Over time it has slowly outgrown is usefulness and it certainly does not measure anything useful anymore.
Early computer engineers naturally liked to compare their latest computers with their rivals' machines. To do this, they'd count the number of instructions each computer could execute per second. The more instructions per second, the faster the computer. In the 1970s, performing at 1 MIPS was a big achievement, like breaking the sound barrier.
Today's microprocessors in PCs and games can easily polish off hundreds of millions of instructions per second, and for less money than those early computers. However, they aren't the same kind of instructions, so comparing today's computers to yesterday's isn't really fair. It's like comparing a modern Formula 1 auto race to a historic bicycle race. Of course the modern racers go around the track faster, but what does that prove?
By the same token, it's not relevant, or fair, to compare the MIPS ratings of different microprocessors. All microprocessors execute different kinds of instructions (called their instruction set), and they're not directly comparable. Person A might be able to read Japanese faster than Person B can read French, but that doesn't tell us anything useful about either one of them.
Tech Talk
How to Pronounce—and Spell—MIPS
If you really want to impress your colleagues, be sure you always pronounce and spell MIPS with the capital S on the end. Don't get caught saying “one MIP” because there's no such thing. It's always “one MIPS” because the final S stands for “seconds,” sort of like the H in miles per hour (MPH). Even though it sounds funny, “one MIPS” is correct.
The Mythical MIPS-per-MHz Ratio
One favorite trick of chip makers is to brag about the “efficiency” of their processors with a MIPS-per-MHz ratio. Obviously this number is derived by taking the number of MIPS (which might be suspect, as described earlier) and dividing by the chip's clock speed measured in megahertz. The larger this number is, the more “efficient” the chip is judged to be.
There are a few problems with this approach. First, nobody cares. Of what value is it to a consumer to know that one chip delivers more MIPS (whatever those are) per millionth of a second? Isn't the end result what matters? Multiplying the two numbers—MIPS times megahertz—might make sense. Dividing one by the other is pointless.
Second, there's the squishy definition of MIPS itself. It might measure how many millions of instructions the chip executes per second, but probably not. More likely it's based on Dhrystone, or another benchmark, or drawn from a hat containing competitors' scores. Even if the true MIPS number is used, that's not a useful indication of work, efficiency, or forward progress.
Finally, the ratio is self-defining; it's a circular reference. Chip makers often define the MIPS rating for their microprocessor by multiplying its frequency (megahertz) by some secret number, essentially reversing the MIPS/MHz equation. With only two numbers, and one of them suspect, the MIPS/MHz ratio is really the “megahertz multiplied by some number we made up in the marketing department” ratio.
Power Consumption and MIPS/Watt
Lately it's become fashionable to design microprocessor chips so that they use the least amount of electricity possible. For battery-operated devices like laptop computers, the motivation for this is obvious. The longer the battery lasts, the happier the customer is. Lower power consumption also means less heat, and sometimes heat is a bigger problem than battery life. Again, customers won't be happy with a laptop computer that gets too hot to touch.
For these reasons and others, engineers have made rapid progress toward reducing the power consumption of high-end microprocessors to manageable levels. It naturally follows, then, that these same chip makers want to trumpet their advancements to their customers and the press. Alas, measuring power consumption is fraught with as many difficulties as measuring performance. The seemingly straightforward task of measuring electrical consumption has become a battlefield among warring microprocessor companies.
The most common measure of power efficiency is the MIPS-per-watt ratio. As we just saw with MIPS-per-MHz, the first of these two numbers is unreliable to begin with. Sadly, the second number can be just as erratic. Measuring the power usage of a processor isn't easy because it changes all the time. The amount of electricity the chip uses depends on what it's doing. It's easy to invent a situation where the processor is relatively idle—vacationing, if you will—and consuming little energy. It's also easy to concoct situations where the processor is fantastically busy and consuming far more electricity. This shouldn't be surprising. Your TV uses more power when it's turned on than when it's turned off. Which condition do you measure?
That depends on whether it's turned on or turned off most of the time. (Let's hope it's the latter.) It also depends on whether you want to know overall power consumption (e.g., over the duration of a month) or instantaneous power consumption (e.g., for a period of 10 seconds). In the same way, a processor's power consumption can be dialed in to nearly any number that suits its promoters.
So we're left with one suspect number (MIPS) divided by another suspect number (watts of power) to produce a monumentally suspect number. Bearing, as it does, no relation to anything tangible or useful, chip makers might as well be measuring furlongs per fortnight.