Environment

The environment is the testbed and the system that will run the software under test. Environments should be representative of realistic platforms and scenarios, yet balanced against noise such as third-party apps or traffic, unneeded applications, or other users. Generally, environments should be as controlled as possible. Even temperature can affect energy measurements.

N-Versions

The first step to any successful attempt at optimization is to measure system performance before optimization. If we are measuring software energy consumption to determine the impact of changing the code, we should measure the system before the modification to allow for comparison. Our work in Green Mining has shown that software does indeed change in energy performance over time, and measuring just one version of the software might not be representative of the versions before or after. It is recommended that multiple versions of the software are measured.

Energy or Power

Energy is the cost of work, or the capacity to do work, and is typically measured in joules (J). Power is the rate of energy consumption, essentially the derivative of energy consumption, measured in watts (W), where 1 W is equal to 1 Js.

When you measure a task that a system executes, ask your system, does this task have a clear beginning or end? Does this task continuously run? Tasks that do not to run continuously, such as sharpening an image or compressing a video file, can be characterized by the energy consumed. Whereas a task that runs continuously, such a sharpening video images of a surveillance web-camera or streaming video compression, is better characterized by its workload, its power, and the rate of energy consumption.

Repeat!

While we already recommended measuring multiple versions of software, it is even more important to repeat your measurements. Modern computers are very noisy and active systems, with lots of background processes and lots of states. They have many peripherals and services. It is hard to guarantee what tasks are executing on a modern system and what the current environment is like. If you're running a test using WIFI, your own cellphone could affect the experiment. Furthermore, we're measuring physical phenomena: energy consumption. Our measurement equipment, our testbeds, or energy measurement devices all have errors in them, error is inherent in physical measurement. Thus, we need to take multiple measurements so we can rely upon statistics to give us a more clear picture of what we measured.

Granularity

When designing an energy consumption test, one has to choose the level of granularity. Do you want to measure method calls, system calls, CPU use, memory use, processes, etc., and at what frequency? Many measurement devices and ICs, such as the Watts Up? Pro device or the TI INA-219 IC, are sampling at thousands of times per second, integrating the results and reporting back to you at a fraction of that rate. If you only have 1 s of granularity from a Watts Up? Pro you won't be measuring the cost of a method call unless you explicitly make a benchmark that repeatedly calls it. If you need method call level measurements, the instrumentation overhead will be high.

Granularity is a concern if you measure just a single process or component, or if you measure the whole system. An example scenario is if you asked the sound card to play a sound for 1 s. The request to play a sound might return in 1 ms to the process, for the next second the OS, the sound card driver and sound card will be interacting, playing the requested 1 s of audio. If we measure at the process level, we miss the induced cost on the system of the software. The software commanded work, but we are not measuring that work. If we measure at the entire system level, we will be including a lot of other processes and drivers that will be irrelevant.

Idle Measurement

Not all applications or tasks have idle behavior, but many user-facing applications and services run continuously or intermittently in the foreground or background. What programs do when they aren't in direct use is important, because usually their idle operation is a waste of resources, unless work is being executed during idle time. An efficient process can stay asleep until it receives input. The operating system's scheduler is very good at sleeping and waking processes.

Idle use often characterizes baseline energy consumption for an application. If executing tasks and idleness use the same amount of energy, perhaps the idle components can be optimized to use less energy. Measuring idle consumption is useful to determine what impact changes might have on the implementation of nonfunctional requirements.

Statistical Analysis

Repeat measurements cause many problems: there is no longer one measurement, there are many. Thus, summary statistics can be employed to describe the distribution of measurements. There is one saving grace: energy measurements are of physical phenomena, and the Normal (or Gaussian) distribution often models the errors within natural measurements well. Thus, the variance and mean are two reasonable descriptors of our multiple runs. This means we can use parametric tests, such as the Student's T-test, to compare two distributions of measurements to see if they are statistically significantly different. Without statistical tools such as the T-test, we would not have much confidence to determine if distributions were different or not based on random chance.

Exceptions

To err is human and to throw uncaught exceptions is to execute code. Mobile devices and modern computers still suffer from crashing software. Exceptions happen, core dumps occur, sometimes apps decide to update, sometimes the network goes down, sometimes a remote site is unavailable. Often the software under test is just inherently buggy and only half of the test runs will complete. When developing tests, one should instrument the tests with auditing capabilities such as screenshots, to enable postmortem investigations. Furthermore, outliers should be investigated and potentially re-run. You wouldn't want to attribute a difference in energy consumption to an erroneous test.