Unavoidable Subjectivity

There is a school of thought that if we make mathematical projections into the future, that we’re not estimating—that we’re projecting or forecasting, instead. We may be forecasting the future by projecting the current trend, that’s true. But as noted in the ​Definitions​, that’s a subset of estimation, not distinct from it.

Why would people make this distinction? One influence is to characterize this forecasting as rational and objective, rather than an emotional and subjective activity. Many people attracted to technical fields have the opinion that rational actions are inherently superior to emotional ones. Study of psychology will show that, at the very least, the rational and emotional aspects of humans are tangled together and ultimately inseparable. One of those emotional aspects is believing that decisions based on numerical calculations are superior to other ways of empowering decisions. This is an example of the Numeracy Bias. (See ​Cognitive Biases​.)

Even if you carefully avoid estimating based on opinion, there are still components of subjectivity that, if you’re not careful, can fool you. First is the choice of model. Even the simplest model is biased by the structure of the model. You may choose a model that seems to agree with the results you’ve seen in the past, or that worked in a different context. You might design or pick a model that gives you the answer you want or that “seems right.” The choice of data to feed into the model is another subjective choice. The model cannot correct for data that is missing or incorrect or biased in some manner.

There are cases where subjectivity has advantages over objectivity. Have you ever had a feeling, without any specific data you could use to prove your hunch, that something was about to change in a project? Perhaps you sense a change in mood of the development team or of someone associated with the project. People can subconsciously observe small nuances that they don’t explicitly notice. Don’t entirely discount the power of subjectivity.

Whatever model you use, you should check its calibration against your past experience. If you put the data from a past project into your model, do you get answers that accurately reflect what happened? Are those answers within the precision limits that you need?

Given the need to calibrate your model, you’ll notice that it’s still a form of estimating by comparison. The comparison has been broken down into two components, measuring the comparison reference and comparing your upcoming work to the measurements. In between, you can break the measurements down into components that are used to model the factors that affect the estimate.

While not fundamentally different from comparison estimation, a model can give you a good starting point with relatively little effort beyond creating the model. And as long as it’s giving you “good enough” estimates, it’s a cheap and easy way to go. It allows you to estimate arbitrary points in the future without a lot of reanalysis. And it should help you with work that doesn’t seem to resemble your past experience.

You should be safe as long as you don’t fall into the trap of believing that the answers your model gives are “the truth.” I find the concept of a singular knowable truth to be a bit slippery in the best of circumstances. It’s a seductive concept, but I find less risk by holding assertions lightly, keeping an eye open for observations that seem to contradict. It should be obvious that, particularly for estimating or forecasting the future, we cannot know “the truth.”

You will be prudent to compare your estimates with your actuals to check the continued validity of your model. Even if you calibrated the model with past data, things might have changed since that data was recorded. There may be factors that the model doesn’t take into account. There may be factors that the model seems to take into account, but not accurately. When the map and the territory disagree, believe the territory.

Armed with that disclaimer, let’s examine some of the different modeling approaches you might use.